From 89c1b942b509b1ca571b34055a715fa874b2098d Mon Sep 17 00:00:00 2001 From: Himanshu Date: Fri, 24 Jul 2026 12:49:44 +0530 Subject: [PATCH 01/36] docs: claude mcp integration --- docs/v2/RESEARCH_PRODUCT_DIRECTION.md | 666 ++++++++++++++++++++++++++ docs/v2/RFC_FOUNDRY_MIGRATION.md | 597 +++++++++++++++++++++++ 2 files changed, 1263 insertions(+) create mode 100644 docs/v2/RESEARCH_PRODUCT_DIRECTION.md create mode 100644 docs/v2/RFC_FOUNDRY_MIGRATION.md diff --git a/docs/v2/RESEARCH_PRODUCT_DIRECTION.md b/docs/v2/RESEARCH_PRODUCT_DIRECTION.md new file mode 100644 index 0000000..0e46936 --- /dev/null +++ b/docs/v2/RESEARCH_PRODUCT_DIRECTION.md @@ -0,0 +1,666 @@ +# Research Study: Real-World Validation & MCP Product Expansion + +Status: **RESEARCH — no code changed, no architecture changed.** +Date: 2026-07-24 +Role posture: Principal Engineer / Staff Observability Engineer / Product Lead. +Scope: evolve the product. The five-module spine, the epoch model, and the +never-over-report invariant are treated as fixed. + +--- + +## Executive Summary + +**The single most important finding in this study: Claude Code, OpenAI Codex CLI, +Goose, and the OpenHands SDK all emit OpenTelemetry natively, today, with zero +instrumentation work.** Claude Code exports metrics, events, *and* (in beta) a full +span hierarchy — `claude_code.interaction` → `llm_request` / `tool` / `hook` — with +W3C `traceparent` propagation into subprocesses and MCP calls. SigNoz publishes +first-party monitoring docs for both Claude Code and Codex. + +This collapses the cost of Part 1 from "instrument an agent" to "set four environment +variables." spanLedger can stop auditing a synthetic Groq loop +([demo/agent-app/app.py](../../demo/agent-app/app.py)) and start auditing the +telemetry of the *judge's own coding agent* — same afternoon, no new code. + +**The second finding reframes the product.** The evidence on OTel collector failure is +unambiguous: when a sending queue fills, spans are *"silently dropped with no error or +warning in most configurations, and no indication in your traces that data was lost."* +Every incumbent detection method is **inferential** — Datadog's Data Observability +infers from row-count anomalies; `otelcol_exporter_send_failed_spans` reports only +what the collector *knows* it dropped. Silent loss is, by definition, loss the +collector did not count. + +spanLedger is the only system in this space with **ground truth by construction**: it +knows exactly what was sent because it sent it. That is not a feature. That is a +different epistemic category, and it is the entire moat. + +**The third finding is the timing.** AI agents now read observability data through MCP +(SigNoz ships a 40+ tool MCP server). An agent querying traces over a window that +silently lost 4% of spans will produce a *confident, well-reasoned, wrong* answer, with +no signal that anything was missing. Nobody is solving this. spanLedger already +computes exactly the number that solves it. + +**Therefore the recommendation is a spanLedger MCP server whose flagship tool is a +trust oracle** — `can_i_trust_this_data(from, to, stream)` — returning a verdict an +agent must consider before reasoning over telemetry. This positions spanLedger as the +**trust layer for agentic observability**: it extends platforms rather than competing +(philosophy preserved), it is a genuinely new category, and it is roughly a week of +work because the SLO engine, ledger, and event spine already compute the answer. + +Ranked directions: **(1) MCP trust oracle**, (2) real-agent validation harness, +(3) honest benchmark + published detection-latency numbers, (4) CI trust gate, +(5) multi-backend verification. If only one thing is built: **the MCP trust oracle.** + +--- + +## Part 1 — Real-World Validation + +### 1.1 Agent survey + +Rated for the actual question: *can this produce real, high-volume, verifiable +telemetry through an audited pipeline with minimal effort?* + +| Agent | Emits OTel natively? | Effort | Telemetry quality | Demo fit | +|---|---|---|---|---| +| **Claude Code** | **Yes** — metrics + events + traces (beta) | **Very low** (env vars) | **Excellent** | **Best** | +| **OpenAI Codex CLI** | **Yes** — opt-in `[otel]` in `config.toml` | Low | Good, with gaps | Strong | +| **Goose** | **Yes** — built-in OTLP/HTTP exporter | Low | Good | Good | +| **OpenHands SDK** | **Yes** — auto-instruments `agent.step`, tools, LLM calls | Low–Medium | Very good | Good | +| **LangGraph / LangChain** | Via `opentelemetry-instrumentation-langchain` | Medium | Good | Medium | +| **CrewAI** | **No** — third-party/OpenLLMetry only | Medium | Variable | Weak | +| **AutoGen** | Via OpenLIT/OpenLLMetry auto-instrumentation | Medium | Variable | Weak | + +#### Claude Code — **the primary target** + +*Native OTel.* `CLAUDE_CODE_ENABLE_TELEMETRY=1` plus standard `OTEL_EXPORTER_OTLP_*` +variables. Traces via `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` + +`OTEL_TRACES_EXPORTER=otlp`. + +- **Effort: essentially zero.** Point `OTEL_EXPORTER_OTLP_ENDPOINT` at the demo + agent's `:14317` and Claude Code's telemetry flows through the *existing* audited + pipeline (agent → toxiproxy → gateway → SigNoz). No new code, no new container. +- **Quality: excellent and unusually well-suited.** Three signals — metrics (60s + interval), events/logs (5s), traces (5s) — exercise all three of spanLedger's probe + legs including the logs leg that [PENDING_WORK.md](PENDING_WORK.md) flags as + unverified (spike S6). The span hierarchy is deep and real. Export intervals are + documented, so expected volume is predictable — which matters for honest loss math. +- **Demo fit: unbeatable.** The audience at a SigNoz hackathon uses Claude Code. The + line *"this is your coding agent's telemetry, and 4% of it silently never arrived"* + lands in one sentence. SigNoz has first-party Claude Code monitoring docs, so the + integration is on-narrative for the host. +- **Limitations, stated honestly.** Traces are **beta** — the span shape can change. + Volume is human-paced (a coding session, not 10k spans/sec), so high-throughput + benchmarks still need the synthetic generator. Content-logging flags + (`OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_RAW_API_BODIES`) are off by default and **must + stay off** for a public demo — they carry prompt text. Standard attributes include + `user.email` and `organization.id`; a recording needs scrubbing or a throwaway account. + +#### OpenAI Codex CLI — **the strong second** + +Opt-in `[otel]` block in `~/.codex/config.toml`, OTLP/HTTP and gRPC, custom +endpoints/headers. SigNoz publishes Codex monitoring docs. + +- **Value: it proves vendor-neutrality.** Auditing two different vendors' agents + through one pipeline demonstrates spanLedger is protocol-level, not + Anthropic-specific — important for credibility and for adoption. +- **Limitation, evidence-backed:** [openai/codex#12913](https://github.com/openai/codex/issues/12913) + reports `codex exec` emits no OTel metrics and `codex mcp-server` emits no telemetry + at all. Coverage is uneven by subcommand. Verify which paths actually emit before + building a demo on it. + +#### Goose / OpenHands — **good, not first** + +Goose has a built-in OTLP/HTTP exporter (`OTEL_EXPORTER_OTLP_ENDPOINT`). OpenHands' +SDK auto-instruments `agent.step`, tool executions, and LiteLLM calls, with per- +conversation session IDs. Both are legitimate targets; neither adds anything Claude +Code and Codex don't already give us, so they are breadth, not priority. + +#### CrewAI / AutoGen / LangGraph — **deprioritize** + +CrewAI *"doesn't emit any OpenTelemetry data on its own"* and needs third-party +instrumentation. LangGraph needs `opentelemetry-instrumentation-langchain`. These are +frameworks you'd *build* an app in, not tools you already run — so validation means +first writing an app, which is synthetic traffic wearing a costume. **Rejected for +Phase 1**: they add engineering effort and subtract authenticity. + +> **Recommendation: Claude Code primary, Codex CLI secondary, synthetic generator +> retained for throughput.** Do not delete [demo/agent-app](../../demo/agent-app) — real +> agents give authenticity, the synthetic generator gives *volume and determinism*. +> A benchmark needs both. Keep them as complementary traffic sources. + +### 1.2 Benchmark scenarios + +The repo already has five chaos scripts. Evaluating against three independent axes: + +| Scenario | Ease of demo | Showcases spanLedger | Production frequency | Verdict | +|---|---|---|---|---| +| **Queue overflow / sending_queue full** | Medium | **Highest** | **Very high** | **Flagship** | +| Backend outage (`outage.sh`) | **Highest** | Medium | High | Opener | +| SIGTERM mid-flight (`sigterm.sh`) | High | **High** | **Very high** | Strong #2 | +| OOM kill (`oomcrash.sh`) | Medium | High | Medium | Keep | +| Partial / signal-specific loss | Medium | **Highest** | Medium | Differentiator | +| Network delay / slow ingestion | **Low** | Medium | High | Metric, not demo | +| High throughput | Low | Medium | High | Benchmark, not demo | +| Exporter auth failure | High | Low | Medium | Skip | + +**Which failures best showcase spanLedger — the key analytical point.** + +There is an inverse relationship between *how easy a failure is to demo* and *how much +it proves*. A full backend outage is trivially visible: dashboards go flat, everyone +sees it, **you don't need spanLedger to notice**. It's the right opener because it's +legible, but it proves the least. + +**Queue overflow is the flagship** precisely because it is invisible. The evidence is +explicit: spans are *"silently dropped with no error or warning… and no indication in +your traces that data was lost."* The application succeeded. The collector kept +running. The dashboard shows a slightly lower line that looks like a traffic dip. +**This is the failure mode spanLedger exists for**, and it is common in production — +the entire `otelcol_exporter_enqueue_failed_spans` / `queue_size` monitoring practice +exists because of it. + +**SIGTERM mid-flight is the strongest second** — the repo already reproduces +[signoz#13853](https://github.com/SigNoz/signoz/issues/13853). Every deploy, every pod +eviction, every autoscale-down is a SIGTERM. It's continuous, universal, and nobody +counts it. + +**Partial / signal-specific loss is the differentiator.** Losing *only logs* while +traces stay green (`gateway-logs-down.yaml`, already written) is something a +volume-anomaly detector cannot cleanly attribute — total volume barely moves. Per-signal +SLIs make it obvious. This is where spanLedger beats the incumbent approach outright. + +**Deliberately excluded from the demo:** network delay and high throughput. Both are +real and both belong in the *benchmark*, but neither is watchable in three minutes. +Slow ingestion in particular is dangerous on stage — it's indistinguishable from +`maturity_delay` doing its job, which is a confusing story, not a compelling one. + +### 1.3 Metrics that matter + +Current metrics are correct but **inward-facing** — they describe the pipeline. An SRE +adopting a new tool asks a different question first: *why should I believe you?* That +demands **auditor-quality metrics**, which spanLedger can produce and no competitor can, +because spanLedger alone has ground truth. + +**Tier 1 — auditor quality (build these; they are the credibility argument)** + +| Metric | Definition | Why it matters | +|---|---|---| +| **Detection latency** | loss onset → `loss` event emitted | The headline number. One number that says how fast you learn. Directly comparable to any alternative. | +| **False positive rate** | `loss` events with no induced fault | Must be **provably 0**. This is the never-over-report invariant expressed as a measurement. An auditor that cries wolf is worse than none. | +| **False negative rate** | induced loss not reported | Bounds the honest claim. Publishing this is *why* the zero-FP claim is believable. | +| **Recovery detection latency** | recovery → `recovery` event | Closes the incident loop; drives MTTR. | +| **Unknown ratio** | `unknown / total` | **The honesty metric.** Quantifies what spanLedger declines to judge. A tool that publishes its own uncertainty is trustworthy in a way one that doesn't can never be. | + +**Tier 2 — pipeline health (already exist, keep)** +`spanledger_sli_ratio`, `error_budget_remaining_ratio`, `burn_rate` (multi-window), +`probe_e2e_latency_seconds`, `probes_unknown_total`. + +**Tier 3 — incident-shaped (mostly derivable from the ledger already)** +Incident duration, loss-gap size distribution, time-to-first-detection per signal, +per-signal SLI divergence (`traces` green while `logs` burns). + +#### Dashboard KPIs — recommendation + +The existing dashboards are Tier-2-centric. Add **one new dashboard, "Auditor +Quality,"** with exactly five tiles. Do not add tiles to the existing three. + +1. **Detection latency (p50 / p95)** — the headline. +2. **False positives: 0** — a stat tile that is a zero, with the induced-fault count + beside it for context. +3. **Delivery SLI vs target, per signal** — the product. +4. **Error budget remaining + fast-burn** — the operator's action trigger. +5. **Unknown ratio** — the honesty tile. + +Rationale: tiles 1, 2, and 5 are *unique to an auditor with ground truth*. They are the +tiles no other vendor can render. That is the dashboard that wins the argument. + +### 1.4 Demonstration strategy — the three-minute demo + +Constraints: visually compelling, technically honest, reproducible, understood by +someone who has never heard of spanLedger. + +**Title: "Your agent's telemetry is lying to you."** + +**0:00–0:30 — Establish the stake.** +Split screen. Left: SigNoz Traces Explorer, live, filling with `claude_code.*` spans — +*"this is Claude Code's real telemetry, flowing through a normal OTel pipeline."* +Right: spanLedger, delivery SLI 100%, budget full, zero events. Both green. + +**0:30–1:15 — Induce silent loss.** +Run the queue-overflow scenario (**not** the outage — the whole point is invisibility). +Narrate: *"I have not stopped anything. The collector is running. Claude Code is +working normally and reports no errors."* +Left panel: spans keep arriving. The graph dips slightly. **It looks like less +traffic.** Say that out loud — that's the hook. A viewer's instinct is "so what?" + +**1:15–2:00 — The reveal.** +spanLedger flips: SLI drops below target, `loss` event with an exact contiguous gap +(`seq 40–59`), burn rate crosses FAST_BURN, budget drains, alert fires in SigNoz. +The killer line: **"spanLedger knows exactly 20 spans are missing, because it sent +them. Nothing else in this stack can know that."** +Then click the `traces_filter` deep link into SigNoz Traces Explorer — the query +returns nothing where the probes should be. **The platform confirms the absence.** + +**2:00–2:40 — Recovery + the trust question.** +Restore. `recovery` event links back to the loss incident. Budget stops draining. +Then, in Claude Code: *"Can I trust the traces from the last ten minutes?"* — Claude +calls spanLedger's MCP server and answers **"No — 3.7% of telemetry in that window was +verified lost between 14:32 and 14:36. Any conclusion about that window is unsafe."** + +**2:40–3:00 — Honesty close.** +Show the auditor-quality tiles: detection latency p95, **false positives: 0**, unknown +ratio. *"It reported zero false alarms across N runs — and here is the fraction it +refuses to judge, published rather than hidden."* + +**Why this structure works.** It opens with a failure the audience will +under-estimate, makes them feel the invisibility, then resolves it with an exact +number. The MCP moment at 2:00 converts a monitoring demo into a *product* demo. The +honesty close pre-empts the obvious skeptical question instead of waiting for it in Q&A. + +**Reproducibility, non-negotiable:** every number on screen comes from +`demo/chaos/run_scenarios.py`. No slides with numbers, no cropped screenshots. The +repo's own standard — *"Every number shown in the demo must be reproducible by running +these scripts"* ([demo/README.md](../../demo/README.md)) — is already the right rule. + +--- + +## Part 2 — MCP Integration + +### 2.1 Should spanLedger expose an MCP server? + +**Yes — and this is the highest-leverage thing in this document.** + +The argument is not "MCP is popular." It is a specific, currently-unaddressed failure: + +> An AI agent reads telemetry through MCP. The window it reads silently lost 4% of +> spans. The agent produces a fluent, confident, **wrong** conclusion — "no errors in +> that service" — and has no way to know its input was incomplete. + +Silent loss was always bad. When a human read a dashboard, a human's skepticism was the +last line of defense. Agents have no such skepticism, they act on conclusions, and +SigNoz's own MCP server (40+ tools) makes this the *default* way telemetry gets read. +**The blast radius of silent loss just increased, and spanLedger already computes the +missing input.** + +That reframing also protects the philosophy: spanLedger's MCP server does not serve +telemetry. It serves **verdicts about whether telemetry can be trusted**. It extends +the platform's MCP server rather than duplicating it. + +### 2.2 Architecture + +``` +Claude Code / Cursor / Codex / Windsurf + │ MCP (stdio, default) + ▼ + spanledger mcp ← new thin process, or `--transport http` + │ localhost HTTP + ▼ + spanLedger daemon :8231 ← UNCHANGED. Existing /api/v2/* only. + │ + ▼ + SQLite ledger (already the one source of truth) +``` + +**Design rules, in priority order:** + +1. **Read-only. Zero write tools.** The 2026 baseline for MCP servers is read-only- + by-design; *"if a server has write access… the bar must be higher."* spanLedger is + already an auditor — read-only is philosophically native, not a compromise. It also + means an injected prompt cannot make spanLedger *do* anything. +2. **A separate process, not a thread in the daemon.** The auditor must stay + independent, and the MCP surface must not be able to stall the verify loop. This + also preserves clean rollback: delete the process, nothing changes. +3. **No new state, no new computation.** Every tool is a projection over + `/api/v2/*`. If a tool needs data the API doesn't expose, that is a signal to + extend the API deliberately — not to let the MCP layer compute its own answers. + *Store facts, compute answers; one source of truth.* +4. **Answers, not rows.** The tools return verdicts with evidence, not JSON dumps for + the model to interpret. Interpretation is where hallucination enters. + +### 2.3 Recommended tools + +Six tools. Each earns its place by answering a question a developer *actually asks*. +Deliberately not a mirror of the REST API. + +**1. `can_i_trust_this_data(from, to, stream?, signal?)` — the flagship.** +Returns `{trustworthy: bool, confidence, verified_ratio, unknown_ratio, incidents[], +recommendation}`. +*Why it exists:* it is the only tool that changes what the agent **does**. Every other +tool is informational; this one is a gate. It is the product thesis compressed into one +call, and no other MCP server in the ecosystem can answer it, because answering requires +ground truth. **If only one tool ships, this is it.** + +**2. `pipeline_status()`** — current SLI, budget, burn rate, open incidents, per stream +and signal. The "is it healthy right now" question. Cheap, and the natural entry point +an agent reaches for first. + +**3. `active_incidents()`** — open `loss` incidents with gap shape, affected signal, +onset, duration, and the `traces_filter` expression. +*Why separate from status:* status answers "healthy?", this answers "what is broken and +where do I look?" Returning the existing `traces_filter` (D15) hands the agent a +ready-to-run SigNoz query — the two MCP servers compose. + +**4. `diagnose_pipeline(stream?)`** — the reasoning tool. Correlates the loss window +against deploy markers, per-signal divergence, gap shape (contiguous vs scattered), and +the opt-in `correlation_hint` (`otelcol_*` deltas by hop). +*Why it exists:* this is what "intelligent teammate" means. Contiguous gap + deploy +marker 30s earlier = deploy-induced. Logs-only divergence = signal-specific exporter +fault. **Hard requirement: it returns evidence and candidate hypotheses with +confidence, never a single asserted root cause.** An auditor that guesses has stopped +being an auditor. + +**5. `telemetry_slo(stream?, window?)`** — SLI vs target, budget remaining, burn rates, +history. The planning question: "do we have budget to ship today?" Feeds the CI gate +(Direction 4) with the same numbers a human would read. + +**6. `incident_history(from, to, class?)`** — the ledger, paginated. Post-incident +review and "has this happened before?" Turns the SQLite journal into an institutional- +memory surface at near-zero cost. + +**Deliberately rejected:** +- `recent_probe_failures()` — leaks the implementation model. Developers care about + *their* telemetry, not spanLedger's probes. Probe mechanics belong in evidence + fields, not a top-level tool. +- Any raw telemetry query tool — that is SigNoz's MCP server's job. Duplicating it + violates "extend, don't compete" and creates a second source of truth. +- Any mutating tool (`silence_alert`, `mark_deploy`). Deploy markers are a legitimate + API write, but they are a *CI* action, not an agent action. Keep the MCP surface + read-only and the security story stays trivially defensible. + +### 2.4 Request flow + +``` +Developer: "Why are traces missing from checkout-service?" + → Claude calls signoz MCP: signoz_search_traces(...) → sparse results + → Claude calls spanledger MCP: can_i_trust_this_data(...) → trustworthy: false + → Claude calls spanledger MCP: active_incidents() → loss, seq 40-59, gateway-a + → Claude calls spanledger MCP: diagnose_pipeline("gateway-a") → contiguous gap, + deploy marker t-30s + → "Traces aren't missing from checkout-service. Your telemetry pipeline dropped + them — a contiguous 20-span gap starting 30s after the gateway v1.4 deploy. + The service is probably fine; the pipeline is not." +``` + +That answer is *categorically better* than what either MCP server produces alone, and +it is the difference between a developer spending an afternoon debugging the wrong +service and fixing the right one in five minutes. **This flow is the product.** + +### 2.5 Security + +Modest surface, but the discipline matters — *66% of 1,808 scanned MCP servers had at +least one security finding.* + +| Concern | Position | +|---|---| +| Write access | **None.** Zero mutating tools. Eliminates the largest class outright. | +| Auth | stdio inherits the developer's local trust boundary. HTTP transport binds `127.0.0.1` by default and requires a token to bind anything else. | +| Input validation | *"If a parameter can be an enum, do not let it be a string."* `signal` and `class` are enums; timestamps are validated ints; `limit` is capped at 500 — the existing `/api/v2` validation is already this strict and should be reused verbatim, not reimplemented. | +| Prompt injection | Telemetry content never reaches the model through spanLedger — only verdicts, counts, and its own event payloads. This is a real advantage of *not* being a telemetry query tool. | +| Data sensitivity | Probes are synthetic; findings contain sequence numbers and stream names. Nothing user-generated. Notably lower risk than the platform's own MCP server. | +| Availability | MCP failure must never affect detection. Separate process; daemon unaware of it. | + +### 2.6 Client portability + +MCP is a protocol, not a Claude feature. A stdio server with a JSON config works +unchanged across **Claude Code, Cursor (`.cursor/mcp.json`), Windsurf, VS Code agents, +Codex, and Goose** — all standard MCP clients. HTTP transport covers remote/team setups. + +Cost of portability: **zero**, provided we don't use client-specific extensions. Ship +config snippets for Claude Code and Cursor; the rest follow the same shape. Worth +stating explicitly in the README — "works with any MCP client" is a genuine adoption +argument, not a hedge. + +--- + +## Part 3 — Product Expansion + +Filtered hard against the philosophy. Anything requiring spanLedger to store telemetry, +become a query engine, or modify the audited pipeline is rejected on sight. + +**A. Telemetry trust gate for CI/CD — strongest adjacency.** +`spanledger check` already exists with a three-outcome exit code and the `2 = +inconclusive` discipline ([CI.md](CI.md)). The adjacent product is a **deploy gate**: +block a rollout when the telemetry pipeline is untrustworthy, because deploying blind +is the actual risk. Natural GitHub Action. Free adoption driver — CI integrations are +how infra tools spread. + +**B. Trust attestation / receipts — the enterprise wedge.** +The ledger already contains signed-quality facts. A **verifiable attestation** — +"between T1 and T2, stream X delivered 99.97% of telemetry, N probes verified, M +unknown" — is exactly what regulated industries need when audit or incident-review +requires proving observability coverage *was actually working*. Nobody can produce this +today because nobody has ground truth. **This is the clearest thing a company would pay +for.** + +**C. Multi-backend verification — the neutrality play.** +The verify leg is a Query API client. Add Grafana Tempo, Honeycomb, Datadog. Then +spanLedger becomes *the* vendor-neutral verification layer and — critically — can +answer **"did the same span reach both backends?"** during a migration. Backend +migration is a real, expensive, recurring pain with no good tooling. Strategically +significant, but do it **after** SigNoz depth is proven; premature breadth would weaken +the hackathon story. + +**D. Cost-of-loss accounting — natural, defer.** +The ledger knows what was lost; billing is volume-based. "You paid for 4.2M spans and +3.9M arrived" is a CFO-legible number. Attractive, but it depends on per-vendor pricing +models and risks pulling the product toward FinOps. **Note it, don't build it.** + +**Rejected outright:** storing/replaying telemetry (second source of truth), auto- +remediating collectors (auditor must not modify the audited system), a general query UI +(competes with SigNoz). + +**Open-source vs. premium split.** Everything in Part 1 and 2 stays open — the auditor, +the MCP server, the dashboards, the CI gate. That's the adoption engine. Premium sits +where organizations, not individuals, feel pain: attestation/compliance reporting, +multi-backend and cross-backend reconciliation, long-horizon ledger retention, and +managed/hosted probes. + +--- + +## Part 4 — Competitive Analysis + +| Product | What it solves | Overlap | Why spanLedger differs | +|---|---|---|---| +| **SigNoz** | Storage, query, dashboards, alerts | **Partner, not competitor** | Cannot verify what never arrived. spanLedger's findings *terminate in SigNoz*. | +| **Datadog Data Observability** | Volume/row-count anomaly detection, freshness | **Closest competitor** | **Inferential vs. deterministic.** Anomaly detection asks "is this less than expected?" spanLedger asks "did probe 47 arrive?" Anomaly detection cannot distinguish real traffic drop from loss. spanLedger can, always. | +| **Datadog Observability Pipelines** | Pipeline routing + its own monitoring | Partial | Monitors *its own* pipeline. Not independent — the auditor is the audited. | +| **Grafana** | Dashboards on Prometheus/Tempo | Low | *"Does not have a native data observability offering"* — requires custom work. | +| **Honeycomb** | High-cardinality debugging | Low | Assumes data arrived. Orthogonal. | +| **OTel collector self-metrics** | `otelcol_exporter_send_failed_spans`, queue depth | **Most important to address** | **Reports only loss the collector counted.** Silent drops are, definitionally, uncounted. Also self-reported — it travels the failing pipeline. spanLedger is out-of-band by design. | +| **Langfuse / Helicone / Arize / Phoenix** | LLM trace quality, evals, cost, drift | **None** | Different layer entirely: they judge *content* quality, spanLedger judges *delivery*. Genuinely complementary — an eval over 96% of traces is silently wrong. | + +**Where differentiation is strongest — the one-sentence version:** + +> Everyone else infers loss from what arrived. spanLedger knows what was sent. + +Two structural advantages follow, and neither is copyable without rebuilding the +architecture: **ground truth by construction** (probes are generated, so the expected +set is known exactly — this enables the provable-zero-false-positive claim nobody else +can make) and **genuine independence** (reporting is out-of-band, so spanLedger stays +diagnostic while the pipeline it audits is broken — precisely when self-reported +metrics fail). + +**Honest competitive risk.** None of this is technically hard to copy. SigNoz could ship +"synthetic telemetry canaries" in a quarter. The defensibility is **not** the probe +loop — it is the accumulated rigor: the never-over-report invariant, the epoch/unknown +model, the finalization watermark, the three-outcome CI discipline. Those are *judgment* +encoded over time, and they are what makes the numbers trustworthy. **Strategic +implication: compete on trustworthiness and integration depth, never on feature count.** + +--- + +## Top Product Directions + +### 1. spanLedger MCP server — the telemetry trust layer ★ RANK 1 + +- **Description.** Read-only MCP server, six tools, flagship `can_i_trust_this_data()`. +- **Problem solved.** AI agents reason over telemetry with no way to know it was + incomplete, and produce confident wrong answers. +- **Effort: Low–Medium.** ~1 week. Thin projection over existing `/api/v2/*`; no new + state, no new math. +- **Demo value: Very high.** The 2:00 moment converts monitoring into product. +- **Long-term value: Very high.** Defines a category — trust layer for agentic + observability — and every AI-observability trend increases its value. +- **Risks.** MCP fatigue (mitigated: this is a gate, not another data source); tools + returning dumps instead of verdicts (mitigated by design rule 4); scope creep into + telemetry querying (mitigated by explicit rejections in §2.3). +- **Why #1.** Highest value-to-effort ratio in the study. It is the only direction that + creates a *new* category rather than improving an existing capability, it strengthens + the philosophy instead of straining it, and it is timely in a way that will not stay + true for long. + +### 2. Real-agent validation harness (Claude Code + Codex) ★ RANK 2 + +- **Description.** Replace/augment the synthetic Groq loop with real Claude Code and + Codex telemetry through the existing audited pipeline. +- **Problem solved.** "Does this work on real telemetry, or only your own probes?" — + the first question any serious evaluator asks. +- **Effort: Low.** Environment variables and documentation. Possibly the highest- + credibility-per-hour work available. +- **Demo value: Very high.** Authenticity is the whole point. +- **Long-term value: Medium-high.** Becomes the regression corpus. +- **Risks.** Claude Code traces are beta (shape may change); PII in default attributes + (`user.email`) needs scrubbing for public demos; human-paced volume insufficient for + throughput tests — keep the synthetic generator. +- **Why #2.** Nearly free and it makes #1 believable. Ranked below #1 only because it + strengthens the existing story rather than creating a new one. + +### 3. Auditor-quality benchmark + published numbers ★ RANK 3 + +- **Description.** Measure and publish detection latency (p50/p95), false-positive rate, + false-negative rate, and unknown ratio across all chaos scenarios. New "Auditor + Quality" dashboard. +- **Problem solved.** "Why should I trust the tool that tells me what to trust?" +- **Effort: Medium.** Instrumentation is easy; running a statistically honest campaign + and resisting flattering numbers is the work. +- **Demo value: High** — the honesty close. +- **Long-term value: High.** Published FP/FN numbers are the strongest possible + technical-credibility signal, and they close several + [PENDING_WORK.md](PENDING_WORK.md) items. +- **Risks.** **The numbers might be unflattering.** That must be published anyway — the + first flinch destroys the thing being sold. +- **Why #3.** Depends on #2 for realistic workloads, and #1 delivers more novel value + per hour. But this is what converts a demo into a credible product. + +### 4. Telemetry trust gate for CI/CD ★ RANK 4 + +- **Description.** GitHub Action wrapping `spanledger check`; block deploys when the + pipeline is untrustworthy; auto-post deploy markers. +- **Problem solved.** Teams deploy while blind and don't know it. +- **Effort: Low–Medium.** `check` and markers already exist. +- **Demo value: Medium** (CI is not visually exciting). **Long-term value: High** — + CI integration is how infra tools achieve durable adoption. +- **Risks.** Gating deploys on telemetry health will feel obstructive; must default to + warn, not block. Exit-2/inconclusive must never silently pass. +- **Why #4.** Excellent product, weak demo. Right after the hackathon, not before. + +### 5. Multi-backend verification ★ RANK 5 + +- **Description.** Tempo/Honeycomb/Datadog verify legs; cross-backend reconciliation. +- **Effort: High.** Every backend is a new query dialect and a new failure surface. +- **Demo value: Low** at a SigNoz event. **Long-term value: Very high** — vendor + neutrality and the migration-assurance use case. +- **Why #5.** Strategically the most valuable long-term item and the most premature. + Breadth before depth would weaken everything above it. + +--- + +## Recommended Roadmap + +**Week 1 — Real telemetry (Direction 2).** +Wire Claude Code into the demo pipeline; verify all three signals arrive; add Codex as +the vendor-neutrality proof; scrub PII; document. *Why first:* lowest effort, highest +credibility gain, and it produces the workload every later phase needs. It also finally +exercises the logs leg that spike S6 left unverified. + +**Week 2 — MCP trust oracle (Direction 1).** +`spanledger mcp`, stdio, read-only. Ship `can_i_trust_this_data`, `pipeline_status`, +`active_incidents` first; `diagnose_pipeline`, `telemetry_slo`, `incident_history` +second. Config snippets for Claude Code and Cursor. *Why second:* it needs real +telemetry from Week 1 to be demonstrable rather than theoretical. + +**Week 3 — Auditor-quality benchmark (Direction 3).** +Run every chaos scenario N times against real traffic; measure detection latency, FP, +FN, unknown ratio; build the Auditor Quality dashboard; publish including anything +unflattering. *Why third:* needs both real workload and a stable system. + +**Week 4 — Demo hardening + CI gate (Direction 4).** +Rehearse the three-minute demo end-to-end until every number is script-reproducible. +Ship the GitHub Action. Close out PENDING_WORK items the previous weeks resolved. + +**Sequencing logic:** each week produces something demonstrable on its own, and each +depends only on weeks before it. If the schedule slips, Weeks 1–2 alone are a complete, +compelling story. Direction 5 is explicitly out of the first month. + +--- + +## Final Recommendation + +**If only one thing gets built: the spanLedger MCP server, with +`can_i_trust_this_data()` as its flagship tool.** + +**The argument.** + +*It is the only direction that creates a category.* Directions 2–4 make spanLedger a +better version of what it already is. This one makes it something the ecosystem does +not currently have: a trust layer that AI agents consult before reasoning over +telemetry. Categories are worth more than features, and this one is being created right +now by the shift to agentic observability. + +*It is timely in a way that will not last.* SigNoz ships a 40+ tool MCP server. Agents +are becoming the default reader of observability data. Every one of them inherits the +silent-loss problem with **less** skepticism than the humans they replace. That gap is +open today and will be filled by someone within a year. + +*It is cheap because the hard work is done.* The SLO engine computes trustworthiness. +The ledger stores incidents. The event spine has severity and links. `traces_filter` +already hands an agent a runnable query. **The MCP server computes nothing new** — it +is a projection over `/api/v2/*`, which is exactly why it is safe and fast to build. + +*It strengthens the philosophy rather than straining it.* Read-only preserves auditor +independence. No new state preserves one-source-of-truth. Serving verdicts instead of +telemetry preserves "extend platforms, don't compete." Every philosophical constraint +is *satisfied more cleanly* by this direction than by any alternative — that is usually +the sign of a correct direction. + +*It converts the demo from impressive to necessary.* A judge watching a chaos demo +thinks "clever." A judge watching Claude Code refuse to answer a question because +spanLedger says the data can't be trusted thinks **"I need this."** That is a different +reaction, and it is the one that produces adoption. + +**The strongest counter-argument, addressed.** One could argue Direction 2 should be +first — it is cheaper and it makes everything credible. That is why it is Week 1 in the +roadmap. But it is not the *one thing*, because real telemetry makes the existing story +believable while the MCP server makes a **new** story possible. If forced to ship +exactly one artifact, ship the one that changes what the product *is*. + +**The risk I would accept.** MCP could turn out to be a passing integration fashion. I +accept it because the underlying asset — a verdict about whether a time window's +telemetry can be trusted — is valuable through *any* interface: REST, CLI, CI gate, +dashboard tile. MCP is the highest-leverage delivery vehicle available today, not the +value itself. If MCP fades, the tools become API endpoints and almost nothing is lost. + +**What I would not do.** Not multi-backend (breadth before depth). Not auto- +remediation (breaks auditor independence). Not a query UI (competes with the platform +we're extending). Not adding tools to the MCP server because they're easy — six tools +that answer real questions beat twenty that mirror an API. + +--- + +## Sources + +[Claude Code monitoring docs](https://code.claude.com/docs/en/monitoring-usage.md) · +[SigNoz: Claude Code monitoring](https://signoz.io/docs/claude-code-monitoring/) · +[SigNoz: Codex monitoring](https://signoz.io/docs/codex-monitoring/) · +[Codex config reference](https://developers.openai.com/codex/config-reference) · +[openai/codex#12913](https://github.com/openai/codex/issues/12913) · +[OpenHands observability](https://docs.openhands.dev/sdk/guides/observability) · +[Goose tracing](https://openobserve.ai/docs/integration/ai/no-code/codename-goose/) · +[OTel collector internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/) · +[OTel collector resiliency](https://opentelemetry.io/docs/collector/resiliency/) · +[Preventing OTel data loss](https://oneuptime.com/blog/post/2026-02-06-prevent-data-loss-opentelemetry-scenarios/view) · +[Diagnosing send_failed_spans](https://oneuptime.com/blog/post/2026-02-06-diagnose-data-loss-exporter-send-failed-spans/view) · +[Datadog Data Observability](https://docs.datadoghq.com/monitors/types/data_observability/) · +[SigNoz MCP server](https://signoz.io/docs/ai/signoz-mcp-server/) · +[MCP security best practices](https://stacklok.com/blog/mcp-security-best-practices-what-every-enterprise-team-needs-to-know-in-2026/) · +[OTel GenAI semconv status](https://opentelemetry.io/blog/2025/ai-agent-observability/) · +[Watching the watchers](https://nxlog.co/news-and-blog/posts/watching-the-watchers) diff --git a/docs/v2/RFC_FOUNDRY_MIGRATION.md b/docs/v2/RFC_FOUNDRY_MIGRATION.md new file mode 100644 index 0000000..ae2a910 --- /dev/null +++ b/docs/v2/RFC_FOUNDRY_MIGRATION.md @@ -0,0 +1,597 @@ +# RFC: Foundry / casting.yaml / MCP compliance migration + +Status: **DRAFT — awaiting approval. No code changed.** +Date: 2026-07-23 +Scope: infrastructure substrate + one new integration surface. No changes to the +five-module core (Prober → Registry → Verifier → Findings Engine → Reporter). + +--- + +## 0. Research basis + +Everything below is grounded in the current official sources, fetched 2026-07-23: + +| Fact | Source | +|---|---| +| `install.sh` deprecated as of SigNoz v0.130.0; foundryctl is the supported installer | [signoz#10924](https://github.com/SigNoz/signoz/issues/10924) | +| foundryctl install: `curl -fsSL https://signoz.io/foundry.sh \| bash`; Windows archives on GH releases | [foundry getting-started](https://github.com/SigNoz/foundry/blob/main/docs/getting-started.md) | +| `cast` = `gauge` (validate tools) → `forge` (generate files + **write `casting.yaml.lock`**) → deploy | [foundryctl CLI reference](https://github.com/SigNoz/foundry/blob/main/docs/reference/cli.md) | +| `forge` output goes to `./pours` (`-p/--pours`); lock file holds checksums of the resolved deployment state | same | +| **Windows prerequisite: WSL 2 + Docker Engine, _not_ Docker Desktop — ClickHouse Keeper crashes under Desktop's virtualization; disable Desktop's WSL integration** | [Docker standalone install](https://signoz.io/docs/install/docker/) | +| MCP is enabled *inside* casting.yaml (`spec.mcp.spec.enabled: true`), serves on :8000, health at `/livez` | same | +| MCP server has 40–50+ tools incl. `signoz_execute_builder_query`, dashboard/alert CRUD, `signoz_search_traces`/`_logs` | [signoz-mcp-server](https://github.com/SigNoz/signoz-mcp-server), [MCP docs](https://signoz.io/docs/ai/signoz-mcp-server/) | +| Modes/flavors: docker/compose, docker/swarm, k8s/helm, k8s/kustomize, systemd, ECS, Render… | [foundry README](https://github.com/SigNoz/foundry) | + +**Known research gap, stated plainly:** `foundry/docs/casting.md` returns 404 and the +README does not enumerate `spec` fields. The full casting schema — in particular +**whether a SigNoz version can be pinned** — is not published in a form I could read. +Consequence: *we must not hand-author `casting.yaml`.* Phase 1 below derives it from +`foundryctl gen examples` and `foundryctl gen schemas`, which are authoritative and +version-matched. Any RFC that invents a schema here would be guessing. + +> Terminology note: Deliverable 6 refers to "ATC". No such component exists in this +> repo; I've read it as spanLedger throughout. Correct me if it meant something else. + +--- + +## Deliverable 1 — Compliance audit + +### REQUIRED (a judge can mechanically fail us on these) + +**Gap 1 — casting.yaml / casting.yaml.lock absent.** + +> Requirement 3: *"Make your deployment reproducible. Your repo must include the +> `casting.yaml` and `casting.yaml.lock`. Judges may re-run Foundry against them to +> reproduce your deployment."* + +Neither file exists in the 196 tracked files. This is the only requirement using the +words "must include", and it is the only one with a stated verification procedure +("judges may re-run Foundry against them"). + +Consequence, concretely: a judge clones the repo and has **no path to a running +system at all**. [demo/compose.yaml](../../demo/compose.yaml) declares +`networks: signoz-network: {external: true}` — it attaches to a SigNoz that must +already exist, installed the way you installed it. `docker compose up` fails +outright on a fresh machine. Our reproducibility story today is "have the same +undocumented v0.133.0 install I have." + +**Gap 2 — SigNoz not installed via Foundry.** + +> Requirement 1: *"Install SigNoz using Foundry. Foundry installs both SigNoz and its +> MCP server in one step."* + +[DECISIONS.md:7](../../DECISIONS.md) records this as a deliberate deviation, and the +stated reason is **correct and now officially confirmed**: SigNoz's own docs say +Docker Desktop is unsupported because ClickHouse Keeper crashes under its +virtualization. The engineering judgment was sound. The problem is that the official +answer to that constraint is "use WSL 2 + Docker Engine", not "use the v0.99.0 +compose manifests" — and the v0.99.0 path is now doubly non-compliant, since +`install.sh`/raw compose was deprecated at v0.130.0. + +Consequence: this gap **causes Gap 1 and Gap 3**. No Foundry → no `forge` → no lock +file, and no `spec.mcp` block → no MCP server. It is the single root cause. + +**Gap 3 — MCP server unused.** + +> Requirement 2: *"Using the SigNoz MCP server, Query Builder, dashboards, and alerts +> is recommended to maximize your chances of winning."* + +Strictly this is worded *recommended*, so it sits in the next tier for scoring — but +requirement 1 asserts Foundry installs MCP "in one step", so its total absence reads +to a judge as direct evidence that requirement 1 was skipped. I'm classifying it as +Required for that reason: it is the visible symptom of the Required gap. + +### RECOMMENDED (scoring surface, not pass/fail) + +- **Query Builder, dashboards, alerts: already satisfied, strongly.** 3 dashboards, + 8 alert rules, config-aware rendering via [assets.py](../../spanledger/assets.py), + v5 Query Builder payloads in [signoz.py](../../spanledger/signoz.py), deep links in + [signoz-links.ts](../../frontend/src/lib/signoz-links.ts). Nothing to fix. +- **Dashboards/alerts have never been imported into a live SigNoz.** + [PENDING_WORK.md](PENDING_WORK.md) is candid that `/api/v1/dashboards` and + `/api/v1/rules` in [validate_assets.py](../../demo/validate_assets.py) are a + "best-effort guess" and the panel JSON schema is "a best-effort reading, not + live-confirmed". A judge who clicks Import and gets a 400 sees the assets as + decorative. **MCP fixes this specific gap** — see Deliverable 6. +- **Three-signal coverage** (traces/metrics/logs) exists; the logs leg is unverified + live (spike S6 outstanding). + +### NICE TO HAVE + +- AI-assistant disclosure (protocol 7) — submission-form item, nothing to change in + the repo, but non-disclosure is a stated DQ. Track it. +- Pinning the SigNoz version in casting.yaml if the schema supports it (unknown — + see research gap). Would make judge reproduction byte-exact. +- MCP dashboard template (`signoz_import_dashboard`) for the MCP server's own health. + +--- + +## Deliverable 2 — Migration strategy + +### The core insight: this is a substrate swap, not a migration + +spanLedger **never embeds SigNoz**. It talks to one over the network, through exactly +three contact points: + +| Contact point | Current value | Defined in | +|---|---|---| +| Query API base | `http://localhost:8080` | `spanledger.yaml` → `signoz.query_url` | +| OTLP ingest (out-of-band reporting) | `localhost:4317` | `spanledger.yaml` → `signoz.ingest_url` | +| In-network collector hostname | `signoz-ingester:4317` | [demo/otel/gateway.yaml](../../demo/otel/gateway.yaml) exporter | +| Docker network | external `signoz-network` | [demo/compose.yaml](../../demo/compose.yaml) | + +That is the entire coupling surface. Everything else — the five modules, the SLO +engine, the SQLite journal, the dashboards, the alerts, the frontend — is indifferent +to *how* SigNoz got installed. + +**So the smallest compliant migration is: replace the hand-rolled v0.99.0 compose +install with a Foundry install, and reconcile those four values. Nothing else moves.** + +### Current architecture + +``` +[hand-installed SigNoz v0.133.0] ← non-compliant, unreproducible + network: signoz-network (external) + host: signoz-ingester:4317, UI :8080 + ▲ ▲ + │ OTLP │ Query API v5 (+ out-of-band OTLP :4317) + │ │ + demo/compose.yaml ──────────┴── spanledger (host process, :8231) + agent → toxiproxy → gateway └─ frontend (Vite) +``` + +### Target architecture + +``` +casting.yaml ──foundryctl forge──▶ pours/ (generated, gitignored) + │ casting.yaml.lock (COMMITTED) + └──foundryctl cast──▶ [SigNoz + MCP :8000] ← compliant, reproducible + network: + ▲ ▲ + │ OTLP │ Query API v5 + demo/compose.yaml ──────────────┴──────────┴── spanledger (unchanged) + agent → toxiproxy → gateway └─ frontend (unchanged) + │ + MCP :8000 ─┘── spanledger MCP client (NEW, additive) +``` + +Deltas, exhaustively: **one new file** (`casting.yaml`), **one new generated-and- +committed file** (`casting.yaml.lock`), **one edited value** in `demo/compose.yaml` +(network name) and **one in `demo/otel/gateway.yaml`** (exporter hostname) *if and +only if* Foundry's names differ, **one `.gitignore` line** (`pours/`), and **one new +additive module** for MCP. Zero changes to `spanledger/` core. + +### Migration steps + +1. Establish a Linux-shaped Docker host (Deliverable 3 — WSL 2 + Docker Engine). +2. Install foundryctl; `foundryctl gen examples` + `gen schemas` to obtain the real + casting schema for the installed foundryctl version. +3. Author minimal `casting.yaml` (docker/compose + `spec.mcp.spec.enabled: true`). +4. `foundryctl gauge` → `forge` → inspect `pours/` → `cast`. +5. Read Foundry's actual network name and collector service name out of + `pours/deployment/*.yaml` and `docker network ls`. Reconcile the four contact + points above. +6. Re-run the existing test suite + the two live drills in [OPERATIONS.md](OPERATIONS.md). +7. Commit `casting.yaml` + `casting.yaml.lock`; gitignore `pours/`. +8. Build the MCP integration (Deliverable 6) — strictly additive, off by default. + +### Risks + +| # | Risk | Severity | Mitigation | +|---|---|---|---| +| R1 | **Foundry installs a newer SigNoz than v0.133.0.** [DECISIONS.md:8](../../DECISIONS.md) validated the v5 `query_range` payload *against v0.133.0's source*, and that endpoint "disallows unknown fields at every level". A schema drift breaks the verify loop — spanLedger's core function. | **High** | Run the v5 payload against the new instance *before* anything else (Phase 2 gate). Pin the version in casting.yaml if the schema allows. `spikes/s2_query_filters.py` already exists as the probe for this. | +| R2 | Dashboard/alert JSON rejected by the newer import API. | Medium | Already unverified today (PENDING_WORK) — this is not a regression, it's the first real test. MCP `signoz_create_dashboard` replaces the guessed REST paths. | +| R3 | Foundry's network/service names differ → demo pipeline can't reach the collector. | Medium | Purely a config reconcile; detected instantly (probes stop arriving, ratio → 0). Step 5 handles it. | +| R4 | Frontend deep links break (`signoz-links.ts` targets "SigNoz v0.x" URL format). | Low | Cosmetic; copy-to-clipboard is already the documented primary action and the link is behind a settings flag. | +| R5 | WSL 2 networking: services in WSL not reachable from Windows-host `localhost`. | Medium | WSL 2 forwards localhost by default; if it fails, run spanLedger inside WSL too. Deliverable 3 covers both placements. | +| R6 | Two SigNoz installs contend for :8080/:4317. | Low | Never run both. Old stack is the rollback target, kept stopped. | + +### Rollback + +Rollback is cheap because the old install is *not deleted* — it is stopped. + +- **Full rollback:** `docker compose -f pours/deployment/compose.yaml down`, restart + the v0.133.0 stack, revert the network/hostname values. The four contact points are + the only things that moved. +- **Partial (keep Foundry, drop MCP):** MCP is additive and default-off; delete the + `spec.mcp` block, re-`cast`. +- **Git:** every phase is one commit (per CLAUDE.md's single-line, no-attribution + format). `git revert` per phase. +- **Data:** none at risk. [OPERATIONS.md](OPERATIONS.md) already states deleting + `spanledger.db` is safe at any time, and SigNoz is not spanLedger's system of record. + +--- + +## Deliverable 3 — Windows strategy + +The decision is unusually clear here because **SigNoz's own docs state the answer**: +*"Linux, macOS, or Windows with WSL 2 + Docker Engine (not Desktop)… Install Docker +Engine natively in WSL, not Docker Desktop. ClickHouse Keeper crashes under Desktop's +virtualization. Disable Desktop's WSL integration in settings."* + +### Option A — Native Windows + Docker Desktop + +- **Pros:** zero setup change; current workflow preserved exactly. +- **Cons:** explicitly unsupported. ClickHouse Keeper crashes — this is precisely what + DECISIONS.md:7 already discovered independently. foundryctl ships a Windows binary, + so `gauge`/`forge` would succeed and `cast` would produce a broken stack: the worst + failure mode, because it yields a plausible-looking `casting.yaml.lock` from a + deployment that never actually worked. +- **Likelihood of success: ~10%.** Rejected. + +### Option B — WSL 2 + Docker Engine installed *inside* WSL (recommended) + +- **Pros:** the officially documented Windows path. Real Linux Docker, no Desktop + virtualization layer, so Keeper is fine. Machine stays your dev machine — repo is + reachable at `/mnt/c/...` or (better) cloned into the WSL filesystem. foundryctl's + Linux binary is the best-tested one. localhost forwarding means `query_url: + http://localhost:8080` likely keeps working *unmodified from Windows*. +- **Cons:** must disable Docker Desktop's WSL integration, which affects any other + Docker work on this machine. Two Docker daemons to keep straight. `/mnt/c` file I/O + is slow (mitigation: clone into `~` in WSL, or keep Python on Windows and only + Docker in WSL — see below). +- **Likelihood of success: ~85%.** + +Within Option B there are two sub-placements. **Recommended: B1.** + +- **B1 — Docker + SigNoz in WSL; spanLedger (Python) + frontend stay on Windows.** + Preserves your current dev loop *exactly* (`.venv`, pytest, ruff, Vite all unchanged + on Windows). Relies on WSL 2 localhost forwarding for `:8080`/`:4317`/`:8000`. This + is the minimum-disruption option and the one that best honors "do not break the + current development workflow". +- **B2 — everything in WSL.** More faithful to a Linux CI environment, removes all + networking doubt, but relocates the entire dev workflow. Fallback if B1's networking + misbehaves (risk R5). + +### Option C — Temporary Linux VM / cloud box + +- **Pros:** cleanest isolation; guaranteed-correct `casting.yaml.lock`; zero impact on + the Windows machine; disposable. +- **Cons:** SigNoz is then *not* on your dev machine, so the demo pipeline, chaos + scripts, and frontend all need to reach it over a network — that's a bigger change + to the working setup than Option B, not a smaller one. Costs money if cloud. The + drills in OPERATIONS.md assume local `docker stop`. +- **Likelihood of success: ~95% for producing the lock file, ~50% for keeping the demo + workflow intact.** + +### Recommendation + +**Option B1**, with **Option C as the escape hatch** used *only* to generate a valid +`casting.yaml.lock` if WSL 2 proves hostile. Rationale: B1 is the officially supported +Windows path, it satisfies the reproducibility requirement with a genuinely-executed +`cast`, and it leaves your Python/Vite loop untouched. C is strictly worse on the +"don't break the working setup" axis despite scoring higher on raw install success. + +**Non-negotiable:** the lock file must come from a real `forge` on a host where `cast` +actually succeeded. A hand-written or copied lock file is worse than none — judges are +told they may re-run Foundry against it. + +--- + +## Deliverable 4 — How Foundry fits + +**Foundry owns the SigNoz *backend* install. It owns nothing else.** + +| Question | Answer | +|---|---| +| Should development happen through Foundry? | **No.** Foundry installs and manages the SigNoz backend. Your dev loop (`python -m spanledger run`, `pytest`, `ruff`, Vite) is unrelated and stays as-is. | +| Should only deployment use Foundry? | Foundry *is* the deployment of the observability backend — for both dev and judging. There is one SigNoz install method now, and it's Foundry. | +| Can Foundry coexist with the current setup? | **Yes, and this is the key point.** `demo/compose.yaml` is a *separate* compose project (`name: spanledger-demo`) that attaches to SigNoz's network as `external`. Foundry generates *its own* compose project in `pours/`. The two are peers. Foundry replaces only the SigNoz half — the audited-pipeline half (agent → toxiproxy → gateway → agent-app) is ours and stays hand-written. | +| What stays unchanged? | All of `spanledger/`. All of `frontend/`. All of `assets/`. `demo/chaos/*`. `demo/agent-app/`. `test/`. `pyproject.toml`. The five-module architecture, the epoch model, the never-over-report invariant, the SQLite journal. | +| What changes? | Adds `casting.yaml` + `casting.yaml.lock`. Possibly two string values (network name, ingester hostname). One `.gitignore` line. README install section. A new DECISIONS.md entry superseding 2026-07-17. | + +**Explicitly do NOT** try to model the demo pipeline as Foundry "moldings". Moldings +are SigNoz's own components (ClickHouse, Postgres, the OTel Collector). Our +agent/toxiproxy/gateway is the *system under audit*, deliberately hand-configured for +fault injection — putting it under Foundry would be a rewrite for zero compliance +benefit and would destroy the chaos scripts. + +--- + +## Deliverable 5 — casting.yaml design + +### Structure + +Confirmed-minimal, from the official Docker quickstart: + +```yaml +apiVersion: v1alpha1 +kind: Installation +metadata: + name: signoz +spec: + deployment: + flavor: compose + mode: docker + mcp: + spec: + enabled: true # requirement 2 — MCP on :8000, health at /livez +``` + +**This is a sketch, not the file to commit.** Per the research gap in §0, the +authoritative schema for the installed foundryctl version comes from: + +```bash +foundryctl gen examples # working casting files for every mode +foundryctl gen schemas # JSON schemas — the real field list +``` + +Phase 1 runs those first and derives the committed file from them. Specifically we +need `gen schemas` to answer: **can the SigNoz version be pinned?** If yes, pin it — +that de-risks R1 and makes judge reproduction exact. If no, record the deployed +version in DECISIONS.md and accept float. + +### What belongs in it + +Only SigNoz's own stack: ClickHouse, ZooKeeper/Keeper, Postgres, the SigNoz query +service + UI, the SigNoz OTel Collector, and the MCP server. Nothing of ours. + +### How our project maps in + +It doesn't — and that's correct. spanLedger is an external observer of SigNoz, not a +component of it. The mapping is by *reference*: casting.yaml produces the endpoints +(`:8080`, `:4317`, `:8000`) that `spanledger.yaml` and `demo/compose.yaml` consume. +The seam is those endpoint strings and the docker network name. + +### How casting.yaml.lock is generated + +`foundryctl forge -f casting.yaml` writes it — checksums capturing the resolved +deployment state, so re-applies are deterministic. It is a **build output that must be +committed**, exactly like `package-lock.json` (which this repo already commits at +[frontend/package-lock.json](../../frontend/package-lock.json)). Never hand-edit; +regenerate by re-running `forge` and commit the diff. + +### Commit / ignore policy + +| Path | Policy | Why | +|---|---|---| +| `casting.yaml` | **commit** | Required by field requirement 3. Hand-authored source of truth. | +| `casting.yaml.lock` | **commit** | Required by field requirement 3. Reproducibility. | +| `pours/` | **gitignore** | Regenerable output of `forge`; deriving it is the judge's step. | + +### Judge reproduction path + +```bash +git clone && cd SpanLedger +curl -fsSL https://signoz.io/foundry.sh | bash +foundryctl cast -f casting.yaml # gauge → forge → deploy, ~5 min +docker compose -f demo/compose.yaml up -d # the audited pipeline +pip install -r requirements.txt && python -m spanledger run --config demo/spanledger.demo.yaml +``` + +This must be written verbatim into README.md and rehearsed once end-to-end. It is the +single most load-bearing artifact of this migration. + +--- + +## Deliverable 6 — MCP integration + +### Principle + +Not a checkbox. The test I applied: **does the MCP server let spanLedger close a gap +that [PENDING_WORK.md](PENDING_WORK.md) currently lists as open?** Three integrations +pass that test. They are ranked, and they compound. + +### M1 — Asset publishing via MCP (replaces guessed REST paths) — **strongest** + +PENDING_WORK.md states that `demo/validate_assets.py`'s `/api/v1/dashboards` and +`/api/v1/rules` are *"a best-effort guess"* and the panel schema is *"a best-effort +reading, not live-confirmed"*. That is a real, documented, unresolved defect: we ship +3 dashboards and 8 alerts that have never been proven to import. + +MCP resolves it directly. `signoz_create_dashboard`, `signoz_create_alert`, +`signoz_list_dashboards`, `signoz_list_alert_rules` are versioned, contract-stable +tools — no path guessing, no schema archaeology. + +**Product form:** `spanledger assets publish --config --via mcp` — a sibling to the +existing `assets render` subcommand in [cli.py](../../spanledger/cli.py:129), reusing +[assets.py](../../spanledger/assets.py) unchanged. `validate_assets.py` is retained as +the REST fallback, not deleted. + +**Why it's the strongest:** it converts our *biggest unverified claim* into a verified +one, it's genuinely the right tool for the job (not MCP-for-MCP's-sake), and it makes +the dashboards demonstrably real to a judge. + +### M2 — Audit-the-auditor cross-check — **best narrative fit** + +Phase 2 success criterion 2 requires that the dashboard's Query Builder formula +(SigNoz computing SLI from raw counters) agrees with `spanledger_sli_ratio` (our +engine computing it). OPERATIONS.md notes *"no code makes this true or false"* and +PENDING_WORK.md says it *"needs a live screenshot for the demo deck"*. + +`signoz_execute_builder_query` turns that manual screenshot into an automated +assertion: ask SigNoz to recompute the SLI from raw counters, diff against our gauge, +fail if they diverge beyond one export interval. + +**Product form:** `spanledger check --cross-verify` — an additional, optional +signal inside the existing bounded CI audit. Exit codes stay the documented three; +a divergence maps to exit 2 (inconclusive), never to a false loss verdict, preserving +the never-over-report invariant. + +**Why it matters:** spanLedger's entire thesis is *"who audits the observability +pipeline?"* An auditor that uses the platform's own AI interface to check its own +arithmetic against the platform is the thesis, executed. This is the demo moment. + +### M3 — Incident enrichment on a loss finding — **best demo optics** + +Loss events already carry `traces_filter` (D15) — the exact Query Builder expression +for the missing probes' neighbourhood, e.g. +`spanledger.stream = 'gateway-a' AND spanledger.seq >= 40 AND spanledger.seq <= 59`. +Today that string powers a clipboard copy and a deep link. Feed the same string to +`signoz_search_traces` / `signoz_aggregate_logs` and an operator gets the surrounding +evidence *attached to the finding* instead of a link to go click. + +**Product form:** opt-in `mcp.enrich_findings: true`, off by default, strictly +off the detection path — any MCP failure degrades to no enrichment, exactly as the +existing `correlation_hint` degrades via `select_top_anomaly` (DECISIONS.md 2026-07-21). + +### Recommendation + +Build **M1 first** (closes a real defect, smallest surface), then **M2** (the +strongest product story), then **M3** if time allows. All three share one thin client +module and one config block. All three are additive and default-off. + +### Deliberately rejected + +- Replacing [signoz.py](../../spanledger/signoz.py)'s hand-rolled v5 client with MCP. + The verify loop is the hot path, runs every `poll_interval`, and its payload shape is + source-validated. Routing the core detection path through an AI-oriented protocol + adds a failure mode to the one thing that must never produce a false verdict. **The + verify loop stays on the direct Query API.** +- MCP-driven auto-remediation. Out of scope, and spanLedger is deliberately read-only + with respect to the audited pipeline. + +--- + +## Deliverable 7 — Implementation plan + +Five phases. Each ends in one single-line commit (CLAUDE.md format, no attribution). +**Phases 1–3 are the compliance-critical path; 4–5 are the scoring surface.** + +--- + +### Phase 0 — Windows/WSL 2 substrate (no commit) + +- **Goal:** a Docker Engine inside WSL 2 that foundryctl's `gauge` accepts. +- **Files modified:** none. +- **Infrastructure:** enable WSL 2; install Docker Engine natively in the distro; + **disable Docker Desktop's WSL integration**; ≥4 GB to Docker; free ports 8080, + 4317, 4318, 8000. +- **Testing:** `docker run hello-world` inside WSL, from a non-Desktop daemon. +- **Validation:** `docker context ls` shows the WSL daemon, not Desktop's. +- **Rollback:** re-enable Desktop integration. Nothing committed. + +--- + +### Phase 1 — foundryctl + casting.yaml (no deploy yet) + +- **Goal:** an authoritative, schema-correct `casting.yaml`. +- **Files modified:** `+casting.yaml`, `.gitignore` (`pours/`). +- **Infrastructure:** install foundryctl; `foundryctl gen examples`; **`foundryctl gen + schemas`** — read it, and settle the version-pinning question (R1). +- **Testing:** `foundryctl gauge -f casting.yaml` exits 0. +- **Validation:** `foundryctl forge -f casting.yaml` produces `pours/` and + `casting.yaml.lock`; inspect the generated compose for the network name, the + collector service name, and the SigNoz version. **Do not commit the lock yet** — it + isn't trustworthy until `cast` has succeeded. +- **Commit:** `Add Foundry casting.yaml for reproducible SigNoz deployment` +- **Rollback:** delete the file; nothing is running yet. + +--- + +### Phase 2 — Foundry deploy + core-path regression gate ← **highest-risk phase** + +- **Goal:** SigNoz running under Foundry with the verify loop proven intact. +- **Files modified:** `casting.yaml.lock` (committed here, post-success); + `demo/compose.yaml` and `demo/otel/gateway.yaml` **only if** Foundry's network / + collector names differ; `spanledger.yaml` + `demo/spanledger.demo.yaml` only if + ports moved. +- **Infrastructure:** stop the v0.133.0 stack (**do not delete it — it is the + rollback target**). `foundryctl cast -f casting.yaml`. Create a service-account API + key → `SIGNOZ_API_KEY`. +- **Testing, in this order — this is the gate:** + 1. `python spikes/s2_query_filters.py` — **does the v5 payload still validate?** + This is R1. If it fails, stop and reassess before touching anything else. + 2. `pytest` — full suite green. + 3. `docker compose -f demo/compose.yaml up -d`; confirm probes arrive and delivery + ratio reaches ~1.0. + 4. The gateway-kill drill from [OPERATIONS.md](OPERATIONS.md) — ratio drops, `loss` + finding emitted, `recovery` on restart, never-over-report holds. + 5. The restart drill — exactly one `epoch_orphaned`, zero spurious `lost`. +- **Validation:** UI at :8080; `curl -fsS localhost:8000/livez` (MCP alive); + ports reachable from the Windows host if running B1. +- **Commit:** `Deploy SigNoz via Foundry and commit casting.yaml.lock` +- **Rollback:** `docker compose -f pours/deployment/compose.yaml down`; restart the + v0.133.0 stack; `git revert`. Contact points are the only moving parts. + +--- + +### Phase 3 — Reproducibility rehearsal + +- **Goal:** prove a judge can actually reproduce this. **This is what requirement 3 + is really testing.** +- **Files modified:** `README.md` (Foundry quickstart), `DECISIONS.md` (new entry + superseding 2026-07-17 — *keep the old entry, record why it changed*), + `demo/README.md` (drop the "already have SigNoz" prerequisite). +- **Infrastructure:** ideally a **second, clean** WSL distro or VM — clone from the + git remote and run the Deliverable 5 command sequence verbatim. +- **Testing:** the four commands, from a clean clone, no local state. +- **Validation:** SigNoz UI up, MCP `/livez` OK, probes flowing, ratio ~1.0 — with + nothing typed that isn't in the README. +- **Commit:** `Document Foundry-based reproducible deployment` +- **Rollback:** docs-only; trivially revertible. + +> **Compliance is achieved at the end of Phase 3.** Phases 4–5 are scoring. + +--- + +### Phase 4 — MCP integration M1 + M2 + +- **Goal:** meaningful MCP use that closes documented defects. +- **Files modified:** `+spanledger/mcp.py` (thin client), `+test/test_mcp.py`, + `spanledger/cli.py` (`assets publish`, `check --cross-verify`), + `spanledger/config.py` (`mcp:` block — optional, default-off, matching the existing + fail-fast unknown-key validation), `docs/v2/OPERATIONS.md`. +- **Infrastructure:** none — MCP is already running from Phase 2. +- **Testing:** unit tests with a mocked MCP transport; then live — + `spanledger assets publish` imports all 3 dashboards + 8 alerts cleanly (**this + closes the PENDING_WORK "never actually POSTed" item**); `check --cross-verify` + agrees with `spanledger_sli_ratio`. +- **Validation:** dashboards visibly present and rendering in the SigNoz UI; MCP + failure injected → degrades to exit 2, never a false loss verdict. +- **Commits:** `Add SigNoz MCP client for asset publishing` / + `Add MCP cross-verification to spanledger check` +- **Rollback:** default-off config; delete the module. Core untouched. + +--- + +### Phase 5 — M3 + cleanup (optional) + +- **Goal:** finding enrichment; retire stale docs. +- **Files modified:** `spanledger/findings.py` or `events.py` (enrichment hook), + `docs/v2/PENDING_WORK.md` (**remove** the items Phases 2–4 actually closed — the + file's own rule is "remove an entry once it's actually done"). +- **Testing:** loss finding carries enrichment; MCP down → no enrichment, no verdict + change. +- **Commit:** `Enrich loss findings with MCP trace context` +- **Rollback:** config flag off. + +--- + +## Success criteria traceability + +| Criterion | Phase | Evidence | +|---|---|---| +| ✓ Foundry deployment | 2 | `foundryctl cast` succeeded; stack running | +| ✓ casting.yaml | 1 | committed, schema-derived | +| ✓ casting.yaml.lock | 2 | generated by a `forge` whose `cast` succeeded | +| ✓ meaningful MCP integration | 4 | closes 2 documented PENDING_WORK defects | +| ✓ reproducible deployment | 3 | rehearsed from a clean clone | +| ✓ Windows-friendly workflow | 0 | WSL 2 + Docker Engine (official path); Python/Vite stay on Windows | +| ✓ no regression | 2 | pytest + both OPERATIONS.md drills | +| ✓ full compliance | 3 | all three Required gaps closed | + +## Constraint compliance + +| Constraint | How honored | +|---|---| +| Do not rewrite working code | Zero changes to the five core modules. Phases 1–3 touch config strings and docs only. | +| Do not replace existing observability | The verify loop stays on the direct Query API v5 client — MCP is explicitly rejected for the hot path. | +| Do not remove dashboards | All 3 retained; Phase 4 *proves they import* for the first time. | +| Do not remove alerts | All 8 retained, published via MCP. | +| Do not remove Query Builder | Retained and extended — `signoz_execute_builder_query` adds a second QB consumer. | +| Extend what exists | `assets publish` is a sibling of `assets render`; `--cross-verify` is a flag on the existing `check`; enrichment reuses the existing `traces_filter`. | + +--- + +## Open questions for approval + +1. **Version pinning.** If `gen schemas` shows casting.yaml cannot pin a SigNoz + version, do we accept float (and re-validate the v5 payload each time), or hold at + the version Foundry installs and record it? +2. **Option B1 vs B2.** B1 keeps Python/Vite on Windows (minimum disruption) but + depends on WSL 2 localhost forwarding. Confirm B1 is the intent — B2 is the + fallback, not a parallel effort. +3. **Phase 4–5 scope.** If time is short, is Phase 3 an acceptable stopping point? + It is the full compliance boundary; 4–5 are scoring upside. +4. **"ATC"** in Deliverable 6 — assumed to mean spanLedger. Confirm. From 62343ed16df1d7a87452e3db2e52c05f6a8e01f0 Mon Sep 17 00:00:00 2001 From: Himanshu Date: Fri, 24 Jul 2026 13:04:55 +0530 Subject: [PATCH 02/36] docs: implementation plan v1 --- docs/v2/IMPL_TRUST_LAYER.md | 568 ++++++++++++++++++++++++++++++++++++ 1 file changed, 568 insertions(+) create mode 100644 docs/v2/IMPL_TRUST_LAYER.md diff --git a/docs/v2/IMPL_TRUST_LAYER.md b/docs/v2/IMPL_TRUST_LAYER.md new file mode 100644 index 0000000..d202593 --- /dev/null +++ b/docs/v2/IMPL_TRUST_LAYER.md @@ -0,0 +1,568 @@ +# Implementation Strategy: Real Telemetry + MCP Trust Layer + +Status: **PLAN — no code changed.** +Date: 2026-07-24 · Target: SigNoz Hackathon, Track 2 +Prereq: research complete ([RESEARCH_PRODUCT_DIRECTION.md](RESEARCH_PRODUCT_DIRECTION.md)), +Foundry migration planned ([RFC_FOUNDRY_MIGRATION.md](RFC_FOUNDRY_MIGRATION.md)). + +Architecture is fixed. Everything below is additive or configuration. + +--- + +## The one technical fact that shapes this entire document + +**spanLedger cannot verify Claude Code's spans individually. It never could, and it +must never claim to.** + +spanLedger verifies *its own* sequence-numbered probes. Claude Code's telemetry is +**user traffic** — the thing that makes the demo real, not the thing being counted. +The honest claim is: + +> "Probes riding the same pipeline, injected at the same entry point, during the same +> window, lost 3.8%. Therefore approximately N of your Claude Code spans were lost." + +The codebase already encodes this correctly: `extrapolated_user_spans_lost` in +[events.py](../../spanledger/events.py:132) and `confidence: "probe_verified"` in the +loss payload. **That field is the bridge between the probe model and the user-facing +claim, and it is the most important thing in the demo.** Every piece of copy — MCP +response, UI, narration — must preserve the distinction between *verified* (probes) +and *extrapolated* (user spans). Blur it once and the never-over-report invariant +becomes marketing. + +--- + +## Part 1 — Real Telemetry Integration + +### Routing strategy: Claude Code becomes a second traffic source at the existing entry + +No parallel pipeline. Claude Code exports to the **same OTLP entry point the probes +use** — `otel-agent:4317`, published on the host as `:14317` +([demo/compose.yaml](../../demo/compose.yaml)). Probes and Claude Code telemetry then +travel byte-for-byte the same path: agent → toxiproxy → gateway → SigNoz. + +``` +Claude Code (host) ─┐ + ├─▶ otel-agent :14317 ─▶ toxiproxy :24317 ─▶ otel-gateway ─▶ SigNoz +spanLedger probes ──┘ (fault injection point) ▲ + │ +spanLedger verify (out-of-band, direct to SigNoz Query API) ──────────────────────┘ +``` + +**Why this is correct and not merely convenient:** the audit claim is only valid if +probes share the failure domain with the traffic they speak for. Same receiver, same +batch processor, same sending queue, same exporter. If Claude Code had its own +collector, the probes would be auditing a *different* queue and the extrapolation would +be a lie. **Co-location at the entry point is a correctness requirement.** + +### Environment variables + +Set in the shell that launches the demo Claude Code session: + +```bash +export CLAUDE_CODE_ENABLE_TELEMETRY=1 +export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 # required for traces +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:14317 +export OTEL_TRACES_EXPORTER=otlp +export OTEL_LOGS_EXPORTER=otlp +export OTEL_METRICS_EXPORTER=otlp +export OTEL_METRIC_EXPORT_INTERVAL=10000 # 10s, not the 60s default +export OTEL_RESOURCE_ATTRIBUTES=service.name=claude-code-demo +``` + +**Two non-obvious settings, both load-bearing:** + +- **`OTEL_METRIC_EXPORT_INTERVAL=10000`.** The default is 60s. In a 3-minute demo + that's three data points — the metrics signal would appear dead. 10s makes the + metrics leg visible. Documented as a debugging value; restore defaults outside demos. +- **`CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`.** Without it there are no spans, and the + traces leg — the one the demo is built on — carries no user traffic at all. + +**Privacy: leave these OFF.** `OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_ASSISTANT_RESPONSES`, +`OTEL_LOG_TOOL_CONTENT`, `OTEL_LOG_RAW_API_BODIES` are off by default — keep them off. +Standard attributes still include `user.email` and `organization.id`, so **use a +throwaway account for any recording.** Add this as a checklist item in +[demo/walkthrough.md](../../demo/walkthrough.md), not as tribal knowledge. + +### Collector configuration + +[demo/otel/agent.yaml](../../demo/otel/agent.yaml) already has `traces` and `logs` +pipelines. Two edits: + +1. **Add a `metrics` pipeline** — same `otlp` receiver, `batch` processor, `otlp` + exporter. Currently absent; Claude Code's metrics would be silently refused at the + receiver. Mirror in [gateway.yaml](../../demo/otel/gateway.yaml). +2. **Tighten the sending queue** so overflow is reachable in demo time (Part 2). + +No new services, no new ports, no routing rules. Probes and user traffic are +distinguished downstream by the `spanledger.*` resource attributes +([probe.py](../../spanledger/probe.py:29)) — the collector treats both as ordinary OTLP. + +### Expected telemetry + +| Signal | Claude Code emits | Interval | Role in demo | +|---|---|---|---| +| **Traces** | `claude_code.interaction` → `llm_request` / `tool` / `hook` | 5s | **Primary.** The spans a developer would actually debug. | +| **Logs/events** | `user_prompt`, `tool_result`, `api_request`, `api_error` | 5s | Secondary. Proves multi-signal; exercises the unverified logs leg (spike S6). | +| **Metrics** | `session.count`, `token.usage`, `cost.usage`, `lines_of_code.count` | 10s (set) | Tertiary. Nice-to-have; do not build the demo on it. | + +**Architectural impact: none.** No changes to `spanledger/`. This is environment +variables plus one collector pipeline. Keep [demo/agent-app](../../demo/agent-app) — +Claude Code gives authenticity, the synthetic generator gives volume and determinism. +A benchmark needs both. + +--- + +## Part 2 — Failure Injection + +### The recommendation: **queue overflow via sustained bandwidth throttle** — `demo/chaos/overflow.sh`, already written. + +### Why this one, and why the others are wrong for this demo + +The brief's own constraints eliminate every alternative: + +| Requirement | overflow | outage.sh | sigterm.sh | oomcrash.sh | +|---|---|---|---|---| +| Nothing crashes | ✅ | ✅ | ❌ collector dies | ❌ collector dies | +| Claude keeps coding | ✅ | ✅ | ✅ | ✅ | +| **SigNoz still receives telemetry** | ✅ | ❌ **all stops** | partial | partial | +| Only *part* silently lost | ✅ | ❌ total | ✅ | ✅ | +| Deterministic | ✅ | ✅ | ⚠️ timing-sensitive | ⚠️ memory-sensitive | + +**Only queue overflow satisfies all five.** A backend outage makes the dashboard go +flat — everyone sees it, and *you don't need spanLedger to notice*. It proves the least +while looking the most dramatic. Overflow is the opposite: the graph dips slightly and +looks like less traffic. That gap between "looks fine" and "is broken" **is the +product**, and it is the most common silent failure in production — the entire +`otelcol_exporter_enqueue_failed_spans` monitoring practice exists because of it. + +### How to inject it + +`overflow.sh` posts a toxiproxy `bandwidth` toxic (`rate: 8` KB/s) on the `gateway` +proxy for N seconds. Upstream of the throttle, the agent's sending queue fills; once +full, new spans are dropped with no error surfaced to the emitter. + +**One tuning change required.** Default collector queue sizes are large enough that a +90-second window may not reach overflow. Set explicitly in `agent.yaml`: + +```yaml +exporters: + otlp: + sending_queue: { enabled: true, queue_size: 64 } + retry_on_failure: { enabled: true, max_elapsed_time: 30s } +``` + +This makes overflow reachable in ~30–45s and — critically — makes the drop +**deterministic in effect**: with a bounded queue and a bounded retry ceiling, loss is +guaranteed, not probabilistic. Rehearse to pin the exact throttle duration. + +**Honesty note to state on stage:** the *exact* sequence numbers lost are not +predetermined. Guaranteed is *that* loss occurs and *that* spanLedger reports the exact +gap it observed. Claiming per-seq determinism would be a lie the invariant forbids. + +### What each party sees + +**spanLedger detects:** delivery ratio drops below target → `loss` event with +contiguous `gap_runs` (`seq_from`/`seq_to` + timestamps), `gap_shape`, `confidence: +probe_verified`, `extrapolated_user_spans_lost`, and `traces_filter`; burn rate crosses +`FAST_BURN` (14.4); error budget drains; on removal, a `recovery` event with +`links: [loss_id]`. + +**SigNoz displays:** `claude_code.*` spans still arriving (nothing looks broken); a +modest dip in span rate; the spanLedger dashboards showing SLI below target and budget +draining; the fast-burn alert pair firing. The Traces Explorer, filtered by +`traces_filter`, returns **nothing** where probes should be — *the platform confirms the +absence*. + +**The audience observes:** an app that works, an agent that keeps coding, a backend +still receiving data, dashboards that look basically fine — and one system saying *"3.8% +of the last four minutes never arrived, here is the exact range."* + +--- + +## Part 3 — MCP Trust Layer + +### Minimum viable surface: **one tool.** + +`spanledger_check_telemetry_trust(from, to, stream?, signal?)` + +The research doc proposed six. For the hackathon, **five are deferred.** Justification: + +- `pipeline_status`, `telemetry_slo`, `active_incidents`, `incident_history` are all + *informational* — they tell an agent things. Only the trust check **changes what the + agent does**. A tool that alters behavior is a product; tools that return data are an + API with extra steps. +- `diagnose_pipeline`'s value (probable cause, evidence, deep link) is **collapsed into + the single tool's response**. Splitting it forces a second round-trip and creates a + path where an agent diagnoses without first checking trust — exactly the failure the + product exists to prevent. +- One tool is impossible to misuse and trivial to explain. Six invites the judge + question *"which one does Claude actually call?"* — a question with no good answer + under demo pressure. + +**One tool, one behavior change, one sentence to explain it.** That is the strongest +possible version of this idea. + +### Request schema + +```jsonc +{ + "from": "2026-07-24T14:30:00Z", // required, RFC3339 + "to": "2026-07-24T14:40:00Z", // required, RFC3339 + "stream": "gateway-a", // optional; omitted = all streams + "signal": "traces" // optional enum: traces|logs|metrics +} +``` + +Enums stay enums, timestamps are validated, and validation **reuses +[httpapi.py](../../spanledger/httpapi.py)'s existing helpers** rather than +reimplementing them. One source of truth applies to validation too. + +### Response schema + +```jsonc +{ + "trustworthy": false, + "confidence": "probe_verified", + "verified_ratio": 0.962, + "unknown_ratio": 0.004, + "window": { "from": "...", "to": "..." }, + "affected_window": { "from": "2026-07-24T14:32:10Z", "to": "2026-07-24T14:36:02Z" }, + "incidents": [{ + "id": "0191...", "class": "loss", "stream": "gateway-a", "signal": "traces", + "gap_runs": [{ "seq_from": 40, "seq_to": 59, "t_from": "...", "t_to": "..." }], + "gap_shape": "contiguous", + "probes_lost": 20, + "extrapolated_user_spans_lost": 1840, + "probable_cause": { + "hypothesis": "exporter queue overflow", + "evidence": ["contiguous gap", "no deploy marker in window", + "otelcol_exporter_enqueue_failed_spans +1840"], + "confidence": "medium" + } + }], + "evidence": { + "traces_filter": "spanledger.stream = 'gateway-a' AND spanledger.seq >= 40 AND spanledger.seq <= 59", + "signoz_url": "http://localhost:8080/traces-explorer?filter=...", + "spanledger_url": "http://localhost:5173/incident/0191..." + }, + "recommendation": "Telemetry for this window is incomplete. ~1840 user spans were probably lost between 14:32:10 and 14:36:02. Conclusions drawn from this window may be unreliable. Verify against the evidence link before diagnosing application behavior." +} +``` + +**Design decisions worth defending:** + +- **`verified_ratio` and `extrapolated_user_spans_lost` are separate fields with + different names.** One is counted, one is estimated. Never merge them. +- **`unknown_ratio` is always present**, including when trustworthy. Publishing + uncertainty is the differentiator; hiding it when convenient forfeits it. +- **`probable_cause.confidence` is explicit and may be `"low"`.** The tool returns + hypotheses with evidence, never an asserted root cause. An auditor that guesses has + stopped being an auditor. +- **`recommendation` is prose.** The consumer is a language model; a natural-language + instruction is far more reliably honored than a boolean it must interpret. +- **`trustworthy: true` still returns evidence links.** Trust should be checkable. + +### Interaction flow + +``` +Developer: "Why are my traces missing?" + │ + ├─▶ Claude calls spanledger_check_telemetry_trust(last 30m) ← FIRST + │ → trustworthy: false, verified_ratio 0.962, incident, evidence + │ + ├─▶ Claude states the trust verdict BEFORE any diagnosis + │ + └─▶ Only then: signoz MCP query, scoped to the trustworthy portion, + or the evidence link offered for inspection +``` + +### Making Claude call it first — the real engineering problem + +**This is the highest-risk element of the entire demo, and it is not a code problem.** +Tool invocation is model-driven; nothing in MCP guarantees Claude calls your tool at +the right moment. Three mitigations, in order of reliability: + +1. **A `CLAUDE.md` in the demo repo** stating: *"Before answering any question about + missing/incomplete/suspicious telemetry, traces, logs, or metrics, call + `spanledger_check_telemetry_trust` first and report its verdict before diagnosing."* + Project instructions are the strongest available lever. +2. **A tool description written as a precondition**, not a capability: *"Call this + BEFORE analyzing any telemetry data to determine whether that telemetry is complete + enough to reason about. Conclusions drawn from unverified telemetry may be wrong."* +3. **Rehearsed phrasing.** "Why are my traces missing?" reliably triggers it. Have a + fallback line ready that names the tool explicitly, and **never present it as + automatic if it isn't** — a judge who spots the gap will discount everything else. + +### Architecture + +Separate process, `spanledger mcp`, stdio transport, **read-only, zero write tools**. +Reads exclusively through the existing `/api/v2/*` surface — computes nothing, stores +nothing. If a field isn't available, that's a signal to extend the API deliberately, +not to let the MCP layer derive its own answers. + +Portable unchanged to Cursor, Windsurf, Codex, and VS Code agents — it's a standard +stdio MCP server. Zero extra cost, real adoption argument; ship a config snippet for +Claude Code and Cursor. + +--- + +## Part 4 — SigNoz Integration + +**Principle: spanLedger renders verdicts. SigNoz renders telemetry. Every path from a +verdict terminates in SigNoz.** + +**1. Fix the deep link first — it is a demo dependency, not polish.** +[signoz-links.ts](../../frontend/src/lib/signoz-links.ts) carries an explicit warning +that the URL format is unverified, with copy-to-clipboard as the documented fallback. +The demo's "View Evidence" moment *is* this link. Verify it against the live instance +in milestone M1; if the format is wrong, fix it or fall back deliberately — do not +discover this on stage. + +**2. "View Evidence" is the seam.** One button on an incident, carrying `traces_filter` +straight into Traces Explorer, time-bounded to `affected_window`. The developer lands on +a query returning nothing where probes should be. **The platform confirms the absence** — +far more persuasive than spanLedger asserting it. + +**3. Trust card as a dashboard panel, not a new dashboard.** Add one panel to the +existing SLO dashboard: trust verdict, verified ratio, unknown ratio, latest incident. +A new dashboard reads as a competing product; a panel reads as a trust layer. + +**4. Findings already terminate in SigNoz** via the Reporter's out-of-band metrics — +unchanged, and the reason the alert pair fires from SigNoz rather than from us. + +**5. Keep the incident page minimal.** Verdict, gap runs, extrapolation, evidence +button. Any analysis surface beyond that is rebuilding SigNoz badly. + +--- + +## Part 5 — Notification Layer + +### Recommendation: **build nothing. Route through SigNoz's existing notification channels.** + +SigNoz already has alert rules and notification channels (Slack, email, webhook), and +we already ship 8 alert rules that fire from platform metrics. Building a second +notifier would mean: + +- a **second source of truth** for "is something wrong" — direct philosophy violation; +- competing with the platform we claim to extend; +- owning delivery, retries, dedup, and silencing — all solved, none of it our problem. + +**The correct move is to make our alert rules good enough that SigNoz's notifications +are sufficient.** They already are. This is the "extend, don't compete" principle +applied to ourselves rather than only to competitors. + +### If a native notifier is built anyway (post-hackathon) + +Exactly **two** events justify a push, both incident-lifecycle, both already deduped: + +| Event | Justification | +|---|---| +| **New verified loss incident** | State change from trusted → untrusted. Actionable. Already dedup'd per open incident. | +| **Recovery** | Closes the loop, carries `links: [loss_id]`. Prevents stale-alarm fatigue. | + +**Explicitly not:** `budget_warning` (threshold noise — that's what dashboards are for), +`burn_rate_high` (already a SigNoz alert; duplicating it is the definition of spam), +per-probe failures. Channel priority: **Slack/Discord webhook** (one integration, both +platforms) → **GitHub PR comment** for the CI trust gate, which is genuinely valuable +because it puts the verdict where the decision is made. Telegram/email add nothing. + +--- + +## Part 6 — Implementation Roadmap + +Five milestones. Each ships something demoable and retires a specific risk. + +### M1 — Real telemetry through the pipeline +- **Objective.** Claude Code telemetry flows through the audited pipeline, visible in + SigNoz, with probes auditing the same path. +- **Files.** `demo/otel/agent.yaml`, `demo/otel/gateway.yaml` (add `metrics` pipeline; + set `sending_queue`), `demo/README.md`, `demo/claude-code-env.sh` (new), + `frontend/src/lib/signoz-links.ts` (verify URL format). +- **Effort.** ~4h. **Dependencies.** Foundry migration Phase 2 (a running SigNoz). +- **Risks.** Traces beta shape; metrics pipeline rejected by the pinned collector + version; deep-link format wrong. +- **Acceptance.** `claude_code.interaction` spans queryable in SigNoz; all three + signals present; probes still verify at ~100%; deep link opens a correct filtered view. + +### M2 — Deterministic overflow scenario +- **Objective.** A rehearsed, reproducible silent-loss window. +- **Files.** `demo/chaos/overflow.sh` (parameterize), `demo/chaos/run_scenarios.py`, + `demo/walkthrough.md`. +- **Effort.** ~4h, mostly rehearsal. **Dependencies.** M1. +- **Risks.** Loss doesn't trigger inside the window (queue too large); *too much* loss, + making it look like an outage and destroying the "silent" framing. +- **Acceptance.** Three consecutive runs each produce a `loss` event with a contiguous + gap within 90s, Claude Code never errors, SigNoz keeps receiving spans throughout. + +### M3 — MCP server, one tool ← **the product** +- **Objective.** `spanledger_check_telemetry_trust` answering from real incidents. +- **Files.** `spanledger/mcp.py` (new), `spanledger/cli.py` (`mcp` command), + `test/test_mcp.py` (new), `.mcp.json` / config snippets, demo repo `CLAUDE.md`. +- **Effort.** ~10h — the largest item, and still small because it computes nothing new. +- **Dependencies.** M1, M2 (needs real incidents to answer about). +- **Risks.** **Claude doesn't call the tool on cue** (highest risk in the project — + three mitigations in Part 3); response too verbose for the model to use well; + temptation to add tools. +- **Acceptance.** Claude Code, asked "why are my traces missing?", calls the tool, + states the verdict *before* diagnosing, and offers the evidence link. Reproducible + three times running. MCP process killed → daemon and detection unaffected. + +### M4 — Evidence seam +- **Objective.** One-click spanLedger incident → filtered SigNoz view. +- **Files.** `frontend/src/pages/incident/Page.tsx`, `signoz-links.ts`, + `assets/dashboards/slo-error-budget.json` (trust panel). +- **Effort.** ~5h. **Dependencies.** M1 (verified URL format). +- **Risks.** Time-range params differ from the assumed format; panel schema rejected on + import — a known open item in [PENDING_WORK.md](PENDING_WORK.md). +- **Acceptance.** Evidence button lands on a Traces Explorer view scoped to the + incident's filter *and* window, showing the absence. + +### M5 — Demo hardening +- **Objective.** Every on-screen number script-reproducible; the run rehearsed end-to-end. +- **Files.** `demo/walkthrough.md`, `demo/chaos/run_scenarios.py`, `README.md`. +- **Effort.** ~6h. **Dependencies.** M1–M4. +- **Risks.** Live coding session is inherently variable; PII leakage in a recording. +- **Acceptance.** Full run completes in under 3 minutes, twice consecutively, from a + clean start, with a throwaway account and content-logging flags off. + +**Total ≈ 29h.** Critical path M1 → M2 → M3. M4 is parallelizable; M5 needs everything. + +--- + +## Part 7 — Challenging This Design + +Reviewing this as if another team proposed it. + +### Risky assumptions + +**1. That Claude Code will call the MCP tool on cue.** The single biggest risk, and +it's behavioral, not technical. Every mitigation is probabilistic. **If it fails live, +the demo's centerpiece fails.** Rehearse to the point of boredom and keep an explicit +fallback phrasing — but never claim automatic invocation you can't reproduce. + +**2. That a live Claude Code session is demo-safe.** It's a real agent doing real work +in front of judges. It might stall on an API error, take an unexpected path, or produce +an awkward answer. **Mitigation: pre-record the coding session as a fallback and be +transparent that it's a recording.** A recorded session with live spanLedger is far +better than a live session that hangs. + +**3. That queue overflow triggers reliably in a 90-second window.** Depends on queue +size, throughput, and throttle rate interacting. Tuning is required and it is finicky. +**Mitigation: pin the values in M2 and never change them afterward.** + +**4. That Claude Code's beta traces stay stable.** Beta means the span shape can change +between versions. **Mitigation: pin the Claude Code version used for the demo and +record it in DECISIONS.md.** + +### Over-engineered + +- **Six MCP tools.** Already cut to one. Resist re-adding under "it's only 20 more + lines." +- **The metrics leg for the demo.** Three signals is architecturally satisfying and + demo-irrelevant. Traces carry the story. Keep metrics wired, don't spend narration on it. +- **`probable_cause` correlation.** The `correlation_hint` path requires scraping + collector self-metrics ([docs/setup-collector-metrics.md](../setup-collector-metrics.md)), + is opt-in and off by default, and has never been exercised end-to-end. **Ship + `probable_cause` with the cheap signals only** — gap shape, deploy-marker proximity, + per-signal divergence — and add the collector-metric evidence only if M1–M4 land early. +- **A native notification layer.** Cut entirely in Part 5. + +### Postpone + +Multi-backend verification, the CI trust gate + GitHub Action (great product, weak +demo), the auditor-quality benchmark (Direction 3 — publish detection latency and FP/FN +*after* the hackathon, with enough runs to be honest), the remaining five MCP tools, +and trust attestation. + +### What could confuse judges + +**1. "Isn't this just monitoring the collector's own metrics?"** The most likely +skeptical question and it must be answered in one sentence, on stage, unprompted: +*"Collector self-metrics report only the loss the collector counted. Silent drops are +by definition uncounted — and self-reported metrics travel the same failing pipeline."* + +**2. Probe-vs-user-span conflation.** If narration says "spanLedger detected your +Claude Code spans were lost," a sharp judge will ask how, and the honest answer — +extrapolation — will sound like backpedaling. **Say "extrapolated" first, proactively.** + +**3. Two MCP servers in one demo** (SigNoz's and spanLedger's) is genuinely confusing. +Draw the boundary explicitly: *"SigNoz's MCP answers what's in the data. Ours answers +whether the data can be trusted."* + +**4. "Why not just alert on span-rate drop?"** Answer: a rate drop can't distinguish +less traffic from lost traffic. spanLedger always can, because it knows what it sent. + +### What I'd simplify + +- **Cut the metrics signal from the narration.** Wire it; don't talk about it. +- **Cut the trust dashboard panel (M4) if time is short.** The MCP response *is* the + product. The panel is a nice-to-have that competes with itself. +- **One stream in the demo, not two.** `demo/spanledger.demo.yaml` configures + `gateway-a` and `gateway-b`. Multi-stream is an architectural strength and a + narrative tax at 3 minutes. Use one; mention multi-stream verbally. +- **Don't demo recovery in detail.** Show the `recovery` event appearing; don't spend + 20 seconds on hysteresis and linking. + +--- + +## If I had only 48 hours before the hackathon, this is exactly what I would build. + +**Hour 0–4 — M1: real telemetry.** Claude Code's telemetry through the existing +pipeline. Four env vars, one collector pipeline, verify the deep-link format. *Cheapest +credibility in the project: it converts "synthetic demo" into "real workload" for +half a day's work, and until it's done nothing else can be demonstrated on real data.* + +**Hour 4–8 — M2: pin the overflow scenario.** Tune `queue_size` and throttle duration +until three consecutive runs each produce a contiguous gap inside 90 seconds. *This is +the demo's physics. If loss isn't reliable, nothing downstream matters — and this is +tuning work that cannot be compressed later.* + +**Hour 8–20 — M3: the single MCP tool.** `spanledger_check_telemetry_trust`, read-only, +stdio, answering from `/api/v2/*`. Response carries verdict, verified ratio, +extrapolation, incident, evidence links, and a prose recommendation. Plus the demo +repo's `CLAUDE.md` precondition and a precondition-shaped tool description. *This is +the product. Everything before it is setup; everything after it is polish.* + +**Hour 20–28 — M4 (evidence seam only).** The "View Evidence" button into a filtered, +time-bounded Traces Explorer. Skip the dashboard panel. *This is the moment the trust +layer visibly composes with the platform instead of replacing it — the clearest +possible demonstration of "extend, don't compete."* + +**Hour 28–40 — M5: rehearse until boring.** Full run, clean start, twice consecutively, +under 3 minutes. Record a fallback coding session. Throwaway account, content-logging +off. *Demos fail on rehearsal, not on code.* + +**Hour 40–48 — buffer.** Do not fill it. It absorbs the two failures that are actually +likely: the deep-link format being wrong, and Claude not calling the tool reliably. If +nothing breaks, spend it on the `probable_cause` evidence fields — the one deferred item +that would visibly strengthen the story. + +### Why this allocation and not another + +**It front-loads the two things that can't be faked and back-loads everything that +can.** Real telemetry (M1) and reliable loss (M2) are physical facts about the system — +if they don't work, no amount of UI recovers the demo. They also happen to be the +cheapest items, so there is no excuse for deferring them. + +**It spends the largest single block on the one thing that is genuinely new.** M1, M2, +M4, and M5 all make an *existing* story better. M3 is the only item that creates a +workflow that does not currently exist — a developer asking about telemetry and being +told, before any analysis, that the telemetry itself cannot be trusted. That is the +hackathon thesis, and it deserves the most hours. + +**It cuts everything that competes with itself.** No notification layer (SigNoz has +one). No new dashboard (a panel would do, and even that is cut first). No extra MCP +tools. No multi-stream narration. Every cut removes a thing a judge could ask a +confusing question about. + +**It reserves 8 hours for the two failures I actually expect.** Most plans fail because +they're scheduled to the last hour and the first surprise cascades. The deep-link +format is unverified in the code's own comments, and MCP invocation is model-driven and +non-deterministic. Both are known unknowns; both get budget. + +**What I would sacrifice first if I lost a day:** M4's evidence seam — the MCP response +carries the `signoz_url` regardless, so a judge can be walked through it manually. What +I would **not** sacrifice under any circumstances: M2's rehearsal and M5's end-to-end +runs. A polished feature set that fails live scores lower than a smaller demo that +works twice in a row. From 9fb20b696fac8d972618bf98af92e8863a66ea3a Mon Sep 17 00:00:00 2001 From: Himanshu Date: Fri, 24 Jul 2026 13:59:21 +0530 Subject: [PATCH 03/36] docs: implementation plan v2 --- docs/v2/IMPL_TRUST_LAYER.md | 694 +++++++++++++++++++++++++++++++++--- 1 file changed, 639 insertions(+), 55 deletions(-) diff --git a/docs/v2/IMPL_TRUST_LAYER.md b/docs/v2/IMPL_TRUST_LAYER.md index d202593..80dfe2a 100644 --- a/docs/v2/IMPL_TRUST_LAYER.md +++ b/docs/v2/IMPL_TRUST_LAYER.md @@ -87,17 +87,80 @@ throwaway account for any recording.** Add this as a checklist item in ### Collector configuration [demo/otel/agent.yaml](../../demo/otel/agent.yaml) already has `traces` and `logs` -pipelines. Two edits: +pipelines. Two edits, both **frozen below — copy verbatim**. -1. **Add a `metrics` pipeline** — same `otlp` receiver, `batch` processor, `otlp` - exporter. Currently absent; Claude Code's metrics would be silently refused at the - receiver. Mirror in [gateway.yaml](../../demo/otel/gateway.yaml). -2. **Tighten the sending queue** so overflow is reachable in demo time (Part 2). +**Edit 1 — `demo/otel/agent.yaml`.** Replace the whole `exporters:` block and append a +`metrics` pipeline. Final file content: + +```yaml +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + +processors: + batch: {} + +exporters: + otlp: + endpoint: toxiproxy:24317 # gateway via toxiproxy (fault injection point) + tls: + insecure: true + # Bounded queue + bounded retry ceiling: makes overflow reachable and loss + # guaranteed (not probabilistic) under the Part 2 bandwidth throttle. + sending_queue: + enabled: true + queue_size: 64 + retry_on_failure: + enabled: true + max_elapsed_time: 30s + +service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlp] + logs: + receivers: [otlp] + processors: [batch] + exporters: [otlp] + metrics: + receivers: [otlp] + processors: [batch] + exporters: [otlp] +``` + +**Edit 2 — `demo/otel/gateway.yaml`.** Append the `metrics` pipeline only. **Do not add +`sending_queue`/`retry_on_failure` here** — the throttle sits between agent and gateway, +so the agent's queue is the one that must overflow. Add exactly: + +```yaml + metrics: + receivers: [otlp] + processors: [batch] + exporters: [otlp] +``` + +**Do not modify** `demo/otel/gateway-logs-down.yaml` (Phase 3 fault config, out of scope +for this plan). No new services, no new ports, no routing rules. Probes and user traffic are distinguished downstream by the `spanledger.*` resource attributes ([probe.py](../../spanledger/probe.py:29)) — the collector treats both as ordinary OTLP. +### Execution order for Part 1 + +1. Edit `demo/otel/agent.yaml` (both blocks above). +2. Edit `demo/otel/gateway.yaml` (metrics pipeline only). +3. Create `demo/claude-code-env.sh` with the exact variable block above, prefixed with + `#!/usr/bin/env sh` and a comment naming the privacy flags that must stay unset. +4. Restart the demo stack; confirm the collectors accept the config (no restart loop). +5. Verify signals in SigNoz (M1 acceptance checklist). +6. Update `demo/README.md` — add a "Real telemetry (Claude Code)" section pointing at + `demo/claude-code-env.sh`. + ### Expected telemetry | Signal | Claude Code emits | Interval | Role in demo | @@ -142,19 +205,28 @@ product**, and it is the most common silent failure in production — the entire proxy for N seconds. Upstream of the throttle, the agent's sending queue fills; once full, new spans are dropped with no error surfaced to the emitter. -**One tuning change required.** Default collector queue sizes are large enough that a -90-second window may not reach overflow. Set explicitly in `agent.yaml`: - -```yaml -exporters: - otlp: - sending_queue: { enabled: true, queue_size: 64 } - retry_on_failure: { enabled: true, max_elapsed_time: 30s } -``` - +**Tuning is already frozen in Part 1** (`queue_size: 64`, `max_elapsed_time: 30s`). This makes overflow reachable in ~30–45s and — critically — makes the drop **deterministic in effect**: with a bounded queue and a bounded retry ceiling, loss is -guaranteed, not probabilistic. Rehearse to pin the exact throttle duration. +guaranteed, not probabilistic. + +**Frozen scenario parameters.** `demo/chaos/overflow.sh` currently hard-codes +`rate: 8` and takes duration as `$1` defaulting to `120`. Change **only** the default +duration and add a rate parameter; do not alter the toxic type or the proxy name: + +| Parameter | Frozen value | Where | +|---|---|---| +| Toxic type | `bandwidth` | `overflow.sh` (unchanged) | +| Proxy name | `gateway` | `demo/toxiproxy.json` (unchanged) | +| `rate` | `8` (KB/s) — `$2`, default `8` | `overflow.sh` | +| Duration | `90` (seconds) — `$1`, default `90` | `overflow.sh` (was `120`) | +| Agent `queue_size` | `64` | `demo/otel/agent.yaml` | +| Agent `max_elapsed_time` | `30s` | `demo/otel/agent.yaml` | +| Probe interval | `1s` (traces) | `demo/spanledger.demo.yaml` (unchanged) | + +**These seven values are the demo's physics. Once M2 acceptance passes, they are frozen +— do not tune them again for any later milestone.** Record the pinned set in +`DECISIONS.md` at the end of M2. **Honesty note to state on stage:** the *exact* sequence numbers lost are not predetermined. Guaranteed is *that* loss occurs and *that* spanLedger reports the exact @@ -203,20 +275,49 @@ The research doc proposed six. For the hackathon, **five are deferred.** Justifi **One tool, one behavior change, one sentence to explain it.** That is the strongest possible version of this idea. -### Request schema +### Where the answer is computed — frozen -```jsonc -{ - "from": "2026-07-24T14:30:00Z", // required, RFC3339 - "to": "2026-07-24T14:40:00Z", // required, RFC3339 - "stream": "gateway-a", // optional; omitted = all streams - "signal": "traces" // optional enum: traces|logs|metrics -} -``` +The plan's own rule: *"If a field isn't available, that's a signal to extend the API +deliberately, not to let the MCP layer derive its own answers."* The trust verdict is +such a field. Therefore: + +- **`GET /api/v2/trust` is added to [httpapi.py](../../spanledger/httpapi.py).** It owns + the entire derivation. +- **`spanledger/mcp.py` is a pure passthrough.** It validates input, calls + `/api/v2/trust`, and returns the body unchanged. **It contains no arithmetic.** If an + implementer finds themselves computing a ratio in `mcp.py`, the code is in the wrong + file. + +This keeps one source of truth, makes the logic unit-testable without an MCP client, and +means the CLI/frontend can consume the identical verdict later at zero cost. + +### Request schema — FROZEN + +Both surfaces take the same four parameters. MCP tool input schema (JSON Schema draft +2020-12) and the `/api/v2/trust` query string are identical in name, type, and rules. -Enums stay enums, timestamps are validated, and validation **reuses -[httpapi.py](../../spanledger/httpapi.py)'s existing helpers** rather than -reimplementing them. One source of truth applies to validation too. +| Field | Type | Required | Default | Validation | On violation | +|---|---|---|---|---|---| +| `from` | string, RFC3339 | **yes** | — | Parseable by `httpapi._parse_at`; must be `< to` | 400 `invalid from` | +| `to` | string, RFC3339 | **yes** | — | Parseable by `httpapi._parse_at`; must be `> from` | 400 `invalid to` | +| `stream` | string | no | `null` (= all streams) | Must be in `slo_engine.stream_names()` | 400 `unknown stream` | +| `signal` | enum `traces`\|`logs`\|`metrics` | no | `"traces"` | `httpapi._signal_param` | 400 `invalid signal` | + +Additional frozen rules: + +- **Unknown query parameters are a 400** — reuse `httpapi._require_params` with the + allowed set `{"from", "to", "stream", "signal"}`. This matches every other `/api/v2` + route; do not relax it. +- **Window length is capped at 24h.** `to - from > 86400s` → 400 `window too large`. + Prevents an agent from requesting a 90-day scan and blocking the handler. +- **`from`/`to` are RFC3339 strings here, not the nanosecond ints** used by + `/api/v2/events`. This is deliberate: the caller is a language model, and RFC3339 is + what it produces reliably. Convert once via `_parse_at` at the handler boundary. + +Enums stay enums, timestamps are validated, and validation **reuses these exact existing +helpers in [httpapi.py](../../spanledger/httpapi.py)** rather than reimplementing them: +`_require_params`, `_single`, `_signal_param`, `_parse_at`, `_rfc3339_ns`, and the +`HttpError` class for all error paths. One source of truth applies to validation too. ### Response schema @@ -235,9 +336,9 @@ reimplementing them. One source of truth applies to validation too. "probes_lost": 20, "extrapolated_user_spans_lost": 1840, "probable_cause": { - "hypothesis": "exporter queue overflow", - "evidence": ["contiguous gap", "no deploy marker in window", - "otelcol_exporter_enqueue_failed_spans +1840"], + "hypothesis": "exporter queue overflow or backend outage", + "evidence": ["gap_shape: contiguous", + "otelcol_exporter_enqueue_failed_spans +1840 at exporter"], "confidence": "medium" } }], @@ -263,6 +364,128 @@ reimplementing them. One source of truth applies to validation too. instruction is far more reliably honored than a boolean it must interpret. - **`trustworthy: true` still returns evidence links.** Trust should be checkable. +### Response field types — FROZEN + +Every field is **always present**. Absent data is `null` or `[]`, never an omitted key — +a model reasons far more reliably over a stable shape. + +| Field | Type | Nullable | Notes | +|---|---|---|---| +| `trustworthy` | bool | no | Never `null`. See verdict table below. | +| `confidence` | enum `probe_verified`\|`unknown`\|`no_data` | no | Verification coverage, not loss severity. | +| `verified_ratio` | float, 4dp | no | `1.0` when nothing was judged. | +| `unknown_ratio` | float, 4dp | no | `0.0` when nothing was judged. | +| `window` | `{from, to}` RFC3339 strings | no | Echoes the request, normalised. | +| `affected_window` | `{from, to}` RFC3339 strings | **yes** | `null` when `incidents` is empty. | +| `incidents` | array of incident objects | no | `[]` when none. Max 50. | +| `evidence` | object (3 keys, below) | no | Always present. | +| `recommendation` | string | no | Always non-empty. Two frozen templates below. | + +Incident object (one per `loss` event, ordered oldest → newest by `emitted_at_ns`): + +| Field | Type | Source (loss payload key) | +|---|---|---| +| `id` | string | `id` | +| `class` | string, always `"loss"` | `class` | +| `stream` | string | `stream` | +| `signal` | string | `signal` | +| `gap_runs` | array of `{seq_from:int, seq_to:int, t_from:str, t_to:str}` | `gap_runs` verbatim | +| `gap_shape` | enum `contiguous`\|`striped`\|`scattered` | `gap_shape` | +| `probes_lost` | int | `probes.missing` | +| `extrapolated_user_spans_lost` | int or `null` | `extrapolated_user_spans_lost.estimate`, else `null` | +| `probable_cause` | object, below | derived — see rules | + +`evidence` object: + +| Field | Type | Nullable | Source | +|---|---|---|---| +| `traces_filter` | string | yes | `traces_filter` of the **last** incident; `null` if no incidents | +| `signoz_url` | string | yes | Built from `traces_filter`; `null` if `traces_filter` is `null` | +| `spanledger_url` | string | yes | `null` if no incidents | + +### Derivation rules — FROZEN + +Implement exactly this in the `/api/v2/trust` handler. No step is optional. + +1. Parse and validate per the request table. Convert `from`/`to` to ns via `_parse_at`. +2. `streams = [stream] if stream else slo_engine.stream_names()`. +3. `rows = store.query_events(EventFilter(stream=None, from_ns=from_ns, to_ns=to_ns, + limit=500))` — reuse `store.query_events` and `store.EventFilter` unchanged. +4. `loss = [r for r in rows if r.class_ == CLASS_LOSS and r.signal == signal and + r.stream in streams]`, preserving query order (already `emitted_at_ns ASC`). + `unreachable = [r for r in rows if r.class_ == CLASS_BACKEND_UNREACHABLE and + r.stream in streams]`. Import both constants from `events.py` — do not inline strings. +5. Sum across `loss` payloads: `sent`, `verified`, `missing`, `unknown` from + `r.payload["probes"]`. +6. `judged = verified + missing`; + `verified_ratio = round(verified / judged, 4) if judged else 1.0`; + `unknown_ratio = round(unknown / sent, 4) if sent else 0.0`. +7. **Verdict table — exhaustive, no other combination exists:** + + | Condition (evaluated top-down, first match wins) | `trustworthy` | `confidence` | + |---|---|---| + | `loss` is non-empty | `false` | `probe_verified` | + | `unreachable` is non-empty | `false` | `unknown` | + | every stream's `slo_engine.snapshot(s, signal)["sli"]` is `None` | `false` | `no_data` | + | otherwise | `true` | `probe_verified` | + +8. `affected_window` = `{from: earliest gap_run.t_from, to: latest gap_run.t_to}` across + all `loss` incidents; `null` if `loss` is empty. +9. `probable_cause` per incident, from `gap_shape` only (frozen mapping): + + | `gap_shape` | `hypothesis` | base `confidence` | + |---|---|---| + | `contiguous` | `"exporter queue overflow or backend outage"` | `medium` | + | `striped` | `"sustained overload causing periodic queue overflow"` | `medium` | + | `scattered` | `"intermittent loss, cause unclear"` | `low` | + + `evidence` array is built by appending, in this order, only the entries that apply: + `"gap_shape: "`; `"deploy marker '