Evidence-grounded incident response, from first signal to approved action.
When a production system breaks, engineers lose the expensive minutes to a research problem, not a coding one: deploy history, CI results, logs, metrics, traces, and support tickets all live in different tools, and someone has to rebuild one timeline before anyone can decide what changed. Rootline ingests those six kinds of evidence, ranks which commit most likely caused the incident, explains the ranking with citations back to specific stored events, drafts the Slack update and the postmortem, and can roll a release back — but only after a second person approves it. It is a working system rather than a mockup: the screenshots below show it running against a real Python API, a real database, and the real scoring, approval, and simulation code paths.
Built by Bruce Moseti. Python, FastAPI, SQLite, OpenAI Structured Outputs, React, TypeScript, Cloudflare Workers. A 3,000-line incident engine, a 1,300-line benchmark harness, and 47 automated tests across both languages, all run in CI on every push.
Two surfaces sit on top of that engine:
- a public incident pulse that normalizes live events from nine official provider status feeds; and
- an Operations Lab that runs the whole response loop against the Python engine.
The bundled checkout incident is clearly labeled synthetic. It exercises the real API, persistence, scoring, artifact, approval, and simulation paths without pretending to have access to a company's private systems. OpenAI, Slack, production telemetry, and production rollback activate only when the operator supplies server-side credentials and policy.
The Operations Lab, driving the local Python engine end to end. Every panel is a real API response, not a rendered mock.
Ingested evidence on the left, the ranked commit with its reasons and score on the right. Customer impact is summarized only from ingested impact events, and reported as missing when absent.
The guarded action layer. The Slack path returns a preview with zero network egress, the postmortem cites stored evidence IDs, and the rollback moves through pending_approval to approved to simulated — a separate, visible step each time.
Most portfolio projects describe a system. This one measures it. The deterministic commit ranker is scored against 240 labeled synthetic incidents, plus 40 where the correct answer is to blame nothing at all, and compared against the heuristic on-call engineers reach for under pressure: suspect the newest release.
| Strategy | Top-1 accuracy | Hit@3 | MRR |
|---|---|---|---|
| Rootline deterministic correlation | 70.0% | 100.0% | 0.850 |
| Baseline: newest release in the affected service | 47.5% | 100.0% | 0.738 |
| Baseline: newest release anywhere | 35.0% | 91.7% | 0.644 |
- It does not page the wrong team. On the 40 incidents caused upstream, the engine returns an empty suspect list every time. The newest-release-anywhere heuristic blames an unrelated team's commit in all 40.
- It says which integrations earn their plumbing. Withholding CI evidence costs 11.2 points of top-1 accuracy. Withholding logs, metrics, and traces costs nothing measurable: they corroborate a hypothesis for the human reading it without changing the ordering. That is an integration-priority answer from an experiment, not a preference.
- It reports where it fails. Slow-burn regressions score 0.0%, in the same table
as the headline. Score calibration is published for the same reason: Brier 0.262 and
expected calibration error 0.357 against a 46.8% base rate, with non-monotonic
reliability buckets. A
0.93is a ranking position, not a 93% chance of causation.
npm run engine:bench # regenerate the report; CI fails if published numbers drift
Full method, per-family results, calibration table, seed-stability sweep, and limits are in the correlation benchmark. The scenarios are generated and labeled as synthetic, so these figures show the ranker applying its intended evidence semantics under confounders — not accuracy on real production incidents. Where Rootline creates value separates what is measured from what is argued, and states what is not claimed.
| If you evaluate | Start here | What it demonstrates |
|---|---|---|
| Quantitative / research engineering | backend/bench/ · benchmark report |
Labeled dataset with adversarial families, top-1 / hit@3 / MRR / score-margin metrics, evidence ablations, Brier score and expected calibration error, seed-stability sweep |
| AI / ML engineering | backend/app/services/ai.py · citation tests |
Structured Outputs with Pydantic validation, mandatory evidence citations, rejection of unknown citations, recursive secret redaction, untrusted-evidence prompt boundary |
| Backend / platform engineering | backend/app/services/rollback.py · backend/app/repository.py |
Atomic rollback state transitions, distinct approver identity, allowlisted adapters, HMAC-signed dispatch, simulation by default |
| Product / frontend engineering | app/OpsLab.tsx · app/api/incidents/route.ts |
End-to-end operator workflow, per-source timeout isolation, partial-result handling, provenance kept visible in the UI |
| Engineering leadership | Where Rootline creates value | What is measured, what is argued, what is not claimed, and how to validate the rest in production |
- Typed ingestion for deployment, CI, log, metric, trace, and customer-impact events.
- SQLite persistence with stable evidence IDs and an inspectable incident audit trail.
- Deterministic suspect-commit ranking based on deployment timing, service match, CI outcome, and corroborating telemetry.
- Explicit correlation language: a high score is a hypothesis, never proof of causation.
- OpenAI Responses API integration using Structured Outputs and Pydantic validation.
- Mandatory evidence citations for every model finding and recommended action; unknown citations are rejected.
- Recursive secret redaction, bounded model payloads, and an untrusted-evidence prompt boundary.
- Search over versioned Markdown runbooks.
- Redacted Slack Incoming Webhook summaries, with a truthful preview when no webhook is configured.
- Evidence-linked Markdown postmortems that require human review.
- Rollback planning through a predefined GitHub Actions adapter or allowlisted, HMAC-signed HTTPS webhook.
- Exact rollback-ID approval, persisted approver identity, atomic execution state, and simulation by default.
- No endpoint for arbitrary shell execution and no conversion of model text into executable commands.
- Nine official feeds: GitHub, Cloudflare, OpenAI, Discord, Reddit, Datadog US1, DigitalOcean, Dropbox, and Coinbase.
- Canonical provider incident model, active-first prioritization, search, filters, timelines, and source links.
- Per-source timeout isolation, partial results, short edge caching, and a labeled browser snapshot.
flowchart LR
subgraph Signals["Evidence sources"]
A["Public status APIs"]
B["Deploy and CI events"]
C["Logs, metrics, and traces"]
D["Customer-impact signals"]
end
subgraph Engine["Python incident engine"]
E["Typed ingestion + SQLite"]
F["Deterministic correlation"]
G["OpenAI structured analysis"]
H["Runbook retrieval"]
end
subgraph Actions["Guarded outputs"]
I["Slack summary"]
J["Markdown postmortem"]
K["Approval-gated rollback dispatch"]
end
A --> L["React incident pulse"]
B --> E
C --> E
D --> E
E --> F
F --> G
F --> H
G --> I
G --> J
F --> K
H --> K
See the incident engine guide for the full request sequence, trust boundaries, and production controls. The original public-feed reliability decisions remain documented in the architecture notes.
Node.js 22.13+ and Python 3.11+ are required.
npm install
npm run engine:setup
npm run demoOpen http://localhost:3000/operations and select Run the one-click demo.
Without credentials, the demo still runs the real Python ingestion, SQLite, correlation, runbook search, Slack preview, postmortem generation, rollback planning, approval, and simulated execution paths. The analysis panel accurately reports OpenAI as not configured rather than fabricating a model result.
To call the engine directly:
curl -sS -X POST http://127.0.0.1:8000/api/v1/demo/scenario
curl -sS http://127.0.0.1:8000/api/v1/config/statusCopy backend/.env.example into your secret manager or export the needed values in the engine process. Do not expose backend secrets through NEXT_PUBLIC_* variables.
| Capability | Required configuration |
|---|---|
| Authenticated private evidence API | Demo mode off plus distinct operator and ingestor credentials |
| OpenAI analysis | OPENAI_API_KEY, optionally OPENAI_MODEL |
| Slack publication | SLACK_WEBHOOK_URL, allowlisted Slack host |
| GitHub rollback dispatch | GitHub token, repository/workflow settings, production mode, explicit enable flag |
| Signed webhook rollback | HTTPS endpoint, hostname allowlist, bearer token and/or signing secret, production mode, explicit enable flag |
Production rollback requires all of the following: demo mode off; distinct operator, ingestor, and approver identities; ROLLBACK_MODE=production; ROLLBACK_PRODUCTION_ENABLED=true; a configured adapter; a persisted independent approval; and a dispatch request. The included GitHub workflow also uses an allowlisted service and protected GitHub environment before it reaches a deployment webhook. Adapter acceptance is recorded as dispatched, not as a completed deployment.
The public pulse reads each provider's official incident endpoint and normalizes only published fields. Provider update bodies are converted to plain text and never injected as HTML. Root cause and customer impact are not inferred from public status labels.
| Provider | Public incident endpoint |
|---|---|
| GitHub | https://www.githubstatus.com/api/v2/incidents.json |
| Cloudflare | https://www.cloudflarestatus.com/api/v2/incidents.json |
| OpenAI | https://status.openai.com/api/v2/incidents.json |
| Discord | https://discordstatus.com/api/v2/incidents.json |
https://www.redditstatus.com/api/v2/incidents.json |
|
| Datadog US1 | https://status.datadoghq.com/api/v2/incidents.json |
| DigitalOcean | https://status.digitalocean.com/api/v2/incidents.json |
| Dropbox | https://status.dropbox.com/api/v2/incidents.json |
| Coinbase | https://status.coinbase.com/api/v2/incidents.json |
The synthetic checkout fixture has its own provenance metadata and never appears as real production evidence.
npm test # web build, types, lint, and rendered-contract tests
npm run engine:test # Python unit, API contract, and benchmark-harness tests
npm run engine:bench # regenerate the correlation benchmark report
npm run engine:bench:check # fail if the published accuracy figures are staleThe Python tests mock OpenAI, Slack, GitHub, and webhook clients; the suite sends no real external requests. Production connector status is visible through the secret-free config endpoint and the Operations Lab.
CI runs all four. The benchmark job recomputes the published accuracy figures on every push, so the numbers in this README cannot silently drift from the code.
app/
api/incidents/route.ts # public provider ingestion and normalization
api/engine/[...path]/route.ts # credential-free, synthetic-only demo gateway
OpsLab.tsx # interactive end-to-end operator workflow
RootlineApp.tsx # public pulse, architecture, and case study
backend/
app/main.py # FastAPI surface and error boundaries
app/services/correlation.py # deterministic suspect-commit scoring
app/services/ai.py # cited OpenAI Structured Outputs analysis
app/services/{slack,postmortem,rollback}.py
bench/scenarios.py # labeled synthetic incidents, including adversarial families
bench/harness.py # correlation runs, naive baselines, evidence ablations
bench/metrics.py # ranking, calibration, abstention, and latency metrics
tests/ # unit, API contract, and benchmark coverage
demo/
synthetic-checkout-incident.json # labeled, versioned demo evidence
docs/
benchmarks/ # committed, CI-verified benchmark results
IMPACT.md # measured versus argued versus not claimed
runbooks/ # versioned operational procedures
.github/workflows/
rootline-rollback.yml # protected rollback dispatch workflow
- Built a Python/FastAPI incident-response engine that ingests deployment, CI, log, metric, trace, and customer-impact evidence and ranks supplied commit SHAs with a traceable heuristic.
- Integrated OpenAI Structured Outputs for evidence-cited diagnosis, with secret redaction, prompt-injection boundaries, schema validation, and rejection of unknown citations.
- Implemented runbook retrieval, redacted Slack publishing, evidence-linked Markdown postmortems, and independently approved rollback dispatch through GitHub Actions or HMAC-signed webhooks.
- Built a reproducible benchmark of 280 labeled incident scenarios showing 70.0% top-1 accuracy against 47.5% for the newest-release heuristic, quantified each telemetry integration's contribution by ablation, and published score calibration and known blind spots; CI recomputes every figure on each push.
I built Rootline as a Python and React incident-response platform. It combines deterministic evidence correlation with constrained LLM analysis: every finding cites a stored event, and model text can never directly execute a command. The response layer retrieves versioned runbooks, publishes redacted Slack updates, generates evidence-linked postmortems, and dispatches only allowlisted rollback adapters after explicit approval. I also built a benchmark for it, because a ranker nobody has measured is a guess: it scores 70% top-1 against labeled synthetic incidents versus 47.5% for the newest-release heuristic, and the report publishes the two scenario families where the heuristic still fails. The included demo is synthetic and safe; production integrations require real server-side credentials and policy.
MIT © Bruce Moseti