Clew

Follow the thread through your microservice labyrinth.

Clew is an open-source, self-hosted AI agent for full-link microservice troubleshooting. It ingests OpenTelemetry traces/logs/metrics, collapses noisy errors into incidents, and runs a Claude-powered investigation agent that follows the causal chain across services — producing a root cause with honest confidence and concrete evidence, not a wall of alerts.

Architecture

OTel SDKs ──→ otel-collector ──→ ClickHouse (telemetry data plane)
                                      │
                worker tick loop ─────┤
                  1. ingest: cursor poll → fingerprint → issues (Postgres)
                  2. intake: issue → incident (heuristic → LLM grouping)
                  3. agent runs: queued investigations → Claude Agent SDK
                                      │
            investigation agent ──────┘
              MCP tools: query_logs · get_trace · get_service_topology
                         compare_error_rate_to_baseline · submit_findings
              findings (structured) → incident patch → human review

Key design decisions (informed by studying superlog, Apache-2.0):

• Two-plane storage: ClickHouse for raw telemetry, Postgres for product state (issues, incidents, agent runs, memories).
• Fingerprinting before AI: one underlying problem = one issue = at most one investigation. Bot sweeps and id/timestamp churn never reach the agent.
• Explicit state machines: incident (open → resolved/noise/merged, reopen on regression) and agent run (queued → running ⇄ awaiting_human → completed/failed) with illegal-transition errors.
• Structured findings, enforced invariants: the agent must finish via the submit_findings tool; high confidence is rejected in code unless the session actually gathered trace + baseline evidence.
• Agent proposes, humans resolve: the only agent-initiated status change is noise auto-close. Everything else is a proposal.
• Cross-run memory: durable facts learned in one investigation are injected into the next.

Monorepo layout

Path	What
apps/worker	Tick loop: ingest cursor, incident intake, agent run scheduler
apps/api	HTTP API (Hono)
apps/sample	Demo playground: 3-service call chain with runtime fault injection
packages/db	Drizzle schema, state machines (Postgres control plane)
packages/fingerprint	Error fingerprinting / message normalization
packages/agent	Claude Agent SDK runner + MCP telemetry tools + findings schema; pluggable evidence backends (TELEMETRY_DATASTORE=clickhouse|cat)

Quick start

Prerequisites: Node 20+, pnpm 9+, Docker.

cp .env.example .env        # add your ANTHROPIC_API_KEY
set -a; source .env; set +a # apps read process.env directly; export before running
docker compose up -d        # postgres + clickhouse + otel-collector
pnpm install
pnpm --filter @clew/db db:generate && pnpm db:migrate
pnpm dev                    # api :4200, worker tick loop

Or run the whole thing — stack, clew, demo services, fault injection, incident assertion — in one shot on any Linux box with Docker:

./scripts/e2e.sh

Point any OTLP exporter at http://localhost:4318 and trigger an error; within a few ticks you'll see an incident at http://localhost:4200/incidents and an investigation run against it.

No app handy? Use the built-in demo playground — three services with fault switches (pnpm demo, then break one with a curl). See apps/sample/README.md for the full cascade scenario.

Running against Meituan CAT (no ClickHouse needed)

In CAT mode the only required store is Postgres — skip ClickHouse and the otel-collector entirely:

docker compose up -d postgres
TELEMETRY_DATASTORE=cat CAT_BASE_URL=http://cat.example.com:8080/cat \
  CAT_DOMAINS=svc-a,svc-b pnpm dev

Triggering polls CAT problem reports (error entities only); evidence flows through CAT's logview/transaction/problem APIs with hop-by-hop cross-service navigation.

Feishu notification setup (app creation, scopes, event subscriptions, troubleshooting): see docs/feishu.md.

Production deployment on a Linux box without containers (systemd units, standalone Postgres, auth options, ops runbook): see docs/deploy-linux.md.

Roadmap

• Span exception ingest + regression reopen
• LLM grouping layer (haiku-tier, inspect-before-join invariant)
• Cost gates & cooldowns: active-run dedup, monthly token budget (fail-open), noise suppression window
• Feishu notification loop: incident card → findings update → resolve/ dismiss buttons (dismiss = 24h suppression); WebSocket, no public URL
• awaiting_human resume flow: ask_human tool pauses the session, the question lands in the card thread, a plain reply resumes the investigation with full context (24h timeout falls back to finish-on-available-evidence)
• Meituan CAT adapter: problem-report trigger + logview trace parsing + transaction-report topology synthesis + hop-by-hop investigation hint
• Fix-PR delivery (repo mount + patch proposal)
• RCAEval benchmark harness (eval/, Python)

License

Apache-2.0