From cf9effdc62355938626038fb0f436f0c3171609e Mon Sep 17 00:00:00 2001 From: Doneyli De Jesus Date: Sun, 8 Feb 2026 11:05:01 -0500 Subject: [PATCH] feat: add trace analysis toolkit with ClickHouse and SDK scripts Add two scripts for analyzing Claude Code session traces: - analyze-traces.sh: queries Langfuse's ClickHouse directly for instant full-dataset analytics (self-hosted only, zero deps beyond curl) - analyze-traces-sdk.py: uses the official Langfuse Python SDK with REST API pagination (works with Cloud and self-hosted) Both produce five analyses: tool usage distribution, session turn distribution, productivity by session length (the efficiency cliff), and read-before-edit behavioral patterns. Includes docs/trace-analysis.md with methodology, ClickHouse schema reference, interpretation guide, and a query cookbook for custom analytics. Co-Authored-By: Claude Opus 4.6 --- README.md | 21 ++ docs/trace-analysis.md | 253 +++++++++++++++++ scripts/analyze-traces-sdk.py | 407 +++++++++++++++++++++++++++ scripts/analyze-traces.sh | 513 ++++++++++++++++++++++++++++++++++ 4 files changed, 1194 insertions(+) create mode 100644 docs/trace-analysis.md create mode 100644 scripts/analyze-traces-sdk.py create mode 100755 scripts/analyze-traces.sh diff --git a/README.md b/README.md index 3a9ada7..305a678 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,27 @@ Follow these steps to get Langfuse observability running in under 5 minutes: | Timing | Duration of each turn and tool call | | Project context | Project name extracted from workspace path | +## Analyze Your Traces + +Once you have traces flowing, run the built-in analyzer to find patterns in how Claude Code uses tools, how session length affects productivity, and whether your prompting habits pay off. + +**Quick start (self-hosted, ClickHouse direct):** +```bash +./scripts/analyze-traces.sh +``` + +**SDK / REST API (works with Cloud too):** +```bash +pip install langfuse +export LANGFUSE_PUBLIC_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_PUBLIC_KEY) +export LANGFUSE_SECRET_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_SECRET_KEY) +python3 scripts/analyze-traces-sdk.py +``` + +Both scripts produce five analyses: tool usage distribution, session turn distribution, productivity by session length, and read-before-edit patterns. Add `--json` for machine-readable output or `--tag ` to filter by project. + +See [docs/trace-analysis.md](docs/trace-analysis.md) for the full methodology, ClickHouse schema reference, and a query cookbook for writing your own analytics. + ## How It Works The Langfuse hook runs as a Claude Code **Stop hook** — it executes after each assistant response completes. diff --git a/docs/trace-analysis.md b/docs/trace-analysis.md new file mode 100644 index 0000000..a27706d --- /dev/null +++ b/docs/trace-analysis.md @@ -0,0 +1,253 @@ +# Analyzing Your Claude Code Traces + +Once you've been collecting traces for a few days, you'll have enough data to find real patterns in how Claude Code works. This guide explains the methodology, the queries, and how to interpret the results. + +## Two Scripts, Two Approaches + +| Script | Backend | Best for | Speed | +|--------|---------|----------|-------| +| `scripts/analyze-traces.sh` | ClickHouse direct (port 8124) | Self-hosted, large datasets | Fast (seconds) | +| `scripts/analyze-traces-sdk.py` | Langfuse REST API via SDK | Cloud or self-hosted, any size | Slower (paginates at 100/page) | + +Both produce the same five analyses. Choose based on your deployment. + +### Why two approaches? + +The Langfuse [Metrics API v2](https://langfuse.com/docs/api-and-data-platform/features/query-via-sdk) — which supports server-side aggregations — is currently **Cloud-only**. On self-hosted deployments, calling it returns: + +``` +"v2 APIs are currently in beta and only available on Langfuse Cloud" +``` + +The REST API v1 works on self-hosted but paginates at 100 items per page with no aggregation support. For a dataset of 2,000+ traces, that's 20+ paginated requests just to list traces, then more for observations. + +Querying ClickHouse directly bypasses all pagination — you get instant full-dataset analytics in a single HTTP call. + +## Quick Start + +### ClickHouse direct (self-hosted) + +```bash +# Requires: Docker running with Langfuse stack +./scripts/analyze-traces.sh + +# JSON output (pipe to jq, use in notebooks) +./scripts/analyze-traces.sh --json | jq . + +# Filter by project tag +./scripts/analyze-traces.sh --tag my-project +``` + +### SDK / REST API (any deployment) + +```bash +# Install the SDK +pip install langfuse + +# Set credentials (self-hosted: extract from Docker) +export LANGFUSE_PUBLIC_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_PUBLIC_KEY) +export LANGFUSE_SECRET_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_SECRET_KEY) +export LANGFUSE_HOST=http://localhost:3050 + +# Run +python3 scripts/analyze-traces-sdk.py + +# JSON output +python3 scripts/analyze-traces-sdk.py --json | jq . + +# Filter by tag +python3 scripts/analyze-traces-sdk.py --tag my-project +``` + +## How the Data Model Works + +Understanding the Langfuse schema helps you write custom queries. + +``` +Session (conversation) + └── Trace (one per turn) + ├── observation: "Claude Response" (type: GENERATION) + ├── observation: "Tool: Read" (type: SPAN) + ├── observation: "Tool: Edit" (type: SPAN) + ├── observation: "Tool: Bash" (type: SPAN) + └── observation: "Tool: Grep" (type: SPAN) +``` + +- **Session**: Groups all turns in a Claude Code conversation. Identified by `session_id`. +- **Trace**: One per user turn. Named `"Turn 1"`, `"Turn 2"`, etc. Contains the user prompt as `input` and assistant response as `output`. +- **Observation**: A span within a trace. Tool calls are named `"Tool: "`. The Claude response is a generation span named `"Claude Response"`. + +### ClickHouse Tables + +> **Warning**: These are internal Langfuse tables. The schema may change between versions. The queries in this guide were tested against Langfuse v3.150.0. + +**`traces`** — One row per turn: + +| Column | Type | Description | +|--------|------|-------------| +| `id` | String | Trace ID | +| `session_id` | Nullable(String) | Session grouping key | +| `name` | String | e.g., `"Turn 1"` | +| `timestamp` | DateTime64(3) | When the turn started | +| `tags` | Array(String) | e.g., `["claude-code", "my-project"]` | +| `input` | Nullable(String) | User prompt (JSON string) | +| `output` | Nullable(String) | Assistant response (JSON string) | +| `project_id` | String | Always `"claude-code"` for this template | + +**`observations`** — One row per tool call or generation: + +| Column | Type | Description | +|--------|------|-------------| +| `id` | String | Observation ID | +| `trace_id` | String | Parent trace | +| `name` | String | e.g., `"Tool: Read"`, `"Claude Response"` | +| `type` | String | `"SPAN"` for tools, `"GENERATION"` for Claude | +| `start_time` | DateTime64(3) | When the tool call started | +| `end_time` | Nullable(DateTime64(3)) | When it completed | +| `input` | Nullable(String) | Tool input (ZSTD compressed) | +| `output` | Nullable(String) | Tool output (ZSTD compressed) | + +## The Five Analyses + +### 1. Overview + +Total traces, sessions, observations, and date range. Sanity check that data is flowing. + +### 2. Tool Usage Distribution + +Counts every `"Tool: *"` observation, excluding meta-tools (TaskCreate, TaskUpdate, ExitPlanMode, AskUserQuestion) that are orchestration overhead rather than coding activity. + +**What to look for:** +- **Bash dominance** — If Bash is 40%+ of tool calls, Claude is spending more time executing commands (running tests, git ops, validation) than writing code. This is normal and healthy. +- **Read:Edit ratio** — A ratio near 1:1 means Claude reads a file for every edit. Higher means more exploration before acting. +- **Grep/Glob usage** — High search activity suggests Claude is working on unfamiliar codebases or large refactors. + +### 3. Session Turn Distribution + +Buckets sessions by how many turns they last. Expect a heavy skew toward single-turn sessions if you have automated workflows (email triage, scheduled tasks). + +**What to look for:** +- **Single-turn percentage** — If >80% are single-turn, your most productive AI pattern is one-shot automation, not interactive coding. +- **Long tail** — Sessions with 15+ turns warrant investigation. Are they productive or spinning? + +### 4. Productivity by Session Length + +The key analysis. Groups multi-turn sessions by length and measures **code changes per turn** (Edit + Write calls divided by turn count). + +**What to look for:** +- **Productivity cliff** — In our data, sessions with 4-7 turns produced 2.16 changes/turn (peak). Sessions with 13+ turns dropped to 1.18 changes/turn — a 45% decline. Each additional turn past the sweet spot produces diminishing returns. +- **Read-to-write ratio increase** — Longer sessions tend to have higher read:write ratios, meaning more searching and less producing. This is the "spinning" signal. + +**Actionable takeaway:** If a session passes ~10 turns without shipping, consider restarting with a refined prompt. Break large tasks into smaller, focused sessions. + +### 5. Read-Before-Edit Pattern + +Checks whether traces that include Edit or Write also include a Read step. Measures the "measure twice, cut once" behavior. + +**What to look for:** +- **High percentage (>85%)** — Claude consistently reads before editing. This is the disciplined pattern. +- **Low percentage (<70%)** — Could indicate rushed sessions or over-reliance on context from prior turns. +- **Exploration → Edit conversion** — What percentage of Read+Grep (deep exploration) sessions result in actual edits? Low conversion means exploration sessions that don't produce output. + +## Writing Custom Queries + +### Connect to ClickHouse + +```bash +# Get the password from the running container +CH_PASS=$(docker exec langfuse-web printenv CLICKHOUSE_PASSWORD) + +# Run any SQL query +curl -s "http://localhost:8124/?user=clickhouse&password=${CH_PASS}" \ + --data-binary "YOUR SQL HERE FORMAT Pretty" +``` + +### Example: Most active projects + +```sql +SELECT + arrayJoin(tags) as tag, + count(DISTINCT session_id) as sessions, + count() as traces +FROM traces +WHERE project_id = 'claude-code' AND is_deleted = 0 +GROUP BY tag +ORDER BY sessions DESC +FORMAT Pretty +``` + +### Example: Average session duration + +```sql +SELECT + round(avg(duration_min), 1) as avg_minutes, + round(median(duration_min), 1) as median_minutes +FROM ( + SELECT + session_id, + dateDiff('minute', min(timestamp), max(timestamp)) as duration_min + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND session_id IS NOT NULL + GROUP BY session_id + HAVING count() > 1 +) +FORMAT Pretty +``` + +### Example: Tool usage by project + +```sql +SELECT + arrayJoin(t.tags) as project, + o.name as tool, + count() as calls +FROM observations o +JOIN traces t ON o.trace_id = t.id AND t.project_id = 'claude-code' +WHERE o.project_id = 'claude-code' AND o.is_deleted = 0 + AND o.name LIKE 'Tool:%' +GROUP BY project, tool +ORDER BY project, calls DESC +FORMAT Pretty +``` + +### Example: First prompt length vs session turns + +```sql +WITH first_turns AS ( + SELECT session_id, length(input) as prompt_len + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND name = 'Turn 1' AND session_id IS NOT NULL +), +session_sizes AS ( + SELECT session_id, count() as turns + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND session_id IS NOT NULL + GROUP BY session_id + HAVING turns > 1 +) +SELECT + multiIf( + prompt_len < 100, 'Short (<100)', + prompt_len < 300, 'Medium (100-300)', + prompt_len < 600, 'Long (300-600)', + 'Very Long (600+)' + ) as prompt_size, + count() as sessions, + round(avg(turns), 1) as avg_turns, + round(median(turns), 1) as median_turns +FROM first_turns f +JOIN session_sizes s ON f.session_id = s.session_id +GROUP BY prompt_size +ORDER BY min(prompt_len) +FORMAT Pretty +``` + +## Caveats + +- **ClickHouse schema is internal**: Langfuse does not document or guarantee the ClickHouse table structure. It may change between versions. Pin your Langfuse version if you build pipelines on these queries. +- **Historical sessions**: Sessions with IDs like `historical-2026-01-25` are imported from prior transcript archives. The `session_id NOT LIKE 'historical-%'` filter excludes them. +- **Automated sessions**: If you run Claude Code for automated tasks (email triage, scheduled jobs), these inflate single-turn session counts. Filter by tag to focus on interactive coding sessions. +- **Observation ordering**: ClickHouse does not guarantee row order within a trace. Use `start_time` if you need tool call sequence within a turn. diff --git a/scripts/analyze-traces-sdk.py b/scripts/analyze-traces-sdk.py new file mode 100644 index 0000000..8c44c2b --- /dev/null +++ b/scripts/analyze-traces-sdk.py @@ -0,0 +1,407 @@ +#!/usr/bin/env python3 +""" +Langfuse Trace Analyzer — SDK / REST API + +Analyzes Claude Code session traces using the official Langfuse Python SDK. +Produces the same analytics as analyze-traces.sh but works with both +self-hosted and Langfuse Cloud deployments. + +Usage: + python3 scripts/analyze-traces-sdk.py # Pretty output + python3 scripts/analyze-traces-sdk.py --json # JSON output + python3 scripts/analyze-traces-sdk.py --tag myproj # Filter by tag + +Prerequisites: + pip install langfuse + +Environment variables: + LANGFUSE_PUBLIC_KEY — Project public key + LANGFUSE_SECRET_KEY — Project secret key + LANGFUSE_HOST — Langfuse URL (default: http://localhost:3050) + +Note: This script paginates through the REST API (100 items/page), so it +will be slower than analyze-traces.sh for large datasets. For self-hosted +deployments with thousands of traces, prefer the ClickHouse-direct script. + +See docs/trace-analysis.md for methodology and interpretation guide. +""" + +import argparse +import json +import os +import sys +from collections import Counter, defaultdict +from datetime import datetime + +try: + from langfuse import Langfuse +except ImportError: + print("Error: langfuse package not installed. Run: pip install langfuse", file=sys.stderr) + sys.exit(1) + + +def parse_args(): + parser = argparse.ArgumentParser(description="Analyze Langfuse traces from Claude Code sessions") + parser.add_argument("--json", action="store_true", help="Output as JSON") + parser.add_argument("--tag", type=str, default="", help="Filter by project tag") + parser.add_argument("--limit", type=int, default=0, help="Max traces to fetch (0 = all)") + return parser.parse_args() + + +def fetch_all_traces(client, tag_filter="", limit=0): + """Fetch all traces with pagination.""" + traces = [] + page = 1 + while True: + batch = client.api.trace.list(limit=100, page=page) + if not batch.data: + break + for t in batch.data: + if tag_filter and tag_filter not in (t.tags or []): + continue + traces.append(t) + if limit and len(traces) >= limit: + traces = traces[:limit] + break + if page >= batch.meta.total_pages: + break + page += 1 + if page % 10 == 0: + print(f" Fetching traces... page {page}/{batch.meta.total_pages}", file=sys.stderr) + return traces + + +def fetch_observations_for_trace(client, trace_id): + """Fetch all observations for a single trace.""" + observations = [] + page = 1 + while True: + batch = client.api.observations.get_many(trace_id=trace_id, limit=100, page=page) + if not batch.data: + break + observations.extend(batch.data) + if page >= batch.meta.total_pages: + break + page += 1 + return observations + + +def analyze(traces, client, tag_filter=""): + """Run all analyses on fetched traces.""" + + # --- 1. Overview --- + session_ids = set() + timestamps = [] + for t in traces: + if t.session_id: + session_ids.add(t.session_id) + if t.timestamp: + timestamps.append(t.timestamp) + + overview = { + "total_traces": len(traces), + "total_sessions": len(session_ids), + "earliest": min(timestamps).isoformat() if timestamps else None, + "latest": max(timestamps).isoformat() if timestamps else None, + } + + # --- Build per-trace observation data --- + # Group traces by session + sessions = defaultdict(list) + for t in traces: + sid = t.session_id or "no-session" + if sid.startswith("historical-"): + continue + sessions[sid].append(t) + + # Fetch observations for traces with tool calls + print(" Fetching observations (this may take a while)...", file=sys.stderr) + trace_observations = {} + tool_counter = Counter() + total_obs = 0 + + # Only fetch observations for multi-turn sessions to save time + traces_to_fetch = [] + for sid, session_traces in sessions.items(): + if len(session_traces) > 1: + traces_to_fetch.extend(session_traces) + + # Also sample single-turn sessions (up to 50) for tool distribution + single_turn_sessions = [ts[0] for sid, ts in sessions.items() if len(ts) == 1] + traces_to_fetch.extend(single_turn_sessions[:50]) + + for i, t in enumerate(traces_to_fetch): + obs_list = fetch_observations_for_trace(client, t.id) + trace_observations[t.id] = obs_list + total_obs += len(obs_list) + for o in obs_list: + if o.name and o.name.startswith("Tool:"): + tool_counter[o.name] += 1 + if (i + 1) % 20 == 0: + print(f" Processed {i + 1}/{len(traces_to_fetch)} traces...", file=sys.stderr) + + overview["total_observations"] = total_obs + + # --- 2. Tool Distribution --- + # Filter out meta tools + meta_prefixes = ("Tool: Task", "Tool: ExitPlan", "Tool: AskUser") + coding_tools = {k: v for k, v in tool_counter.items() + if not any(k.startswith(p) for p in meta_prefixes)} + total_coding = sum(coding_tools.values()) + + tool_distribution = [] + for name, count in sorted(coding_tools.items(), key=lambda x: -x[1]): + pct = round(count * 100.0 / total_coding, 1) if total_coding else 0 + tool_distribution.append({"name": name, "count": count, "pct": pct}) + + # Grouped categories + categories = Counter() + for name, count in tool_counter.items(): + if name == "Tool: Read": + categories["READ (understand)"] += count + elif name in ("Tool: Grep", "Tool: Glob"): + categories["SEARCH (find)"] += count + elif name in ("Tool: Edit", "Tool: Write"): + categories["WRITE (modify)"] += count + elif name == "Tool: Bash": + categories["EXECUTE (run/test)"] += count + elif name in ("Tool: WebSearch", "Tool: WebFetch"): + categories["WEB (research)"] += count + elif name.startswith("Tool: mcp"): + categories["MCP (external tools)"] += count + elif name.startswith("Tool: Task") or name.startswith("Tool: ExitPlan") or name.startswith("Tool: AskUser"): + categories["META (task mgmt)"] += count + else: + categories["OTHER"] += count + + total_all = sum(categories.values()) + category_distribution = [] + for cat, count in sorted(categories.items(), key=lambda x: -x[1]): + pct = round(count * 100.0 / total_all, 1) if total_all else 0 + category_distribution.append({"category": cat, "count": count, "pct": pct}) + + # --- 3. Session Turn Distribution --- + turn_buckets = Counter() + for sid, session_traces in sessions.items(): + n = len(session_traces) + if n == 1: + turn_buckets["1 turn"] += 1 + elif n <= 3: + turn_buckets["2-3 turns"] += 1 + elif n <= 7: + turn_buckets["4-7 turns"] += 1 + elif n <= 12: + turn_buckets["8-12 turns"] += 1 + else: + turn_buckets["13+ turns"] += 1 + + bucket_order = ["1 turn", "2-3 turns", "4-7 turns", "8-12 turns", "13+ turns"] + session_distribution = [{"bucket": b, "sessions": turn_buckets.get(b, 0)} for b in bucket_order] + + # --- 4. Productivity by Session Length --- + productivity = [] + for bucket_name, min_turns, max_turns in [("2-3 turns", 2, 3), ("4-7 turns", 4, 7), + ("8-12 turns", 8, 12), ("13+ turns", 13, 999)]: + bucket_sessions = [] + for sid, session_traces in sessions.items(): + n = len(session_traces) + if min_turns <= n <= max_turns: + changes = 0 + reads = 0 + bashes = 0 + for t in session_traces: + for o in trace_observations.get(t.id, []): + if o.name == "Tool: Edit" or o.name == "Tool: Write": + changes += 1 + elif o.name == "Tool: Read": + reads += 1 + elif o.name == "Tool: Bash": + bashes += 1 + bucket_sessions.append({ + "turns": n, "changes": changes, "reads": reads, "bashes": bashes + }) + + if bucket_sessions: + avg_changes = round(sum(s["changes"] for s in bucket_sessions) / len(bucket_sessions), 1) + avg_reads = round(sum(s["reads"] for s in bucket_sessions) / len(bucket_sessions), 1) + avg_bashes = round(sum(s["bashes"] for s in bucket_sessions) / len(bucket_sessions), 1) + avg_turns = sum(s["turns"] for s in bucket_sessions) / len(bucket_sessions) + changes_per_turn = round(avg_changes / avg_turns, 2) if avg_turns else 0 + r2w = round(avg_reads / max(avg_changes, 0.1), 1) + productivity.append({ + "bucket": bucket_name, + "sessions": len(bucket_sessions), + "avg_changes": avg_changes, + "avg_reads": avg_reads, + "avg_bashes": avg_bashes, + "changes_per_turn": changes_per_turn, + "read_to_write_ratio": r2w, + }) + + # --- 5. Read-Before-Edit Pattern --- + traces_with_edit = 0 + edit_with_read = 0 + traces_with_write = 0 + write_with_read = 0 + exploration_traces = 0 + exploration_to_edit = 0 + + for tid, obs_list in trace_observations.items(): + tool_names = {o.name for o in obs_list if o.name and o.name.startswith("Tool:")} + if len(tool_names) < 3: + continue + + has_read = "Tool: Read" in tool_names + has_edit = "Tool: Edit" in tool_names + has_write = "Tool: Write" in tool_names + has_grep = "Tool: Grep" in tool_names + + if has_edit: + traces_with_edit += 1 + if has_read: + edit_with_read += 1 + + if has_write: + traces_with_write += 1 + if has_read: + write_with_read += 1 + + if has_read and has_grep: + exploration_traces += 1 + if has_edit: + exploration_to_edit += 1 + + patterns = { + "traces_with_edit": traces_with_edit, + "edit_with_read": edit_with_read, + "pct_read_before_edit": round(edit_with_read * 100.0 / max(traces_with_edit, 1), 1), + "traces_with_write": traces_with_write, + "write_with_read": write_with_read, + "pct_read_before_write": round(write_with_read * 100.0 / max(traces_with_write, 1), 1), + "exploration_traces": exploration_traces, + "exploration_to_edit": exploration_to_edit, + "pct_exploration_to_edit": round(exploration_to_edit * 100.0 / max(exploration_traces, 1), 1), + } + + return { + "overview": overview, + "tool_distribution": tool_distribution, + "category_distribution": category_distribution, + "session_distribution": session_distribution, + "productivity_by_length": productivity, + "patterns": patterns, + } + + +def print_pretty(results): + """Print results in a human-readable format.""" + BOLD = "\033[1m" + BLUE = "\033[0;34m" + DIM = "\033[2m" + NC = "\033[0m" + + def header(title): + print(f"\n{BLUE}{'━' * 58}{NC}") + print(f"{BOLD} {title}{NC}") + print(f"{BLUE}{'━' * 58}{NC}\n") + + # Overview + ov = results["overview"] + header("1. Overview") + print(f" Traces: {ov['total_traces']}") + print(f" Sessions: {ov['total_sessions']}") + print(f" Observations: {ov['total_observations']}") + print(f" Date range: {ov['earliest'][:10] if ov['earliest'] else 'N/A'} to {ov['latest'][:10] if ov['latest'] else 'N/A'}") + + # Tool distribution + header("2. Tool Usage Distribution") + print(f" {'Tool':<35} {'Calls':>6} {'%':>6}") + print(f" {'─' * 49}") + for t in results["tool_distribution"][:15]: + print(f" {t['name']:<35} {t['count']:>6} {t['pct']:>5.1f}%") + + print(f"\n {DIM}Grouped by action type:{NC}") + print(f" {'Category':<25} {'Calls':>6} {'%':>6}") + print(f" {'─' * 39}") + for c in results["category_distribution"]: + print(f" {c['category']:<25} {c['count']:>6} {c['pct']:>5.1f}%") + + # Session distribution + header("3. Session Turn Distribution") + print(f" {'Bucket':<15} {'Sessions':>10}") + print(f" {'─' * 27}") + for s in results["session_distribution"]: + print(f" {s['bucket']:<15} {s['sessions']:>10}") + + # Productivity + header("4. Productivity by Session Length") + print(f" {'Bucket':<12} {'Sessions':>8} {'Changes':>9} {'Reads':>7} {'Chg/Turn':>10} {'R:W':>6}") + print(f" {'─' * 54}") + for p in results["productivity_by_length"]: + print(f" {p['bucket']:<12} {p['sessions']:>8} {p['avg_changes']:>9} {p['avg_reads']:>7} {p['changes_per_turn']:>10} {p['read_to_write_ratio']:>5.1f}") + print(f"\n {DIM}changes_per_turn = avg(Edit + Write) / avg(turns){NC}") + + # Patterns + header("5. Read-Before-Edit Pattern") + pat = results["patterns"] + print(f" Traces with Edit: {pat['traces_with_edit']:>5} ({pat['pct_read_before_edit']}% also Read first)") + print(f" Traces with Write: {pat['traces_with_write']:>5} ({pat['pct_read_before_write']}% also Read first)") + print(f" Exploration (Read+Grep): {pat['exploration_traces']:>5} ({pat['pct_exploration_to_edit']}% lead to Edit)") + + print(f"\n{BLUE}{'━' * 58}{NC}") + print(f"{BOLD} Done.{NC} {DIM}See docs/trace-analysis.md for interpretation guide.{NC}") + print(f"{BLUE}{'━' * 58}{NC}\n") + + +def main(): + args = parse_args() + + # Initialize client + host = os.environ.get("LANGFUSE_HOST", "http://localhost:3050") + public_key = os.environ.get("LANGFUSE_PUBLIC_KEY") + secret_key = os.environ.get("LANGFUSE_SECRET_KEY") + + if not public_key or not secret_key: + print("Error: Set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY environment variables.", file=sys.stderr) + print(f" Tip: Extract from Docker with:", file=sys.stderr) + print(f" export LANGFUSE_PUBLIC_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_PUBLIC_KEY)", file=sys.stderr) + print(f" export LANGFUSE_SECRET_KEY=$(docker exec langfuse-web printenv LANGFUSE_INIT_PROJECT_SECRET_KEY)", file=sys.stderr) + sys.exit(1) + + client = Langfuse(public_key=public_key, secret_key=secret_key, host=host) + + # Verify connection + try: + health = client.api.health.health() + if not args.json: + print(f"\n Connected to Langfuse {health.version} at {host}") + except Exception as e: + print(f"Error: Could not connect to Langfuse at {host}: {e}", file=sys.stderr) + sys.exit(1) + + # Fetch traces + if not args.json: + print(" Fetching traces...", file=sys.stderr) + traces = fetch_all_traces(client, tag_filter=args.tag, limit=args.limit) + + if not traces: + print("No traces found.", file=sys.stderr) + sys.exit(0) + + if not args.json: + print(f" Found {len(traces)} traces. Analyzing...", file=sys.stderr) + + # Analyze + results = analyze(traces, client, tag_filter=args.tag) + + # Output + if args.json: + print(json.dumps(results, indent=2, default=str)) + else: + print_pretty(results) + + client.shutdown() + + +if __name__ == "__main__": + main() diff --git a/scripts/analyze-traces.sh b/scripts/analyze-traces.sh new file mode 100755 index 0000000..686798d --- /dev/null +++ b/scripts/analyze-traces.sh @@ -0,0 +1,513 @@ +#!/usr/bin/env bash +set -euo pipefail + +# ============================================================================= +# Langfuse Trace Analyzer — ClickHouse Direct +# +# Analyzes Claude Code session traces by querying Langfuse's internal +# ClickHouse database directly. Produces tool usage distributions, session +# metrics, productivity analysis, and behavioral patterns. +# +# Usage: +# ./scripts/analyze-traces.sh # Pretty terminal output +# ./scripts/analyze-traces.sh --json # Machine-readable JSON +# ./scripts/analyze-traces.sh --tag myproj # Filter by project tag +# +# Prerequisites: +# - Docker running with Langfuse stack (docker compose up -d) +# - curl installed +# +# Why ClickHouse direct instead of the REST API? +# The Langfuse REST API paginates at 100 items per page and has no +# aggregation support. The Metrics API v2 (which does aggregation) is +# Cloud-only. For self-hosted deployments, querying ClickHouse directly +# is the only practical path for full-dataset analytics. +# +# See docs/trace-analysis.md for methodology and query explanations. +# ============================================================================= + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +CYAN='\033[0;36m' +BOLD='\033[1m' +DIM='\033[2m' +NC='\033[0m' + +# Defaults +OUTPUT_FORMAT="pretty" +TAG_FILTER="" +CH_PORT="${LANGFUSE_CH_PORT:-8124}" +CH_USER="clickhouse" +CONTAINER_NAME="${LANGFUSE_CONTAINER:-langfuse-web}" + +# Parse arguments +while [[ $# -gt 0 ]]; do + case $1 in + --json) + OUTPUT_FORMAT="json" + shift + ;; + --tag) + TAG_FILTER="$2" + shift 2 + ;; + --help|-h) + echo "Usage: $0 [--json] [--tag ]" + echo "" + echo "Options:" + echo " --json Output results as JSON (pipe to jq)" + echo " --tag Filter traces by project tag" + echo " -h, --help Show this help message" + echo "" + echo "Environment variables:" + echo " LANGFUSE_CH_PORT ClickHouse HTTP port (default: 8124)" + echo " LANGFUSE_CONTAINER Web container name (default: langfuse-web)" + exit 0 + ;; + *) + echo "Unknown option: $1" + exit 1 + ;; + esac +done + +# --- Helpers --- + +print_header() { + if [[ "$OUTPUT_FORMAT" == "pretty" ]]; then + echo "" + echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e "${BOLD} $1${NC}" + echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo "" + fi +} + +print_subheader() { + if [[ "$OUTPUT_FORMAT" == "pretty" ]]; then + echo -e "${CYAN} $1${NC}" + echo "" + fi +} + +check_pass() { + if [[ "$OUTPUT_FORMAT" == "pretty" ]]; then + echo -e "${GREEN} ✓ $1${NC}" + fi +} + +check_fail() { + echo -e "${RED} ✗ $1${NC}" >&2 + exit 1 +} + +query_ch() { + local sql="$1" + local format="${2:-Pretty}" + curl -sf "http://localhost:${CH_PORT}/?user=${CH_USER}&password=${CH_PASSWORD}" \ + --data-binary "$sql FORMAT $format" 2>/dev/null +} + +query_ch_raw() { + local sql="$1" + curl -sf "http://localhost:${CH_PORT}/?user=${CH_USER}&password=${CH_PASSWORD}" \ + --data-binary "$sql" 2>/dev/null +} + +# Build tag filter clause for SQL +tag_where() { + if [[ -n "$TAG_FILTER" ]]; then + echo "AND has(tags, '${TAG_FILTER}')" + fi +} + +# For observations, filter via trace join +tag_obs_where() { + if [[ -n "$TAG_FILTER" ]]; then + echo "AND trace_id IN (SELECT id FROM traces WHERE project_id = 'claude-code' AND is_deleted = 0 AND has(tags, '${TAG_FILTER}'))" + fi +} + +# --- Preflight Checks --- + +if [[ "$OUTPUT_FORMAT" == "pretty" ]]; then + echo "" + echo -e "${BOLD}Langfuse Trace Analyzer${NC} ${DIM}(ClickHouse direct)${NC}" +fi + +# Check Docker +if ! docker info &>/dev/null; then + check_fail "Docker is not running. Start Docker and try again." +fi + +# Check container is running +if ! docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then + check_fail "Container '${CONTAINER_NAME}' is not running. Run: docker compose up -d" +fi + +# Extract ClickHouse password from running container +CH_PASSWORD=$(docker exec "$CONTAINER_NAME" printenv CLICKHOUSE_PASSWORD 2>/dev/null) || \ + check_fail "Could not read CLICKHOUSE_PASSWORD from ${CONTAINER_NAME}" + +# Verify ClickHouse connectivity +if ! curl -sf "http://localhost:${CH_PORT}/ping" &>/dev/null; then + check_fail "ClickHouse not reachable on port ${CH_PORT}" +fi + +check_pass "Connected to ClickHouse on port ${CH_PORT}" + +if [[ -n "$TAG_FILTER" ]]; then + check_pass "Filtering by tag: ${TAG_FILTER}" +fi + +# ============================================================================= +# JSON output mode — collect all results in a single JSON object +# ============================================================================= + +if [[ "$OUTPUT_FORMAT" == "json" ]]; then + + # Overview + overview=$(query_ch_raw " + SELECT + count() as total_traces, + count(DISTINCT session_id) as total_sessions, + min(timestamp) as earliest, + max(timestamp) as latest + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + $(tag_where) + FORMAT JSONEachRow + ") + + obs_count=$(query_ch_raw " + SELECT count() as total_observations + FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 + $(tag_obs_where) + FORMAT JSONEachRow + ") + + # Tool distribution + tool_dist=$(query_ch_raw " + SELECT + name, + count() as count, + round(count() * 100.0 / ( + SELECT count() FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND name LIKE 'Tool:%' + AND name NOT LIKE 'Tool: Task%' + AND name NOT LIKE 'Tool: ExitPlan%' + AND name NOT LIKE 'Tool: AskUser%' + $(tag_obs_where) + ), 1) as pct + FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND name LIKE 'Tool:%' + AND name NOT LIKE 'Tool: Task%' + AND name NOT LIKE 'Tool: ExitPlan%' + AND name NOT LIKE 'Tool: AskUser%' + $(tag_obs_where) + GROUP BY name + ORDER BY count DESC + FORMAT JSONEachRow + ") + + # Session distribution + session_dist=$(query_ch_raw " + WITH session_turns AS ( + SELECT session_id, count() as turns + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND session_id IS NOT NULL AND session_id NOT LIKE 'historical-%' + $(tag_where) + GROUP BY session_id + ) + SELECT + multiIf( + turns = 1, '1', + turns <= 3, '2-3', + turns <= 7, '4-7', + turns <= 12, '8-12', + '13+' + ) as bucket, + count() as sessions, + round(avg(turns), 1) as avg_turns + FROM session_turns + GROUP BY bucket + ORDER BY min(turns) + FORMAT JSONEachRow + ") + + # Productivity + productivity=$(query_ch_raw " + WITH session_metrics AS ( + SELECT + t.session_id, + count(DISTINCT t.id) as turns, + countIf(o.name = 'Tool: Edit') + countIf(o.name = 'Tool: Write') as changes, + countIf(o.name = 'Tool: Read') as reads, + countIf(o.name = 'Tool: Bash') as bashes + FROM traces t + LEFT JOIN observations o ON t.id = o.trace_id AND o.project_id = 'claude-code' + WHERE t.project_id = 'claude-code' AND t.is_deleted = 0 + AND t.session_id IS NOT NULL AND t.session_id NOT LIKE 'historical-%' + $(tag_where | sed 's/tags/t.tags/g') + GROUP BY t.session_id + HAVING turns > 1 + ) + SELECT + multiIf(turns<=3,'2-3',turns<=7,'4-7',turns<=12,'8-12','13+') as bucket, + count() as sessions, + round(avg(changes), 1) as avg_changes, + round(avg(reads), 1) as avg_reads, + round(avg(bashes), 1) as avg_bashes, + round(avg(changes) / avg(turns), 2) as changes_per_turn, + round(avg(reads) / greatest(avg(changes), 0.1), 1) as read_to_write_ratio + FROM session_metrics + GROUP BY bucket + ORDER BY min(turns) + FORMAT JSONEachRow + ") + + # Read-before-edit + patterns=$(query_ch_raw " + WITH trace_tools AS ( + SELECT + trace_id, + has(groupArray(name), 'Tool: Read') as has_read, + has(groupArray(name), 'Tool: Edit') as has_edit, + has(groupArray(name), 'Tool: Write') as has_write, + has(groupArray(name), 'Tool: Grep') as has_grep, + length(groupArray(name)) as tool_count + FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 AND name LIKE 'Tool:%' + $(tag_obs_where) + GROUP BY trace_id + HAVING tool_count >= 3 + ) + SELECT + countIf(has_edit) as traces_with_edit, + countIf(has_edit AND has_read) as edit_with_read, + round(countIf(has_edit AND has_read) * 100.0 / greatest(countIf(has_edit), 1), 1) as pct_read_before_edit, + countIf(has_write) as traces_with_write, + countIf(has_write AND has_read) as write_with_read, + round(countIf(has_write AND has_read) * 100.0 / greatest(countIf(has_write), 1), 1) as pct_read_before_write, + countIf(has_read AND has_grep) as exploration_traces, + countIf(has_read AND has_grep AND has_edit) as exploration_to_edit, + round(countIf(has_read AND has_grep AND has_edit) * 100.0 / greatest(countIf(has_read AND has_grep), 1), 1) as pct_exploration_to_edit + FORMAT JSONEachRow + ") + + # Assemble JSON + echo "{" + echo " \"overview\": ${overview}," + echo " \"observation_count\": ${obs_count}," + echo " \"tool_distribution\": [$(echo "$tool_dist" | paste -sd, -)]," + echo " \"session_distribution\": [$(echo "$session_dist" | paste -sd, -)]," + echo " \"productivity_by_length\": [$(echo "$productivity" | paste -sd, -)]," + echo " \"patterns\": ${patterns}" + echo "}" + + exit 0 +fi + +# ============================================================================= +# Pretty output mode +# ============================================================================= + +# --- 1. Overview --- +print_header "1. Overview" + +query_ch " +SELECT + count() as traces, + count(DISTINCT session_id) as sessions, + formatDateTime(min(timestamp), '%Y-%m-%d') as earliest, + formatDateTime(max(timestamp), '%Y-%m-%d') as latest +FROM traces +WHERE project_id = 'claude-code' AND is_deleted = 0 + $(tag_where) +" + +obs_total=$(query_ch_raw "SELECT count() FROM observations WHERE project_id = 'claude-code' AND is_deleted = 0 $(tag_obs_where)") +echo -e " ${DIM}Total observations: ${obs_total}${NC}" +echo "" + +# --- 2. Tool Usage Distribution --- +print_header "2. Tool Usage Distribution" +print_subheader "Coding tools only (excludes TaskCreate/Update, ExitPlanMode, AskUser)" + +query_ch " +SELECT + name as tool, + count() as calls, + round(count() * 100.0 / ( + SELECT count() FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND name LIKE 'Tool:%' + AND name NOT LIKE 'Tool: Task%' + AND name NOT LIKE 'Tool: ExitPlan%' + AND name NOT LIKE 'Tool: AskUser%' + $(tag_obs_where) + ), 1) as pct +FROM observations +WHERE project_id = 'claude-code' AND is_deleted = 0 + AND name LIKE 'Tool:%' + AND name NOT LIKE 'Tool: Task%' + AND name NOT LIKE 'Tool: ExitPlan%' + AND name NOT LIKE 'Tool: AskUser%' + $(tag_obs_where) +GROUP BY tool +ORDER BY calls DESC +LIMIT 15 +" + +print_subheader "Grouped by action type" + +query_ch " +SELECT + multiIf( + name IN ('Tool: Read'), 'READ (understand)', + name IN ('Tool: Grep', 'Tool: Glob'), 'SEARCH (find)', + name IN ('Tool: Edit', 'Tool: Write'), 'WRITE (modify)', + name IN ('Tool: Bash'), 'EXECUTE (run/test)', + name IN ('Tool: WebSearch', 'Tool: WebFetch'), 'WEB (research)', + name LIKE 'Tool: mcp%', 'MCP (external tools)', + 'OTHER' + ) as category, + count() as calls, + round(count() * 100.0 / ( + SELECT count() FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 AND name LIKE 'Tool:%' + $(tag_obs_where) + ), 1) as pct +FROM observations +WHERE project_id = 'claude-code' AND is_deleted = 0 AND name LIKE 'Tool:%' + $(tag_obs_where) +GROUP BY category +ORDER BY calls DESC +" + +# --- 3. Session Turn Distribution --- +print_header "3. Session Turn Distribution" +print_subheader "How many turns do sessions typically last?" + +query_ch " +WITH session_turns AS ( + SELECT session_id, count() as turns + FROM traces + WHERE project_id = 'claude-code' AND is_deleted = 0 + AND session_id IS NOT NULL AND session_id NOT LIKE 'historical-%' + $(tag_where) + GROUP BY session_id +) +SELECT + multiIf( + turns = 1, '1 turn', + turns <= 3, '2-3 turns', + turns <= 7, '4-7 turns', + turns <= 12, '8-12 turns', + '13+ turns' + ) as bucket, + count() as sessions, + round(avg(turns), 1) as avg_turns_in_bucket +FROM session_turns +GROUP BY bucket +ORDER BY min(turns) +" + +# --- 4. Productivity by Session Length --- +print_header "4. Productivity by Session Length" +print_subheader "Code changes per turn — does efficiency drop in longer sessions?" + +query_ch " +WITH session_metrics AS ( + SELECT + t.session_id, + count(DISTINCT t.id) as turns, + countIf(o.name = 'Tool: Edit') + countIf(o.name = 'Tool: Write') as changes, + countIf(o.name = 'Tool: Read') as reads, + countIf(o.name = 'Tool: Bash') as bashes + FROM traces t + LEFT JOIN observations o ON t.id = o.trace_id AND o.project_id = 'claude-code' + WHERE t.project_id = 'claude-code' AND t.is_deleted = 0 + AND t.session_id IS NOT NULL AND t.session_id NOT LIKE 'historical-%' + $(tag_where | sed 's/tags/t.tags/g') + GROUP BY t.session_id + HAVING turns > 1 +) +SELECT + multiIf(turns<=3,'2-3 turns',turns<=7,'4-7 turns',turns<=12,'8-12 turns','13+ turns') as bucket, + count() as sessions, + round(avg(changes), 1) as avg_code_changes, + round(avg(reads), 1) as avg_reads, + round(avg(bashes), 1) as avg_bashes, + round(avg(changes) / avg(turns), 2) as changes_per_turn, + round(avg(reads) / greatest(avg(changes), 0.1), 1) as read_to_write_ratio +FROM session_metrics +GROUP BY bucket +ORDER BY min(turns) +" + +echo -e " ${DIM}changes_per_turn = avg(Edit + Write) / avg(turns)${NC}" +echo -e " ${DIM}read_to_write_ratio = avg(Read) / avg(Edit + Write)${NC}" +echo "" + +# --- 5. Read-Before-Edit Pattern --- +print_header "5. Read-Before-Edit Pattern" +print_subheader "Do successful edits start with reading the file first?" + +query_ch " +WITH trace_tools AS ( + SELECT + trace_id, + groupArray(name) as tools, + has(tools, 'Tool: Read') as has_read, + has(tools, 'Tool: Edit') as has_edit, + has(tools, 'Tool: Write') as has_write, + has(tools, 'Tool: Grep') as has_grep, + length(tools) as tool_count + FROM observations + WHERE project_id = 'claude-code' AND is_deleted = 0 AND name LIKE 'Tool:%' + $(tag_obs_where) + GROUP BY trace_id + HAVING tool_count >= 3 +) +SELECT + 'Traces with Edit' as pattern, + countIf(has_edit) as total, + countIf(has_edit AND has_read) as with_read, + round(countIf(has_edit AND has_read) * 100.0 / greatest(countIf(has_edit), 1), 1) as pct +FROM trace_tools + +UNION ALL + +SELECT + 'Traces with Write', + countIf(has_write), + countIf(has_write AND has_read), + round(countIf(has_write AND has_read) * 100.0 / greatest(countIf(has_write), 1), 1) +FROM trace_tools + +UNION ALL + +SELECT + 'Exploration (Read+Grep) → Edit', + countIf(has_read AND has_grep), + countIf(has_read AND has_grep AND has_edit), + round(countIf(has_read AND has_grep AND has_edit) * 100.0 / greatest(countIf(has_read AND has_grep), 1), 1) +FROM trace_tools +" + +# --- Summary --- +if [[ "$OUTPUT_FORMAT" == "pretty" ]]; then + echo "" + echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e "${BOLD} Done.${NC} ${DIM}See docs/trace-analysis.md for interpretation guide.${NC}" + echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo "" +fi