🏠 agent-house

Lighthouse for agents - score an agent run, then tell it how to get faster and cheaper.

The ecosystem is full of tracers (Langfuse, LangSmith, Opik) that show you what happened. agent-house is a profiler: it ingests an agent run trace and produces a scored report (cost, latency, reliability, context-hygiene) with concrete, ranked fixes - redundant tool calls, missed parallelization, oversized context, uncached retrievals, model-tier mismatches - each with an estimated $ / ms saved and a code-level hint.

It's a build-time tool: a CLI + library + static HTML report. Zero required backend, zero runtime dependencies.

  agent-house - before / after

  naive ReAct agent     score 72/100   cost $0.245   7.0s
  after top-3 fixes     score 100/100   cost $0.024   2.0s

  ▲ +28 points · saved $0.221 and 5.0s per run

---

Install

npm install agent-house        # library
npx agent-house run trace.json # or just run the CLI

Requires Node 18+. Ships as ESM with TypeScript types.

Quick start

CLI

# Score a trace and print a report (+ optional HTML/JSON output)
npx agent-house run ./trace.json --out report.html --json report.json

# Fail CI when the score drops below a threshold
npx agent-house assert ./trace.json --min-score 80

# Compare against a saved baseline report and gate on regressions
npx agent-house ci ./trace.json --baseline ./baseline.json --min-score 80

Library

import { audit } from 'agent-house'

const report = await audit(trace)
// {
//   score: 72,
//   categories: [{ category: 'cost', score: 77, ... }, ...],
//   audits: [{ id, title, category, score, savings: { usd, ms }, findings, hint }, ...],
//   savings: { usd: 0.235, ms: 6400 },
//   run: { source, durationMs, spans, llmCalls, toolCalls, tokens, totalCostUsd },
// }

import { renderHtml, parseTrace } from 'agent-house'
const run = parseTrace(trace)
const html = renderHtml(audit(run), run) // self-contained, Lighthouse-style

audit() accepts a raw trace (auto-detected) or an already-normalized run.

Supported trace formats

Ingest is built on the OpenTelemetry GenAI semantic conventions, with thin adapters for popular agent frameworks. Format is auto-detected.

Framework	Format	Detected as
OpenTelemetry GenAI	OTLP/JSON with gen_ai.* attributes	otel
Vercel AI SDK	OTLP/JSON with ai.* attributes	vercel
LangGraph / LangSmith	LangSmith run tree (nested or flat)	langgraph
Google ADK	OTLP/JSON with gcp.vertex.agent.*	adk

Read from a JSON file (CLI) or pass parsed JSON (library). See examples/ for a runnable example per framework.

Scores

Each run gets four category scores (0–100) and a weighted overall score:

Category	Default weight	What it measures
Cost	0.30	wasted spend - wrong model tier, uncached prompts, bloat
Latency	0.30	wall-clock waste - duplicate and serial-when-parallelizable calls
Reliability	0.20	errors, retries, and loops
Context	0.20	oversized prompts relative to a healthy budget

Weights are configurable (audit(trace, { weights })).

Audits

Audit	Category	Flags
duplicate-tool-calls	latency	identical tool calls repeated in a run
parallelizable-tool-calls	latency	independent calls run serially that could be Promise.all
model-tier-mismatch	cost	a frontier model used for a trivial step
uncacheable-prompts	cost	large prompts re-sent across calls without caching
errors	reliability	spans that ended in an error
retry-loops	reliability	immediate retries and repeated-call loops
context-bloat	context	LLM calls sending oversized context

Every audit returns a 0–1 score, the specific spans it flagged, an estimated saving, and a code-level hint for the fix.

The report

• JSON - the full structured report (the value audit() returns).
• Static HTML - a self-contained, Lighthouse-style page (inline CSS, no external assets, no backend) with score gauges and an SVG span waterfall. The report JSON is embedded in the page, so it doubles as a portable artifact.

Configuration

audit(trace, {
  format: 'auto',            // 'otel' | 'vercel' | 'langgraph' | 'adk'
  models: [/* ModelPrice */],// override the price table with your negotiated rates
  thresholds: { contextBloatTokens: 12000 },
  weights: { cost: 0.4, latency: 0.3, reliability: 0.2, context: 0.1 },
})

The built-in price table covers Anthropic, OpenAI, Google Gemini, and xAI Grok (standard-tier list prices, as of 2026-06; older model ids still resolve to the nearest current entry). The $/ms estimates are only as credible as this table, so it's fully overridable.

How accurate are the savings?

The savings are estimates and may overlap across audits (a duplicate call is also "extra latency"). They're designed to be directionally credible and to rank fixes by impact - not to be exact invoices. The per-run cost and duration totals come straight from the trace and are exact.

Non-goals (MVP)

No hosted dashboard, no continuous monitoring, no auto-fix. It's a build-time tool: a CLI + library + static report, not a SaaS.

Architecture

trace (OTLP / Vercel / LangGraph / ADK)
        │  ingest adapters → normalize
        ▼
   AgentRun (spans: kind, timing, model, usage, …)
        │
        ├─ pricing engine  → cost per span
        ├─ audits          → findings + savings + hints
        └─ scoring         → category + overall scores
        ▼
   Report  →  JSON  +  static HTML (gauges + waterfall)

Adding a framework is one adapter that normalizes to AgentRun; nothing downstream changes.

Contributing

See CONTRIBUTING.md. The project is test-driven - every feature lands with Vitest tests. Run npm test.

License

MIT © Addy Osmani