Agent Harness Observability — detect errors, context rot, and regressions in your AI agent systems before users do.
Quick Start |
What It Does |
Packages |
CLI |
Contributing
aster-ho is an observability framework purpose-built for AI agent harnesses. It instruments your LLM calls, tool invocations, and agent loops — then detects problems automatically:
| Problem |
Detection |
| Tool call errors spike after a model update |
Per-model x tool baselines + z-score anomaly alerts |
| Model loops on the same tool call |
Context rot: repeated calls pattern |
| One tool failure causes a cascade |
Context rot: error cascade detection |
| Prompt grows uncontrollably |
Context rot: token bloat detection |
| Unknown errors accumulate silently |
Fingerprinted, tracked, auto-ticketed when count exceeds threshold |
When something goes wrong, aster-ho can:
- Create a GitHub Issue or Linear ticket automatically
- Generate a weekly ops report (Markdown or JSON)
- Pinpoint the exact trigger span in a trace with
ho root-cause
- Compare error rates before and after a harness change
# Install
pnpm add @ho/sdk @ho/instrumentation-openai @ho/exporter-file
# Instrument your agent
import { init } from "@ho/sdk";
import { OpenAIInstrumentation } from "@ho/instrumentation-openai";
import { FileExporter } from "@ho/exporter-file";
init({
serviceName: "my-agent",
instrumentations: [new OpenAIInstrumentation()],
exporters: [new FileExporter({ filePath: "./traces.jsonl" })],
});
// That's it. All OpenAI calls are now traced.┌─────────────────────────────────────────────────────────────────┐
│ Data Ingestion │
│ ┌──────────────────┐ ┌─────────────────────────────────────┐ │
│ │ Auto-Instrument │ │ Receivers │ │
│ │ OpenAI, Anthropic│ │ SWE-bench, Inspect AI, METR │ │
│ │ │ │ Docker, E2B, Modal sandboxes │ │
│ └────────┬─────────┘ └────────────────┬────────────────────┘ │
└───────────┼──────────────────────────────┼──────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Processing Pipeline (@ho/sdk) │
│ │
│ Span → CostEnricher → ErrorClassify → ContextRot → AlertProc │
│ ($) (category) (rot type) (z-score) │
└────────────────────────────────┬────────────────────────────────┘
│
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ File/Langfuse│ │ Prometheus /metrics│ │ Baseline (SQLite) │
│ (traces) │ │ (real-time) │ │ (anomaly detection)│
└──────────────┘ └────────────────────┘ └─────────┬──────────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────────┐ ┌────────────────┐
│ Auto-Ticket │ │ CLI Reports │
│ (GH / Linear)│ │ compare/report │
└──────────────┘ └────────────────┘
Full interactive diagram: docs/architecture.excalidraw (open with excalidraw.com)
| Package |
Description |
@ho/sdk |
Core SDK — init(), tracing, EnrichingExporter pipeline |
@ho/cli |
CLI tool — validate, serve, replay, baseline, compare, report, root-cause |
# Validate config
ho validate --config ho.config.yaml
# Run observability pipeline
ho serve --config ho.config.yaml
# Replay traces through pipeline
ho replay --file traces.jsonl
# Show per-model baselines
ho baseline show --format json
# Compare before/after a harness change
ho compare --base 7d --target 1d
# Weekly ops report
ho report weekly --format md
# Root-cause a failing trace
ho root-cause abc123-trace-id --file traces.jsonl
# ho.config.yaml
service_name: my-agent
enrichers:
- cost
- error-classify
- context-rot
exporters:
file:
path: ./traces.jsonl
prometheus:
port: 9464
baseline:
db_path: ./ho-baseline.sqlite
anomaly_zscore: 2.5
alerts:
rules:
- name: error-spike
condition:
type: anomaly
metric: error_rate
zscore_threshold: 3.0
min_samples: 50
window_ms: 300000
notifiers: [ticket]
tickets:
provider: github
github:
repo: your-org/your-repo
labels: [ho-auto]MIT
