agent-observatory

An LLM-maintained observability wiki for multi-agent pipelines. Capture what your pipeline did on every run, then let a coding agent keep an interlinked markdown wiki of what those runs mean — patterns, per-agent behavior, failure modes, experiments.

Inspired by Andrej Karpathy's LLM-wiki pattern: instead of re-deriving insight on every query, an agent compiles it once into a structured, self-maintaining knowledge base.

Built as the observability layer for Thesis, an AI equity-research pipeline. See the write-up: Backtesting Agentic Equity Research.

The three layers

1. Raw capture (observatory/runs/) — one immutable folder per run: manifest, per-agent records, verdict check. The source of truth.
2. The wiki (observatory/agents/, tickers/, patterns/, failure-modes/, experiments/, prompt-versions/) — LLM-maintained markdown, best viewed as an Obsidian vault.
3. Governance (observatory/CLAUDE.md + scripts/observatory-lint.js) — the schema the maintaining agent follows, plus consistency checks that keep the wiki from rotting.

What's here

scripts/    six Node CLI tools: init, record-agent, record-event, finalize, lint, query
observatory/  the vault (ships empty/scaffolded) + CLAUDE.md governance + example answer key
templates/  empty page templates for each wiki page type
ADAPTING.md how to wire it into your own pipeline

Quickstart

npm install
npm test          # end-to-end smoke test against a temp vault

Then read ADAPTING.md to wire it into your pipeline.

Requirements

Node ≥ 20. One dependency (yaml). No API keys — the wiki is maintained by your coding agent, not a bundled model call.

Acknowledgments

Built with Claude Code (Anthropic's Claude). This is a human-directed, AI-assisted build — the architecture, methodology, and decisions are mine; Claude did much of the implementation under that direction.

License

MIT