Agent Activity Monitor

A monitoring system that ingests AI agent event streams, detects behavioral issues (loops, drift, failures), and surfaces insights via API and UI.

Setup

cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Start the API server
uvicorn main:app --port 8000

# In another terminal — run a simulation
python agent.py --scenario loop
python agent.py --scenario drift
python agent.py --scenario failure
python agent.py --scenario normal

API endpoints:

• POST /events — ingest an event
• GET /sessions — list all sessions with status and stats
• GET /sessions/{session_id} — session detail with events and detections

Architecture

POST /events → Validation & Dedup → SQLite (WAL mode) → Background Detection
                                                              │
                                                    ┌─────────┼─────────┐
                                                    ▼         ▼         ▼
                                               Loop Det   Drift Det  Failure Det
                                                    └─────────┼─────────┘
                                                              ▼
                                                     Session State
                                                     (status + evidence)

Key design choices:

• SQLite with WAL mode for concurrent reads/writes without a DB server
• In-memory session cache for fast detection, persisted to disk for durability
• Deduplication via (session_id, step, input_hash) unique constraint
• Events sorted by (step, timestamp) to handle out-of-order arrival
• Background detection runs after each event — no batch delay

Detection Logic

Three independent detectors run on every event. Priority: failing > looping > drifting > healthy.

Loop Detection

Two signals (either can trigger):

1. N-gram window Jaccard — Tokenize events as action:filename, slide two adjacent windows of 4 events, extract trigrams, compare with Jaccard similarity. Threshold starts at 0.55 and rises adaptively for longer sessions (cap 0.75).

2. Fingerprint repeats — Hash (action, normalized_input) into fingerprints. If the same fingerprint appears ≥3 times in recent events, it's a loop. Threshold scales with session length: max(3, n×0.25, cap 8).

Normalization strips UUIDs, timestamps, and numbers so that pytest test.py -v and pytest test.py -x get different prints, but session-abc123 and session-def456 collapse to the same.

Drift Detection

Splits session in half, compares using 3 signals (requires 2 of 3 to trigger):

1. Action distribution JSD > 0.20 — Measures if the type of work changed (e.g., from reading/thinking to mostly running commands).

2. File focus Jaccard < 0.30 — Different files being touched in each half. Requires ≥2 files per half to avoid false positives.

3. LLM keyword overlap < 0.30 — Top-10 keywords from LLM prompts diverge. Requires ≥2 LLM calls per half.

Failure Detection

Three signals (any one triggers — optimized for recall):

1. ≥3 consecutive failures at the session tail
2. >50% failure rate in an adaptive recent window (max(10, n×0.4) events)
3. Retry pattern — same action+input retried >2 times after failing

Trade-offs

Decision	Rationale
SQLite over Postgres	Zero-config, single-file, sufficient for local monitoring. WAL mode handles concurrent access.
Real-time over batch	Detection runs in background after each event. Adds ~ms latency per event but gives immediate feedback.
Adaptive thresholds over fixed	3 repeats in 10 events is a loop; 3 in 100 is normal. Session length matters.
2-of-3 voting for drift	Any single signal can false-positive (action mix naturally shifts, agent touches related files). Two signals agreeing = high confidence.
Any-1-of-3 for failure	Failures need immediate attention. False positives (overreporting) are less costly than missed failures.
Heuristics over ML	Interpretable evidence with every detection. No training data required. Every alert explains why.
In-memory cache + disk persistence	Fast detection (no DB queries during analysis) with crash recovery on restart.
MD5 fingerprinting	Fast, short (8 chars), collision-resistant enough for action dedup. Not security-sensitive.

Limitations (time constraints)

• No streaming/WebSocket push — frontend polls
• Single-process — no horizontal scaling
• Detection re-runs on full session each event (acceptable for typical session sizes <200 events)
• No configurable alert thresholds via API (requires code change or config override)