Developer Notes

Technical architecture and implementation details for developers working on or extending LLM Tracer.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                 Your LangChain/LangGraph Application            │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  LLMTraceCallback (Client SDK)                            │  │
│  │  - Implements LangChain's BaseCallbackHandler             │  │
│  │  - Background thread for async transmission               │  │
│  │  - Batches and compresses events                          │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ HTTP POST (gzip, X-API-Key header)
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                     API Server (FastAPI)                        │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  TraceService                                             │  │
│  │  - Processes batched events                               │  │
│  │  - Maintains run hierarchy                                │  │
│  │  - Computes costs via CostCalculator                      │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  SQLite Database                                          │  │
│  │  - applications, components, api_keys                     │  │
│  │  - traces, runs, model_costs                              │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ REST API
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Web Frontend (React)                        │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  TanStack Query - Data fetching and caching               │  │
│  │  React Router - Client-side routing                       │  │
│  │  TailwindCSS - Styling                                    │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Design Principles

1. Zero Latency Impact

The SDK never slows down the host application:

Background threading: Network I/O on separate thread
Non-blocking queue: Events added without waiting
Graceful degradation: Errors logged, never raised

def on_llm_end(self, response, **kwargs):
    event = self._create_event(response)
    self.queue.put(event)  # Non-blocking

2. Fail-Safe Operations

The SDK never crashes the host application:

try:
    self._send_batch(events)
except Exception as e:
    logger.warning(f"Failed to send: {e}")
    # Continue operating

3. API Key Authentication

Traces are secured with per-application API keys:

Keys start with pt_ prefix
Hashed in database (bcrypt)
Scoped to single application
Can be revoked

4. Wire Protocol Stability

Event format between SDK and API is stable:

@dataclass
class TraceEvent:
    event_type: EventType      # run_start, run_end
    timestamp: datetime
    component: str | None
    payload: dict[str, Any]    # Run data

Client SDK Architecture

Components

┌─────────────────────────────────────────────────────────────┐
│                    LLMTraceCallback                         │
│  - LangChain BaseCallbackHandler                            │
│  - Exception-safe callback methods                          │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    TraceCollector                           │
│  - Run lifecycle management                                 │
│  - Trace hierarchy (parent-child)                           │
│  - Thread-safe with mutex                                   │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    BatchQueue                               │
│  - Thread-safe event buffer                                 │
│  - Daemon worker thread                                     │
│  - Size/time-based flushing                                 │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    HttpTransport                            │
│  - Gzip compression                                         │
│  - Exponential backoff retries                              │
│  - X-API-Key authentication                                 │
└─────────────────────────────────────────────────────────────┘

Key Classes

Class	File	Responsibility
`LLMTraceCallback`	`callback.py`	LangChain integration
`TraceCollector`	`collector.py`	Run hierarchy management
`BatchQueue`	`queue.py`	Event buffering
`HttpTransport`	`transport.py`	HTTP communication
`Config`	`config.py`	Configuration management

Event Flow

LangChain triggers callback (e.g., on_llm_start)
LLMTraceCallback wraps in exception handler
TraceCollector creates/updates Run object
TraceEvent created and enqueued to BatchQueue
Background worker sends batch when threshold reached
HttpTransport sends gzip-compressed POST

Token Extraction

The SDK handles multiple token key variations:

# Checked locations (in order):
llm_output["token_usage"]["prompt_tokens"]
llm_output["usage"]["input_tokens"]
response.usage_metadata["input_tokens"]

Model Name Extraction

# Checked locations (in order):
kwargs["model_name"]
kwargs["model"]
serialized["id"][-1]  # LangChain class name

API Server Architecture

Routers

Router	Path	Purpose
`applications`	`/api/applications`	App management
`api_keys`	`/api/api-keys`	Key management
`components`	`/api/components`	Component management
`hierarchy`	`/api/hierarchy`	Sidebar tree
`traces`	`/api/traces`	Ingestion & queries
`stats`	`/api/stats`	Statistics
`model_costs`	`/api/model-costs`	Pricing config

Services

Service	Responsibility
`TraceService`	Event processing, run management
`QueryService`	Trace retrieval, filtering
`CostCalculator`	LLM cost computation
`AuthService`	API key validation

Database Schema

-- Applications (top-level containers)
CREATE TABLE applications (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

-- Components (sub-divisions within apps)
CREATE TABLE components (
    id TEXT PRIMARY KEY,
    application_id TEXT REFERENCES applications(id),
    name TEXT NOT NULL,
    description TEXT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

-- API Keys (authentication)
CREATE TABLE api_keys (
    id TEXT PRIMARY KEY,
    application_id TEXT REFERENCES applications(id),
    key_hash TEXT NOT NULL,
    key_prefix TEXT NOT NULL,
    name TEXT,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP
);

-- Traces (aggregated data)
CREATE TABLE traces (
    id TEXT PRIMARY KEY,
    application_id TEXT REFERENCES applications(id),
    component_id TEXT REFERENCES components(id),
    root_run_name TEXT,
    start_time TIMESTAMP,
    end_time TIMESTAMP,
    total_runs INTEGER,
    total_tokens INTEGER,
    total_cost REAL,
    has_error BOOLEAN,
    latency_ms INTEGER
);

-- Runs (individual executions)
CREATE TABLE runs (
    id TEXT PRIMARY KEY,
    trace_id TEXT REFERENCES traces(id),
    parent_run_id TEXT,
    run_type TEXT,
    name TEXT,
    status TEXT,
    inputs TEXT,  -- JSON
    outputs TEXT, -- JSON
    error TEXT,   -- JSON
    start_time TIMESTAMP,
    end_time TIMESTAMP,
    model_name TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,
    cost REAL
);

-- Model pricing
CREATE TABLE model_costs (
    model TEXT PRIMARY KEY,
    input_cost_per_1k REAL,
    output_cost_per_1k REAL
);

Web Frontend Architecture

Component Hierarchy

App
├── Layout
│   ├── Sidebar (navigation)
│   └── Main Content
│       ├── TracesPage
│       │   ├── HierarchySidebar
│       │   ├── TraceFilters
│       │   └── TraceList
│       ├── TraceDetailPage
│       │   ├── TraceHeader
│       │   ├── RunTree
│       │   └── RunDetails
│       ├── ApplicationsPage
│       │   └── APIKeyManager
│       ├── StatsPage
│       │   ├── SummaryCards
│       │   └── Charts
│       └── SettingsPage
│           └── ModelCostEditor

State Management

TanStack Query for server state:

const { data, isLoading } = useQuery({
  queryKey: ['traces', filters],
  queryFn: () => fetchTraces(filters),
  staleTime: 30000,
});

const mutation = useMutation({
  mutationFn: deleteTrace,
  onSuccess: () => queryClient.invalidateQueries(['traces']),
});

Extension Points

Adding New Run Types

Client SDK: Add to RunType enum in models.py
API Server: No changes needed (generic handling)
Frontend: Add icon/color in run type mapping

Adding New Metadata Fields

Use extra field in events
API server stores in JSON column
Frontend displays in run details

Custom Cost Calculation

Modify CostCalculator service:

class CostCalculator:
    def calculate(self, model: str, tokens: dict) -> float:
        pricing = self.get_pricing(model)
        return (
            tokens["input"] * pricing.input_cost / 1000 +
            tokens["output"] * pricing.output_cost / 1000
        )

Adding New API Endpoints

Create router in api/routers/
Add service logic in api/services/
Register in main.py

# api/routers/custom.py
from fastapi import APIRouter

router = APIRouter(prefix="/api/custom", tags=["custom"])

@router.get("/endpoint")
async def custom_endpoint():
    return {"data": "value"}

# api/main.py
from api.routers import custom
app.include_router(custom.router)

Testing Strategy

Client SDK

Unit tests: Mock HTTP transport
Integration tests: Mock API server
Target: 80%+ coverage

def test_batch_queue_flush():
    queue = BatchQueue(batch_size=10)
    for i in range(10):
        queue.put(create_event(i))
    assert queue.should_flush() is True

API Server

Unit tests: Services in isolation
Integration tests: Endpoints with test database
Target: 80%+ coverage

async def test_trace_ingestion(client: AsyncClient):
    response = await client.post(
        "/api/traces/batch",
        headers={"X-API-Key": test_api_key},
        json={"events": [create_test_event()]}
    )
    assert response.status_code == 200

Web Frontend

Component tests: Individual components
Integration tests: Page interactions
E2E tests: Full user flows

test('TraceList renders traces', () => {
  render(<TraceList traces={mockTraces} />);
  expect(screen.getByText('ChatOpenAI')).toBeInTheDocument();
});

Performance Considerations

Client SDK

Batch size: 100 events (default)
Flush interval: 5 seconds (default)
Queue size: 10,000 events max
Compression: gzip for all batches

API Server

SQLite: Single writer, file-based
No connection pooling needed
Pagination for large result sets

Web Frontend

TanStack Query caching
Virtual scrolling for large lists
Lazy loading for pages

Security

API Key Management

Keys generated with cryptographically secure random
Stored as bcrypt hash
Only prefix shown after creation
Can be revoked (soft delete)

Input Validation

Pydantic models for all inputs
SQL injection prevented via parameterized queries
JSON size limits on trace data

CORS

Configurable allowed origins
Default: allow all for development

Debugging

SDK Debug Mode

import logging
logging.getLogger("llm_tracer").setLevel(logging.DEBUG)

# Or via environment
# LLM_TRACER_DEBUG=true

API Server Debug Mode

LOG_LEVEL=DEBUG uvicorn api.main:app --reload

Frontend DevTools

import { ReactQueryDevtools } from '@tanstack/react-query-devtools';

Common Issues

Event Loss

Increase max_queue_size
Ensure flush() called before exit
Check API server connectivity

Memory Growth

Reduce max_queue_size
Ensure background thread running
Check for event processing bottlenecks

Slow Queries

Add database indexes
Use pagination
Filter by date range

FilesExpand file tree

developer-notes.md

Latest commit

History

developer-notes.md

File metadata and controls

Developer Notes

Architecture Overview

Design Principles

1. Zero Latency Impact

2. Fail-Safe Operations

3. API Key Authentication

4. Wire Protocol Stability

Client SDK Architecture

Components

Key Classes

Event Flow

Token Extraction

Model Name Extraction

API Server Architecture

Routers

Services

Database Schema

Web Frontend Architecture

Component Hierarchy

State Management

Extension Points

Adding New Run Types

Adding New Metadata Fields

Custom Cost Calculation

Adding New API Endpoints

Testing Strategy

Client SDK

API Server

Web Frontend

Performance Considerations

Client SDK

API Server

Web Frontend

Security

API Key Management

Input Validation

CORS

Debugging

SDK Debug Mode

API Server Debug Mode

Frontend DevTools

Common Issues

Event Loss

Memory Growth

Slow Queries