Glasspane: LLM Observability Proxy

Glasspane is a lightweight, high-fidelity LLM observability proxy service designed for real-time logging, analysis, and diagnostics of LLM traffic.

It is set up as a monorepo containing:

• /backend: A FastAPI proxy service that intercepts LLM calls, forwards them transparently to the Groq API (OpenAI-compatible), logs details (prompts, completions, latency, status code, tokens) to ClickHouse asynchronously via Redis Streams, and evaluates response quality (relevance, hallucinations) in a non-blocking queue.
• /frontend: A React + Vite dashboard displaying live-updating aggregate stats (including average relevance/hallucination), interactive quality and drift metrics timeline charts (SVG), detailed trace lists, and a searchable/expandable trace explorer table with score progress bars.

---

Getting Started

1. Pre-requisites

• Python 3.11+
• Node.js (v18+) & npm

2. Backend Setup

Install the Python dependencies using uv, ensure a local Redis server is running, and launch the server processes:

# Install dependencies
uv sync

# Navigate to backend
cd backend

# Run the FastAPI proxy server
uv run uvicorn main:app --reload --port 8000

# In a separate terminal, run the Redis Stream consumer
uv run uvicorn consumer:app --reload --port 8001

The backend proxy server will run on http://localhost:8000, and the log consumer service will run on http://localhost:8001. The backend logs will be stored in a local ClickHouse server on port 8123.

3. Frontend Setup

Navigate into the /frontend folder, install npm dependencies, and start the development server:

# Navigate to frontend
cd frontend

# Install react & vite
npm install

# Run frontend development server
npm run dev

The React dashboard will be available at http://localhost:5173.

---

How to Route LLM Traffic Through the Proxy

You only need to change the base_url (and optionally authorization tokens) in your code to direct your LLM requests through the proxy.

Python OpenAI/Groq SDK Client Override

Instead of pointing to Groq's default URL, configure the base_url to direct requests to Glasspane:

from openai import OpenAI

# Initialize the client pointing at Glasspane's proxy port
client = OpenAI(
    base_url="http://localhost:8000/openai/v1",
    api_key="your_groq_api_key" # The proxy forwards this to the actual Groq API
)

# Non-streaming request (Logged in ClickHouse)
response = client.chat.completions.create(
    model="llama3-8b-8192",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "What is Glasspane?"}
    ],
    stream=False
)
print(response.choices[0].message.content)

# Streaming request (Parsed, reconstructed, and logged in ClickHouse on completion)
stream = client.chat.completions.create(
    model="llama3-8b-8192",
    messages=[{"role": "user", "content": "Count from 1 to 5"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

---

Example API / cURL Commands

You can query the observability proxy endpoints directly or trigger calls to verify operations.

1. Trigger a Proxy Completion Request (Non-streaming)

curl -X POST http://localhost:8000/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_GROQ_API_KEY" \
  -d '{
    "model": "llama3-8b-8192",
    "messages": [{"role": "user", "content": "Hello! Explain what a proxy is in 10 words."}],
    "stream": false
  }'

2. Trigger a Proxy Completion Request (Streaming)

curl -X POST http://localhost:8000/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_GROQ_API_KEY" \
  -d '{
    "model": "llama3-8b-8192",
    "messages": [{"role": "user", "content": "Count from 1 to 5."}],
    "stream": true
  }'

3. Fetch Recent Telemetry Logs

Returns the most recent logged request/response pairs (defaults to 50):

curl http://localhost:8000/logs?limit=10

4. Fetch Details of a Single Trace

Returns full logged record by its UUID trace_id:

curl http://localhost:8000/logs/YOUR_TRACE_UUID

5. Fetch Aggregated Metrics

Returns metrics like total requests, error rate, average latency, token consumption, and average evaluation scores:

curl http://localhost:8000/stats

Automated Evaluation (LLM-as-a-Judge)

Whenever an LLM chat completion is successfully routed through Glasspane and returns a 200 OK status, a background worker asynchronously evaluates the prompt-response interaction using Groq's llama-3.1-8b-instant model.

Two metrics are calculated and logged:

1. Relevance Score: Represents how well the response answers the prompt.

   • 1.00: Perfectly relevant, directly answers the prompt or complies with constraints.
   • 0.75: Relevant, but misses minor details or includes minor unnecessary text.
   • 0.50: Partially relevant, off-topic in parts.
   • 0.25: Mostly irrelevant, fails to answer the main prompt.
   • 0.00: Completely irrelevant or gibberish.

2. Hallucination Score: Indicates the severity of false details or fabricated claims in the response.

   • 1.00: Severe hallucinations, completely fabricated or factually incorrect.
   • 0.75: Significant hallucinations or factually incorrect claims.
   • 0.50: Moderate hallucinations, contains some unsupported/made-up claims.
   • 0.25: Minor hallucinations, slightly misleading but mostly grounded.
   • 0.00: No hallucinations, completely grounded and factually correct, or complies with non-factual constraint.

These scores are stored as float values (between 0.0 and 1.0) in the database, displayed inline in the Traces Table, visualized as progress bars, plotted on the Quality & Drift Trends SVG Line Chart on the dashboard, and averaged under overview KPI cards.

---

Manual Verification & Testing

To ensure the proxy is working correctly, perform the following steps:

1. Verify Backend Offline Log Handling: Fire a curl request with an invalid/empty key or structure:

   curl -X POST http://localhost:8000/openai/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3-8b"}'

   Confirm that the error response is returned and immediately logged. Check http://localhost:8000/logs to see the logged failed trace.

2. Verify Live Updates in Dashboard: Open http://localhost:5173 in your browser. Ensure the connection badge displays "Active Proxy" (green indicator). Trigger a few requests using the cURL examples above and watch the Stats Cards and the Traces table update live.

3. Verify Trace Drill-down: Click on any row inside the traces table in the dashboard to expand it. You should see the pretty-printed JSON payloads for the request and response.

4. Run System Benchmarks: Compare the legacy synchronous logging + LLM evaluation processing against the decoupled Redis Stream queuing using the benchmark script:

   python benchmark.py

---

Benchmarks

By implementing a decoupled queueing architecture with Redis Streams, Glasspane achieves significant performance improvements by offloading high-I/O database writes and network-bound quality evaluations from the main request path:

• Proxy Latency Overhead: Reduced proxy-induced overhead from ~1027.3ms (legacy sync SQLite insert + blocking HTTP evaluation call) to ~1.1ms (asynchronous Redis Stream XADD under persistent connection), a 99.9% reduction in ingestion latency.
• Ingestion Throughput: Scaled maximum ingestion throughput by 90x+ (from ~1.0 rps when blocked by synchronous DB and API processes to ~900+ rps under parallel streaming).
• Resource Decoupling: Offloaded 100% of disk writes and evaluation request payloads to an asynchronous consumer service, eliminating event loop blocking.
• Data Reliability: Leveraged Redis Streams consumer group acknowledgements (XACK), guaranteeing zero loss of LLM telemetry even during database downs or restarts.