LLM Live Monitor

A transparent, fail-open tap for OpenAI-compatible LLM streams. It sits between any OpenAI-compatible client (Hermes, Open WebUI, the OpenAI SDK, plain curl …) and any model server (llama.cpp, LM Studio, vLLM, or a cloud API), forwards every request and response verbatim, and taps the streaming response so you can watch reasoning, content, and tool calls live in your terminal.

Most clients and cloud APIs hide reasoning_content. Because the tap sits in the request path, it sees and rebroadcasts those tokens — you can watch the model think even when the client never shows it.

Features

• Zero patching — any OpenAI-compatible client just points its base_url at the tap. Nothing is modified, so upstream updates never break it.
• Verbatim, fail-open passthrough — every request/response is forwarded untouched. If the upstream errors, the error body is relayed straight back.
• Live reasoning tap — reasoning_content, content, and tool-call deltas are broadcast to WebSocket monitors in real time.
• Raw upstream log — a verbatim copy of everything the server sends, for debugging.
• Endpoint-agnostic — local servers and remote APIs alike; only the standard OpenAI streaming delta fields are required.

OpenAI-compatible client ──► LLM Live Monitor (:9000) ──► model server

Install

git clone https://github.com/cgpp5/LLM-Live-Monitor.git
cd llm-live-monitor
pip install .

For development (with tests and linting):

pip install -e ".[dev]"

Quick start

1. Run the tap

Point it at your model server. Either export the variable or put it in a .env file (see Configuration):

export UPSTREAM_URL=http://127.0.0.1:8080   # your model server (OpenAI-compatible)
uvicorn app.main:app --host 0.0.0.0 --port 9000

2. Attach the console monitor

python -m app.cli_monitor

The monitor connects to the tap over WebSocket and shows every model call as a live stream.

3. Make a request

With the tap running, send a chat completion through it:

curl http://localhost:9000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"any","messages":[{"role":"user","content":"hi"}],"stream":true}'

The monitor now shows the request, the model's reasoning, and its content as they stream.

4. Use it from your own client

Any OpenAI-compatible client can use the tap as its endpoint — just point its base_url at http://<host>:9000/v1. Example with the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:9000/v1", api_key="unused")

api_key can be any value — the tap forwards whatever it receives upstream.

Windows users: double-click launch_monitor.bat in the repo root to start the server and the console monitor together in one window. It reads the configuration from your .env file; the port defaults to 9000 (override with a PORT environment variable).

Configuration

The tap reads its settings from environment variables. Easiest way: create a .env file in the project root based on the shipped example:

cp .env.example .env
# then edit .env and set UPSTREAM_URL to your model server

.env is loaded automatically from the working directory and is never committed.

Variable	Default	Description
UPSTREAM_URL	http://127.0.0.1:8080	The model server behind the tap. Must be reachable from the tap host
UPSTREAM_KEY	(empty)	Optional bearer token sent upstream (empty = forward caller's auth)
TOKEN_QUEUE_LIMIT	200	Per-client asyncio queue size for monitor broadcasts
WS_KEEPALIVE_SEC	30	Seconds between keepalive pings; idle connections closed after 2× this value
DEBUG	false	Verbose logging: debug log level + full tracebacks
REDACT_ENABLED	true	Scrub credentials/PII from broadcasts and logs (emails, hex tokens, sensitive keys)
LOG_FILE_ENABLED	true	Write console logs to live_monitor.log
RAW_LOG_ENABLED	true	Write upstream responses verbatim to raw_upstream.log
SYSTEM_PROMPT_ENABLED	true	Broadcast the full system prompt in the request event; off = empty (saves bandwidth for large personas)

Log files are written to the working directory and are excluded from the repo (via .gitignore).

HTTP endpoints

Method	Path	Description
POST	/v1/chat/completions	Passthrough to the upstream server; stream tapped and broadcast
GET	/v1/models	Forwarded verbatim
POST	/v1/completions, /v1/embeddings	Forwarded verbatim
any	/{path}	Catch-all verbatim passthrough
GET	/metrics	Connections, tokens broadcast/dropped, active streams
WS	/ws	Live stream monitor endpoint

WebSocket protocol

Client → tap

{"type":"hello","subscribe":"all"}
{"type":"pong"}

Every client receives every model stream.

Tap → client

type	fields	meaning
request	run_id, system, prompt, model, ts	New model call started (system prompt + last user input)
reasoning_content	run_id, text, ts	Live reasoning delta
content	run_id, text, ts	Live content delta
tool_call	run_id, name, input, status, ts	Tool invocation (running/completed)
run_event	run_id, event, ts	Stream lifecycle: started, completed, failed, cancelled
dropped_tokens	run_id, count	Backpressure: N token frames dropped for this client
ping	—	Keepalive; reply with {"type":"pong"}

Project layout

app/
  main.py          FastAPI app: passthrough, stream tap, WS hub
  hub.py           WebSocket client registry, per-client queues, backpressure
  redact.py        Sensitive-value scrubber
  config.py        Env-var configuration
  cli_monitor.py   ANSI terminal monitor (console client)

tests/
  conftest.py      Singleton reset fixtures
  test_proxy.py    Unit + integration tests (upstream server mocked)
  mock_llama_server.py   Standalone mock upstream server (local testing)
  demo_ws_client.py      Minimal WebSocket console client

The mock_llama_server.py and demo_ws_client.py helpers are dev-only tools: they are not imported by the application and are useful when experimenting without a real model server.

Testing

pytest -v

Tests cover redact(), hub backpressure (dropped_tokens), the streaming passthrough (verbatim forwarding + reasoning/content broadcast to a WebSocket client against a mocked upstream server), non-streaming passthrough, /v1/models, /metrics, and the raw log.

Security notes

• The tap sits in the critical model path: if it stops, your client cannot reach the model.
• Broadcast events are scrubbed by redact() unless REDACT_ENABLED=false — emails, long hex strings, and keys named secret, api_key, password, token, authorization, auth, credential.
• Only set REDACT_ENABLED=false while debugging redaction itself; keep it on otherwise.

License

GPL-3.0-or-later. See LICENSE.