bottleneck

Find out why your LLM API calls are slow — model latency, rate limits, retries, or oversized prompts — instead of guessing.

bottleneck instruments OpenAI-compatible LLM API calls (OpenAI, NVIDIA NIM, Anthropic, OpenRouter, or any provider speaking the OpenAI /chat/completions shape) and tells you exactly where the time went on every call. Local-first: no server, no account, no dashboard to sign up for — just a JSONL file on disk and a CLI report.

Table of contents

• The problem
• Install
• Quickstart
• Real example
• What gets recorded
• CLI reference
• How it compares
• Roadmap
• Contributing

The problem

Your AI app is slow. Is it:

• the model itself taking longer to generate?
• rate limiting (429) forcing silent retries?
• a server error (5xx) or a timeout triggering a retry loop?
• a prompt that's grown too large without you noticing?

Most people guess. bottleneck measures it, per call, and shows you which one it actually was.

Install

git clone https://github.com/blueforgeai-svg/bottleneck
cd bottleneck
pip install -e .

Single dependency: httpx. Python 3.10+.

Quickstart

Wrap the HTTP call you're already making:

import httpx
from bottleneck import Tracker

tracker = Tracker()  # writes to .bottleneck/events.jsonl by default
client = httpx.Client()

payload = {"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
resp = tracker.request(
    client, "POST", "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": "Bearer ..."}, json=payload,
)

tracker.request retries on 429/5xx/timeouts with exponential backoff (or the Retry-After header if the provider sends one), then logs one event: latency, retry count/reason, time spent in backoff vs. actual response wait, and prompt/response size.

Building the request yourself (e.g. inside an SDK's internals)? Use the decorator form instead:

@tracker.track(model="claude-3-opus")
def call():
    return client.post("https://api.anthropic.com/v1/messages", json=payload)

resp = call()

Then see the breakdown:

bottleneck report

Real example

Pulled from wiring bottleneck into a live pipeline calling NVIDIA NIM (meta/llama-3.1-70b-instruct):

Total calls: 3
p50 latency: 0.554s
p95 latency: 7.040s
Time lost to retry/backoff: 0.0%

Top 5 slowest calls:
  7.761s  model=meta/llama-3.1-70b-instruct  retries=0  reason=none
  0.554s  model=meta/llama-3.1-70b-instruct  retries=0  reason=none
  0.527s  model=meta/llama-3.1-70b-instruct  retries=0  reason=none

retries=0 and reason=none on the 7.76s call rules out rate limiting, server errors, and timeouts as the cause — the tool proves it wasn't your retry logic, so you stop tuning backoff and look elsewhere (provider-side variance, in this case). That's the whole value: turning "the model felt slow" into "here's what it definitely wasn't, and here's the number."

What gets recorded

Field	Meaning
latency_s	total wall-clock time for the call, including retries
retry_count / retry_reasons	number of retries and why (rate_limit, server_error, timeout)
backoff_s	time spent sleeping between retries
wait_s	time spent actually waiting on a response (latency_s - backoff_s)
prompt_chars / response_chars	raw character counts
prompt_tokens_est / response_tokens_est	chars / 4, a rough estimate — not a real tokenizer
status_code	final HTTP status of the call

Events are appended as JSON lines to .bottleneck/events.jsonl (path configurable per-Tracker).

CLI reference

bottleneck report                  # reads .bottleneck/events.jsonl
bottleneck report --path FILE.jsonl  # point at a different log

Prints: total calls, p50/p95 latency, % of total time lost to retry/backoff, and the 5 slowest calls with their retry reason.

How it compares

	bottleneck	LangSmith / Helicone	Datadog / generic APM
Setup	one wrapper call, local file	account + SDK + dashboard	agent install + dashboards
Cost	free, open source	usage-based pricing	enterprise pricing
Scope	retry/backoff vs. model-wait breakdown for LLM calls	full tracing, evals, prompts	general infra observability, not LLM-specific
Best for	"why was this one call slow," fast local debugging	production LLM app observability at scale	teams already on an APM stack

bottleneck isn't trying to replace those — it's the tool you reach for before you're ready to wire up a hosted observability platform, or when you just want the answer locally in 30 seconds.

Roadmap

Not built yet, in rough priority order:

• Per-model latency baselines — track wait_s distribution per model over time so a report can say "this model is normally ~0.5s but spikes to 7s+ in ~10% of calls" instead of a one-off number. (Came directly from asking "why did this one call take 7s" and realizing the tool can rule out causes but can't yet characterize normal variance.)
• Time-to-first-token for streaming responses — mentioned in the original design, not yet implemented.
• Real token counts via each provider's actual tokenizer, replacing the chars/4 estimate.
• Cost estimates using per-model pricing tables.
• Opt-in hosted dashboard for teams who want trends across a shared log, instead of just a local CLI report.

No hosted dashboard, no auto-instrumentation/monkeypatching, and no multi-language SDKs in v1 — see .claude/agents/bottleneck-builder.md for the full product scope and reasoning behind what's deliberately out.

Contributing

Issues and PRs welcome. Keep changes to the smallest diff that solves the problem — this project intentionally stays small.

License

MIT