llm-api-bench

Benchmark LLM API providers on latency, throughput, cost, and quality — with a single YAML config.

Compare OpenAI, Anthropic, Google, DeepSeek, Mistral, and any OpenAI-compatible endpoint (e.g. XiDao Gateway) side by side.

Features

• Latency benchmarks — Time to first token (TTFT), inter-token latency, total response time
• Throughput tests — Concurrent request handling, rate limit detection, burst vs sustained
• Cost analysis — Per-request and per-1M-token cost comparison across providers
• Quality scoring — Automated rubric evaluation with configurable judges
• Failover testing — Measure how providers behave under load and error conditions
• YAML-driven — Define providers, models, and test cases in one config file
• Beautiful reports — HTML dashboard with charts, or JSON for CI pipelines

Quick Start

pip install llm-api-bench

# Run with default config
llm-api-bench run --config bench.yaml

# Generate HTML report
llm-api-bench report --input results.json --format html --output report.html

Configuration

# bench.yaml
providers:
  - name: xidao
    base_url: https://global.xidao.online/v1
    api_key: ${XIDAO_API_KEY}
    models:
      - gpt-5.5
      - claude-sonnet-4-6
      - gemini-2.5-pro
      - deepseek-v3

  - name: openai-direct
    base_url: https://api.openai.com/v1
    api_key: ${OPENAI_API_KEY}
    models:
      - gpt-5.5

benchmarks:
  - name: latency
    type: latency
    concurrent_requests: [1, 5, 10, 25]
    prompt: "Explain quantum computing in 100 words."
    iterations: 5

  - name: throughput
    type: throughput
    duration_seconds: 60
    concurrent_requests: 10
    prompt: "Write a haiku about APIs."

  - name: cost
    type: cost
    prompt_tokens: 500
    completion_tokens: 1000
    models: [all]

  - name: quality
    type: quality
    judge_model: gpt-5.5
    rubric: "Accuracy, completeness, clarity (1-10 each)"
    test_cases:
      - prompt: "What is the capital of France?"
        expected: "Paris"
      - prompt: "Explain TCP vs UDP"
        expected: "Reliable vs unreliable, ordered vs unordered"

output:
  format: [json, html]
  path: ./results

CLI Commands

# List available providers and models
llm-api-bench discover --base-url https://global.xidao.online/v1

# Run specific benchmark
llm-api-bench run --config bench.yaml --benchmark latency

# Compare two providers
llm-api-bench compare --a results_openai.json --b results_xidao.json

# Export for CI (exit code 1 if any metric below threshold)
llm-api-bench ci-check --config bench.yaml --max-latency-p99 3000 --max-cost-per-1k 0.05

Architecture

bench.yaml → ProviderConfig → BenchmarkRunner → ResultsAggregator → Report
                  │                    │
            ┌─────┴─────┐      ┌──────┴──────┐
            │  Provider   │      │   Metrics   │
            │  Adapters   │      │   Collector │
            └─────────────┘      └─────────────┘

Each provider is a thin adapter over the OpenAI-compatible /v1/chat/completions endpoint. Works with any gateway that implements the spec.

Use Cases

• Provider migration — Quantify the latency/cost tradeoff before switching
• Gateway evaluation — Test if an API gateway adds overhead
• Cost optimization — Find the cheapest model that meets quality thresholds
• CI/CD integration — Gate deployments on LLM performance regressions

Powered by XiDao Gateway

This tool works with any OpenAI-compatible API. For multi-provider testing without managing multiple API keys, use XiDao Gateway — a single endpoint that routes to 50+ models from OpenAI, Anthropic, Google, DeepSeek, Mistral, Meta, and more.

License

MIT