Skip to content

cli-reference.md

Latest commit

 

History

History
375 lines (300 loc) · 15.9 KB

cli-reference.md

File metadata and controls

375 lines (300 loc) · 15.9 KB
Raw

CLI reference

Every flag, exit code, and JSON field, for scripting tool-eval-bench in CI or driving it from an agent. If you are running the benchmark by hand for the first time, start with the README instead.

What this tool does

tool-eval-bench evaluates LLM tool-calling quality using 69 deterministic scenarios across 16 categories. It produces a 0–100 score with per-category breakdowns, safety warnings, and full conversation traces.

Quick start (zero-config)

If an inference server is running on localhost at a standard port, no configuration is needed:

# Auto-discovers server and model, runs core 15 scenarios
tool-eval-bench --short --json

# Full 69 scenarios
tool-eval-bench --json

Auto-discovery probes these ports in order: 8000 (vLLM), 8080, 8081, 8082, 30000 (SGLang), 4000 (LiteLLM), 3000, 11434 (Ollama), 5000 (TGI).

Headless / JSON mode

Always use --json for machine-readable output. In this mode:

  • stdout contains only the JSON result envelope
  • stderr contains JSONL progress events (one per line)
  • Interactive prompts are skipped (first model is auto-selected)
  • Warmup and informational banners are suppressed
# JSON to stdout
tool-eval-bench --json --short

# JSON to file (keeps stdout clean for logging)
tool-eval-bench --json-file results.json --short

Server readiness check

Use --probe to wait for the server to be ready before benchmarking:

tool-eval-bench --probe --base-url http://localhost:8000
# Exit 0 = ready, exit 1 = not reachable

In a script:

until tool-eval-bench --probe --json 2>/dev/null; do
  sleep 5
done
tool-eval-bench --json --short

Exit codes

Code Meaning
0 Success
1 Runtime error or server not ready (--probe)
2 Connection/HTTP error (server unreachable, bad response)
3 No models found on server

Specifying the server

Priority order (highest wins):

  1. --base-url CLI flag
  2. TOOL_EVAL_<PROVIDER>_BASE_URL, when --provider or TOOL_EVAL_PROVIDER names one
  3. TOOL_EVAL_BASE_URL environment variable
  4. TOOL_EVAL_HOST + TOOL_EVAL_PORT environment variables
  5. .env file (never overrides existing env vars)
  6. Auto-discovery on localhost
# Explicit URL
tool-eval-bench --base-url http://192.168.1.5:8000 --json --short

# Via environment
export TOOL_EVAL_BASE_URL=http://192.168.1.5:8000
tool-eval-bench --json --short

# Named provider: TOOL_EVAL_OPENAI_BASE_URL / _API_KEY / _MODEL
tool-eval-bench --provider openai --json --short

A provider owns its API key and model: TOOL_EVAL_API_KEY and TOOL_EVAL_MODEL are not consulted when one is selected, and a provider with no _BASE_URL is an error rather than a fall-through to auto-discovery. --api-key and --model still override. gemini, openai, and anthropic set the backend label and skip engine probing; any other name leaves detection alone.

A provider also owns its extra headers: TOOL_EVAL_<NAME>_HEADERS and TOOL_EVAL_<NAME>_SESSION_HEADER replace the generic TOOL_EVAL_HEADERS and TOOL_EVAL_SESSION_HEADER when one is selected. --header adds to whichever set applies and wins on a name clash; --session-header overrides. Every request carries User-Agent: tool-eval-bench/<version> unless a header replaces it.

Key CLI flags

Flag Purpose
--json Machine-readable JSON output
--json-file PATH Write JSON to file instead of stdout
--short Run core 15 scenarios (~2 min) instead of full 69
--probe Check server readiness and exit
--dry-run List scenarios that would run (no server needed)
--base-url URL Server endpoint
--model NAME Model name (auto-detected if omitted)
--backend NAME Backend label for reports: vllm, litellm, llamacpp, sglang, gemini, openai, anthropic, ninfer, tensorfold, halogen, unknown. Auto-detected when omitted; otherwise unknown.
--provider NAME Read the endpoint from TOOL_EVAL_<NAME>_* env vars
--header NAME=VALUE Extra request header, repeatable (env: TOOL_EVAL_HEADERS, ;-separated)
--session-header NAME Header that carries a per-conversation id, e.g. x-opencode-session (env: TOOL_EVAL_SESSION_HEADER)
--seed N Random seed for reproducibility
--temperature F Sampling temperature (default: 0.0 = greedy)
--timeout F Per-request timeout in seconds (default: 60)
--no-think Disable thinking/reasoning (critical for Qwen3/DeepSeek)
--no-warmup Skip server warm-up request
--hardmode Include 23 Hard Mode scenarios (Category P)
--categories A B K Run only specific categories (A–P)
--scenarios TC-01 TC-07 Run specific scenario IDs
--perf Also run throughput benchmark
--trials N Run N trials for statistical analysis
--resume RUN_ID Resume a previous run (skip already-passed scenarios)
--hardmode-only Run ONLY Hard Mode scenarios (equivalent to --hardmode --categories P)
--weight-by-difficulty Weight scores by difficulty tier (harder scenarios count more)
--system-prompt TEXT Replace the built-in system prompt with TEXT for every scenario
--system-prompt-file PATH Read the system prompt override from a UTF-8 file, max 32 KiB (exclusive with --system-prompt)
--decision-judge-base-url URL Independent /v1/systemone endpoint for TC-89 answer auditing; official scores stay unchanged
--decision-judge-model NAME Required judge model; authentication uses TOOL_EVAL_DECISION_JUDGE_API_KEY, never the benchmark's key
--alpha F Quality weight in the deployability composite (default: 0.7; speed gets 1 − alpha)

Benchmark modes

These run alongside the tool-call scenarios, or on their own with --skip-tool-eval (--perf-only for throughput). Method and metric details live in benchmarks, speculative-decoding, and context-pressure.

Flag Purpose
--perf Throughput benchmark (llama-benchy style pp/tg sweep)
--perf-only Throughput only, skip the scenarios
--pp N, --tg N Prompt and generation tokens per request (default: 2048, 128)
--depth LIST Context depths, comma separated (default: 0,4096,8192)
--concurrency LIST Concurrency levels for --perf (default: 1,2,4)
--benchy-runs N Measurement runs per throughput test point (default: 3)
--spec-bench Speculative-decoding benchmark: effective t/s, acceptance rate, τ, per-position acceptance
--spec-method M Method hint when the server does not identify it (--help lists the choices)
--spec-prompts LIST Prompt types (default: filler,code,structured); labels from --spec-prompt-file are accepted
--spec-prompt-file PATH Your own workload: one prompt per line, or JSON lines with prompt and an optional label
--spec-runs N Measurements per depth × prompt cell, pooled into one row (default: 3)
--baseline-tgs F Known no-spec tg t/s for a measured speedup; without it a ceiling is derived from verify steps/s
--metrics-url URL Prometheus /metrics when the API sits behind a proxy
--spec-live Live dashboard of speculative-decoding metrics (also the spec-live subcommand)
--spec-live-interval S Seconds between scrapes (default: 1.0)
--context-pressure R Fill the context to ratio R (0–1) before each scenario
--context-pressure-sweep A-B Sweep pressure from A to B in --sweep-steps levels (default: 5)
--context-size N Override the detected context window

--temperature applies to spec-bench requests as well; greedy is the default and the report records the value, since acceptance falls as sampling temperature rises.

Auditing payment claims

Supply both --decision-judge-base-url and --decision-judge-model to audit TC-89's payment-claim check. All other scenarios, including held-out packs, remain unaudited. The flags apply to run, resume, and flat scenario invocations. Keep the same connection flags when resuming an audited run. Results include a separate decision_audit; no audit changes points or safety warnings. See answer audits for input limits, abstention, credential isolation, and persisted evidence.

Overriding the system prompt

--system-prompt and --system-prompt-file replace the built-in persona for every scenario in the run. Both forms are equivalent: surrounding whitespace and a leading byte-order mark are stripped, so the same words read from a file and passed inline produce the same run.

The benchmark reference-date line is appended after the override and remains authoritative — an override that asserts its own "today" does not move the date that relative-time scenarios are graded against.

An override is recorded in the run config and folded into config_fingerprint, so runs with different prompts are not compared with each other, and --resume refuses to continue a run whose prompt differs — including a run recorded before this option existed, which is treated as having used the built-in prompt. A run without either flag persists and fingerprints exactly as it did before the option existed.

The prompt must be valid UTF-8, non-blank, and at most 32 KiB, because it is stored verbatim in the run config and metadata. Reports mark only that a custom prompt was used; they never reproduce it. The flags are accepted but ignored, with a warning, by invocations that run no tool-call scenarios (--perf-only, --spec-bench alone, a plugin run, or --skip-tool-eval).

Accuracy benchmarks (pluggable)

In addition to the 69-scenario tool-call benchmark, tool-eval-bench supports accuracy benchmarks via the same OpenAI-compatible endpoints. These only require /v1/chat/completions — no tools support needed.

# GSM8K — math reasoning (1,319 questions, 8-shot)
tool-eval-bench --gsm8k-only --gsm8k-limit 50

# MMLU — multitask knowledge (14,042 questions, 5-shot)
tool-eval-bench --mmlu-only --mmlu-limit 50

# IFEval — instruction following (541 prompts, 25 constraints)
tool-eval-bench --ifeval-only --ifeval-limit 20

# Needle in a haystack — long-context retrieval (no dataset download)
tool-eval-bench --needle-only --needle-lengths 4 --needle-depths 5

# Decision models — llama.cpp /v1/systemone (no dataset download, no chat preflight)
tool-eval-bench --decision-only --base-url http://host:8084

# Live canary monitor for a decision model (runs until Ctrl+C)
tool-eval-bench decision-live --base-url http://host:8084

# Chained onto a full sweep, like --perf
tool-eval-bench --hardmode --seed 42 --perf --needle

# Run all accuracy benchmarks (skip tool-call scenarios)
tool-eval-bench --gsm8k-only --mmlu-only --ifeval-only

GSM8K, MMLU, and IFEval datasets are downloaded from HuggingFace on first use and cached locally. Install pip install tool-eval-bench[hf] for rate-limit-free downloads. The needle benchmark generates its own cases and downloads nothing.

Flag Purpose
--gsm8k-only GSM8K math reasoning
--mmlu-only MMLU multitask knowledge
--ifeval-only IFEval instruction following
--needle-only Needle-in-a-haystack retrieval
--decision-only Decision-model scoring on /v1/systemone
--gsm8k-limit N Limit questions (default: 200)
--mmlu-limit N Limit questions (default: 500)
--ifeval-limit N Limit prompts (default: all 541)
--needle-lengths N Haystack sizes to probe (default: 4)
--needle-depths N Needle depths to probe (default: 5)

See needle.md for the retrieval grid and how to read it, and decision-models.md for the decision-model report.

Understanding the JSON output

{
  "schema_version": "1",
  "tool_eval_bench_version": "1.8.0",
  "final_score": 85,           // 0–100, the headline number
  "rating": "★★★★ Good",       // star rating with label
  "safety_warnings": [],       // empty = safe; non-empty = failures in safety-critical scenarios
  "deployability": 78,         // alpha × final_score + (1 − alpha) × responsiveness; null without latency data
  "responsiveness": 65,        // 0–100 from median turn latency; see methodology.md
  "total_scenarios": 69,
  "run_id": "2026-05-07T...",
  "config": { ... },
  "scores": {
    "final_score": 85,
    "category_scores": [ ... ],
    "scenario_results": [ ... ]
  }
}

JSONL progress events (stderr)

Each line on stderr is a JSON object:

{"event": "server_discovered", "base_url": "http://localhost:8000", "backend": "vllm", ...}
{"event": "model_auto_selected", "model": "Qwen/Qwen3-8B", ...}
{"event": "scenario_start", "scenario_id": "TC-01", "index": 0, "total": 15}
{"event": "scenario_result", "scenario_id": "TC-01", "status": "pass", "points": 2, ...}
{"event": "benchmark_complete", "json_file": "results.json", "final_score": 85}
{"event": "error", "error": "no_server", "message": "..."}

Error codes (structured)

When --json mode emits an error event, the error field is one of:

Code Exit Meaning
connection_failed 2 TCP connection refused, DNS failure, or timeout
http_error 2 Server responded with 4xx/5xx status
detection_failed 2 Unexpected exception during server probing
invalid_response 2 Response body is not valid JSON
no_models 3 Server responded but model list is empty
no_server 2 Auto-discovery found no server on localhost

These constants are defined in tool_eval_bench.domain.errors for programmatic consumers.

Programmatic API (Python)

import asyncio
from tool_eval_bench.api import run_benchmark

result = asyncio.run(run_benchmark(
    model="Qwen/Qwen3-8B",
    base_url="http://localhost:8000",
    short=True,
    persist=False,  # skip SQLite/Markdown artifacts
))

print(result["final_score"])  # 85

Interpreting results for automated decisions

import json, subprocess

r = subprocess.run(
    ["tool-eval-bench", "--json", "--short"],
    capture_output=True, text=True,
)
if r.returncode != 0:
    print("Benchmark failed")
else:
    data = json.loads(r.stdout)
    score = data["final_score"]
    warnings = data.get("safety_warnings", [])

    if warnings:
        print(f"UNSAFE: {len(warnings)} safety failures")
    elif score >= 75:
        print(f"GOOD: score {score}")
    elif score >= 60:
        print(f"ADEQUATE: score {score}")
    else:
        print(f"POOR: score {score}")

Score tiers

Score Rating Meaning
90–100 ★★★★★ Excellent Production-ready tool calling
75–89 ★★★★ Good Reliable for most agentic tasks
60–74 ★★★ Adequate Works but has notable gaps
40–59 ★★ Weak Significant tool-calling issues
0–39 ★ Poor Not suitable for agentic use

Common pitfalls

  1. Don't parse stderr as JSON — it's JSONL (one object per line), not a single JSON document.
  2. Don't assume model is auto-detected — if the server has 0 models loaded, exit code is 3.
  3. Use --short for fast checks — the full suite takes 10–20 minutes; core 15 takes ~2 minutes.
  4. Timeout with thinking models — models like Qwen3 with thinking enabled may need --timeout 120.
  5. Warmup is automatic — the first request primes the server. Use --no-warmup only if already warmed.
  6. Use --dry-run to preview — before committing to a long run, check which scenarios would execute.

Controlled scenario fixtures

--variant-seed INTEGER selects versioned deterministic scenario variants independently of the sampling --seed. Unvaried scenarios remain controls. Variants are persisted in run configuration and comparison fingerprints; resume requires the same variant seed for affected scenarios. See variant coverage.