diff --git a/DISCOVERY.md b/DISCOVERY.md index 9c07691..043f40c 100644 --- a/DISCOVERY.md +++ b/DISCOVERY.md @@ -17,7 +17,8 @@ Everything below was observed directly. Metrics not listed here are not shown as Workload at probe time: a TP=2 inference job (`serve_stats.py`, container `c5-2B-r0`, `nvcr.io/nvidia/pytorch:26.07-py3`) using ~86 GB of GPU-allocated memory and ~90% GPU utilisation **on both nodes**. Heavy benchmarks would disrupt it, so at first the benchmark suite was verified in dry-run and demo mode -only. Every benchmark was later run on both idle nodes: see "Measured on hardware (2026-10-07)" below. +only. Every benchmark was later run on both idle nodes: see "Measured on hardware (2026-10-07)" below. The +tool-call eval (`tool-eval`), added on 2026-10-21, has not run against a real model endpoint yet. ## GPU — NVIDIA GB10 (Blackwell, CC 12.1, 6144 CUDA cores) @@ -122,6 +123,7 @@ counters come from `.../counters/` and `.../hw_counters/` (e.g. `symbol_error`, | nccl-tests (`all_reduce_perf`) | ❌ | ❌ | Not used: it needs an MPI launcher across nodes. The NCCL benchmark uses `torch.distributed` (NCCL backend) in the PyTorch container and reports the same algbw/busbw | | `fio` | ❌ | ❌ | Uses fio if installed. Otherwise a built-in O_DIRECT runner, clearly labelled | | LLM engine | vLLM images present, custom servers | same | Engine-agnostic: an OpenAI-compatible endpoint with streaming TTFT/decode timing | +| tool-eval-bench (PyPI) | ✅ 2.8.0 in the dev venv | — | The tool-call eval imports it from the agent's venv; `install.sh` installs it hash-pinned from `server/requirements.lock`. The endpoint needs tool calling enabled (vLLM `--enable-auto-tool-choice --tool-call-parser …`) | | `sudo` without a password | ❌ | — | `install.sh` must be run with sudo (`--dry-run` and `--destdir` work without). Dev runs use user-space processes | ## Measured on hardware (2026-10-07) diff --git a/PLAN.md b/PLAN.md index ed126eb..62f26c5 100644 --- a/PLAN.md +++ b/PLAN.md @@ -132,6 +132,7 @@ Rollups store avg/min/max per key per bucket. `energy_j` stores the last value, | `rdma-bw-lat` | `ib_write_bw` / `ib_write_lat` over the CX7 rail | perftest (installed) | 2 nodes, server/client orchestration | | `storage-fio` | seq/rand read/write profiles | `fio` if present, else built-in O_DIRECT runner (labelled) | test file in data dir, removed after | | `llm-inference` | TTFT, prefill tok/s, decode tok/s, memory footprint | OpenAI-compatible endpoint (vLLM/TRT-LLM/llama.cpp server), streaming | engine-agnostic | +| `tool-eval` | tool-calling score (pass/partial/fail per multi-turn scenario with mock tools) | tool-eval-bench 2.8.0 as a library in a host payload, OpenAI-compatible endpoint | needs the server's tool-call parser | | `thermal-soak` | 10–30 min sustained GEMM; clocks, throttling, temps over time | PyTorch container | heavy | Each run records driver/CUDA/NVML versions, GPU name/UUID, clocks and temps at start and end, agent version, suite git hash, @@ -140,3 +141,30 @@ hostnames, params, and timestamps. Regression = a result worse than the baseline ## Phases (a) agent + console + streaming + demo · (b) web shell + design system · (c) live views · (d) benchmarks · (e) polish / a11y / perf / tests / 24 h soak. Each phase is committed separately. + +## 2026-10-21: concurrency streams and tool-call eval + +- **live/detail on NodeBenchState.** `live` is a bounded snapshot of what the payload is doing, replaced by each + `{"live": …}` line; `detail` holds per-run records, and each `{"detail": …}` line replaces only its top-level + keys. The agent drops a snapshot over 256 KiB, a detail update that would pass 1 MiB in all, values nested over + 64 levels and non-objects (logged once per job). Both are saved with the run (and its JSON export) and stripped + from `list()`; their defaults keep older runs and agents valid (SCHEMA_VERSION stays 1). +- **Streams.** A level runs L stream slots in a closed loop (a slot sends its next request when its last one + ends). `sweep` runs 1, 2, 4 … streams up to `concurrency`; every level runs max(`requests`, L) requests, also + without the sweep. Snapshots every 0.25 s; `detail.llm` gets each finished level's summary and request + timeline. On vLLM, `stream_options.continuous_usage_stats` keeps live token counts exact when one chunk holds + several tokens (MTP, speculative decoding). +- **Baselines.** `params_key` leaves out params added after release while they hold their neutral default + (`KEY_NEUTRAL`, so far `sweep: false`), so existing llm-inference baselines still match. +- **Clean SIGTERM.** The LLM payload stops its workers and shuts their sockets so blocked reads return, sends a + last snapshot and exits within about a second instead of waiting for SIGKILL; tool-eval cancels its task and + exits 143 after a last snapshot. +- **Tool-call eval.** tool-eval-bench runs as a library in a host payload (`tool_eval.py`, the agent venv's + Python) through `api.run_benchmark(persist=False)`, with callbacks that never raise. `tool-eval-bench==2.8.0` + comes from PyPI, hash-pinned in `requirements.lock`, and runs with upstream's defaults, so scores match its CLI + and sparkrun. A preflight tool call fails the run when the server rejects tool calls; a run with nothing graded + fails, and exclusions are reported as the share graded. Scores compare only within one version and config + fingerprint (both in the score's extras). +- **Demo.** LLM runs play at 2× real time with every level 4–60 s on screen, so the streams can be watched; + tool-eval uses upstream's scenario registry and scoring. The seeded history has a finished 32-stream sweep and + a finished evaluation. diff --git a/README.md b/README.md index 9171d18..ad741c1 100644 --- a/README.md +++ b/README.md @@ -122,10 +122,13 @@ The console picks up the agent token from the agent's config on the same host. | ![Unified memory breakdown on spark1, light theme](docs/screenshots/node-memory-light.png) | ![Per-core heatmap of spark1's performance and efficiency cores](docs/screenshots/node-cpu-dark.png) | | ![ConnectX-7 rails on spark1](docs/screenshots/network-dark.png) | ![Alerts: open alerts and the event timeline](docs/screenshots/alerts-dark.png) | | ![A GPU matmul benchmark running, with live telemetry](docs/screenshots/bench-run-dark.png) | ![Two benchmark runs side by side, light theme](docs/screenshots/bench-compare-light.png) | +| ![An LLM inference sweep at 8 concurrent streams: every stream's text, state and tokens/s, live](docs/screenshots/bench-streams-dark.png) | ![A finished concurrency sweep from 1 to 32 streams: output, decode speed and time to first token per level, light theme](docs/screenshots/bench-sweep-light.png) | +| ![A tool-calling evaluation running: the score so far, the scenario board and the latest verdicts](docs/screenshots/bench-tool-eval-dark.png) | ![A finished tool-calling evaluation: score, rating, safety gate and share graded, light theme](docs/screenshots/bench-tool-eval-score-light.png) | | ![A node whose agent is unreachable: cause, fix and retry](docs/screenshots/offline-state-dark.png) | ![Wall mode at 1920×1080](docs/screenshots/wall-dark.png) | The memory, CPU, network and offline screenshots come from a real two-node cluster (taken while spark2's agent was stopped). The others come from `--demo`, which can show load, a thermal episode and benchmark runs on demand. +The concurrency-stream and tool-calling screenshots show its simulated runs: no model was involved. ## Features @@ -145,10 +148,12 @@ was stopped). The others come from `--demo`, which can show load, a thermal epis the normal interface counters), and errors and drops for both ConnectX-7 rails, plus LAN throughput. - **Alerts.** Editable thresholds (GPU temperature, throttling, memory, node offline, link down, disk full, NVMe temperature) and an event timeline with acknowledgement. -- **Benchmarks.** Eight whitelisted, parameter-validated jobs, from GEMM TFLOPS to a 30-minute thermal soak. - Each one states its duration and impact up front, asks for confirmation, shows live telemetry while it - runs, cancels cleanly, and saves results with full metadata. Baselines, regression flags, side-by-side - comparison, JSON/CSV export and a shareable summary card are included. +- **Benchmarks.** Nine whitelisted, parameter-validated jobs, from GEMM TFLOPS to a 30-minute thermal soak + and a tool-calling evaluation of your served model. Each one states its duration and impact up front, asks + for confirmation, shows live telemetry while it runs, cancels cleanly, and saves results with full + metadata. The LLM benchmark shows every concurrent stream as it generates and can sweep concurrency levels; + the tool-calling evaluation shows each scenario's verdict as it finishes. Baselines, regression flags, + side-by-side comparison, JSON/CSV export and a shareable summary card are included. - **Wall mode, command palette and keyboard shortcuts.** Dark and light themes that follow the system, with reduced-motion and reduced-transparency support. - **Demo mode** (`--demo`): three simulated nodes, including a thermal episode, a stalled stream and a node @@ -344,8 +349,17 @@ Sparkline is built for a trusted LAN. It has no TLS of its own. process leaves the node (`--api-key`, `--hf-token`, `--password`, `--secret…`, `NAME_TOKEN=`/`NAME_KEY=`, `sk-…` and `hf_…` keys, URL passwords all become `***`). Both services mask `Authorization` and `Cookie` header values in their logs, even at `--log-level debug`. The LLM benchmark never copies an endpoint's - response body into a run error (only an OpenAI-style error message), and refuses SSH, Docker API and - Sparkline ports as endpoints. + response body into a run error (only an OpenAI-style error message). It and the tool-calling evaluation + refuse SSH, Docker API and Sparkline ports as endpoints. +- **Model output is stored with the run.** For its stream view, the LLM benchmark keeps the last 280 + characters of each stream's answer and 160 of its reasoning, and a timeline of its requests (times and token + counts). The tool-calling evaluation keeps each scenario's trace (prompts, tool calls and the model's + replies, cut to 4,000 characters) and its result in tool-eval-bench's own format. The traces are + tool-eval-bench's logs: they, its scenario summaries and its lines in the run's log can include up to about + 200 characters of an error response from your server, stored like the rest of the trace. All of it is saved + with the run in `console.sqlite`, included in its JSON export and visible to anyone signed in to the + console. The web app shows it as plain text, never as HTML or Markdown. tool-eval-bench's tools are mocks: + nothing the model asks for is executed. - **An agent without a token** starts only on a loopback address, and then answers only local, non-browser requests. - **No shell execution.** No endpoint runs commands. Benchmarks are a fixed registry, and every parameter is @@ -357,8 +371,8 @@ Sparkline is built for a trusted LAN. It has no TLS of its own. `/etc/systemd/system/sparkline-agent.service.d/10-benchmarks.conf`. Anyone who controls a process in the docker group can control the host. Enable benchmarks only on machines where that trade-off is acceptable. Without the flag the agent is monitor-only: GPU benchmarks report Docker as unavailable, the host-side - benchmarks (CPU, storage, RDMA, LLM) report the agent as CPU-capped, and container names in the process - table fall back to short container ids. + benchmarks (CPU, storage, RDMA, LLM inference, tool-calling evaluation) report the agent as CPU-capped, and + container names in the process table fall back to short container ids. - **systemd sandboxing.** Both services run as the unprivileged `sparkline` user, with no capabilities, `NoNewPrivileges`, a read-only filesystem except their state directory (`ProtectSystem=strict`), a private /tmp, a system-call filter, and network access limited to IPv4, IPv6, Unix and netlink sockets. The agent @@ -401,7 +415,8 @@ Benchmarks run only when you confirm them in the console. One job runs per node | NCCL all-reduce (`nccl-allreduce`) | All-reduce bus bandwidth between the two nodes over the CX7 rails | about 10 s | Medium: stalls a TP=2 job | Docker + PyTorch image, 2 nodes | | RDMA bandwidth and latency (`rdma-bw-lat`) | `ib_write_bw` / `ib_write_lat` over one CX7 rail | about 10 s | Low: saturates that rail | `perftest`, 2 nodes | | NVMe storage (`storage-fio`) | Sequential 1 MiB and random 4 KiB read/write: MB/s, IOPS, p99 latency | about 35 s | Low, but heavy disk I/O | `fio` (optional) | -| LLM inference (`llm-inference`) | TTFT, prefill and decode tokens/s, memory footprint | seconds to minutes, depending on the model | Medium: real requests to your server | An OpenAI-compatible endpoint on the LAN | +| LLM inference (`llm-inference`) | TTFT, prefill and decode tokens/s, memory footprint, with a live view of every concurrent stream and an optional sweep over concurrency levels | seconds to minutes, depending on the model | Medium: real requests to your server | An OpenAI-compatible endpoint on the LAN | +| Tool-call eval (`tool-eval`) | How well the served model uses tools: tool-eval-bench's multi-turn scenarios with mock tools, graded pass, partial or fail into a 0–100 score, with category scores and a trace per scenario | Core (15 scenarios) a few minutes, Standard (69, the default) 20–30 min, Hard Mode (28) about 15 min, all 97 30–45 min, depending on the model | Medium: real requests to your server, for minutes | An OpenAI-compatible endpoint on the LAN with tool calling enabled (vLLM: `--enable-auto-tool-choice --tool-call-parser …`) | | Thermal soak (`thermal-soak`) | Clock stability, throttling, temperature and TFLOPS drift under sustained load | 10 to 30 min | High: holds the GPU at full load | Docker + PyTorch image | Notes: @@ -414,6 +429,25 @@ Notes: - Without `fio`, the storage benchmark uses a built-in O_DIRECT runner. The method is recorded with the results, and results from different methods are never compared. Install fio with `sudo apt install fio`. - `sudo apt install perftest` enables the RDMA benchmark. +- The LLM benchmark keeps Concurrency streams busy: each one sends its next request as soon as its last one + ends, and a level runs the larger of Requests and its number of streams. With **Sweep concurrency** it + runs 1, 2, 4 … streams up to Concurrency, one level after another (24 gives 1, 2, 4, 8, 16, 24), and + reports output tokens/s, decode tokens/s per request and TTFT for each level, plus the peak. While it + runs, every stream shows its state (waiting for the first token, decoding, done), its tokens, recent + tokens/s and the end of its text; afterwards the run keeps a timeline of every request. On vLLM it asks for + usage in every streamed chunk (`continuous_usage_stats`), so live token counts stay exact when one chunk + carries several tokens (MTP or speculative decoding). +- The tool-calling evaluation runs [tool-eval-bench](https://github.com/SeraphimSerapis/tool-eval-bench) + 2.8.0 (SeraphimSerapis, MIT) in a host process under the agent, with tool-eval-bench's defaults + (temperature 0, 120 s per request, 8 turns, one scenario at a time). The agent installs it hash-pinned from + `server/requirements.lock`; an agent installed before it was added says "tool-eval-bench isn't installed" + until you re-run `install.sh agent --enable-benchmarks`. Before the scenarios, one tool-calling request + checks the server: if it rejects tool calls (vLLM without `--enable-auto-tool-choice`), the run stops with + the fix instead of scoring the few scenarios that got through. Thinking models also need a reasoning parser + (vLLM `--reasoning-parser …`); without one the run warns that reasoning text arrives in the answer, which + lowers the score. Scenarios that time out or hit server errors are excluded and the share graded is shown + with the score; a run in which nothing could be graded fails. The run page offers tool-eval-bench's own + JSON result for comparing with its CLI, sparkrun or Spark Arena. - Each result records driver, CUDA and NVML versions, GPU name and UUID, clocks and temperatures at start and end, agent version, the suite's git hash (when run from a checkout) and SHA-256, hostnames, parameters and timestamps. @@ -434,6 +468,7 @@ caveats. | NCCL all-reduce, both rails | ~22.5 GB/s bus bandwidth at 1 GiB, ~13 µs for 8 B | | Thermal soak, 10 min at 8192 | ~95 TFLOPS BF16, under 1 % drop first to last minute, ~81 °C peak, SM clock ~2.15–2.18 GHz, no throttle reasons | | LLM inference | depends on your model and server; Qwen2.5-0.5B on vLLM decodes ~160 tokens/s per request | +| Tool-call eval | not measured on these nodes yet. The score depends on the model and how it is served (chat template, tool-call and reasoning parsers, thinking on or off), not on the Spark, which mostly sets how long a run takes | Each CX7 rail is its own PCIe Gen5 x4 link (~109 Gb/s of payload), but both rails share one 200 Gb/s port: run together they carry ~185 Gb/s, which is the ceiling NCCL reaches. @@ -530,9 +565,18 @@ One Python package (`server/`, package `sparkline`) provides both commands, `spa `agent_token`. - Only GPU 0 is charted (a DGX Spark has one GPU). The cross-node benchmarks assume two nodes. - GPU benchmarks need the PyTorch image already pulled, and the docker group (root-equivalent). -- Host-side benchmarks (CPU, storage, RDMA, LLM) run inside the agent's cgroup. Without - `--enable-benchmarks` the agent is capped at half a core, so the catalog marks them unavailable ("CPU-capped", - read from the cgroup's `cpu.max`) rather than letting a throttled run produce wrong numbers. +- Host-side benchmarks (CPU, storage, RDMA, LLM inference, tool-calling evaluation) run inside the agent's + cgroup. Without `--enable-benchmarks` the agent is capped at half a core, so the catalog marks them unavailable + ("CPU-capped", read from the cgroup's `cpu.max`) rather than letting a throttled run produce wrong numbers. +- Tool-calling scores compare only between runs of the same tool-eval-bench version and settings (scenarios, + thinking on or off); each run records the version and tool-eval-bench's config fingerprint. Sparkline pins + 2.8.0. In 2.8.0 a thinking model can lose TC-45: its `tool_choice: "required"` check allows 256 tokens, and + a model that spends them thinking or answers in prose gets the scenario excluded, so the share graded drops + below 100 % (tool-eval-bench issue #299). A score with exclusions isn't comparable with a complete run. +- Since 2026-10-21 the LLM benchmark runs at least one request per stream: each level runs the larger of + Requests and its number of streams, with or without the sweep. Runs with Requests below Concurrency + therefore aren't compared with runs made before that, which sent fewer requests at once; set a new baseline + for such a configuration. - Agents run on the host, not in containers: they need NVML, `/proc` and sysfs of the machine they measure. - `install.sh` targets systemd-based Linux (DGX OS / Ubuntu 24.04) and needs PyPI access at install time. - Sparkline is built and tested on DGX Spark (GB10, aarch64). The agent also runs on other Linux machines, with or @@ -542,5 +586,7 @@ One Python package (`server/`, package `sparkline`) provides both commands, `spa ## License -[MIT](LICENSE). Sparkline is an independent project, not affiliated with or endorsed by NVIDIA. NVIDIA, DGX and -ConnectX are trademarks of NVIDIA Corporation. +[MIT](LICENSE). The tool-calling evaluation uses +[tool-eval-bench](https://github.com/SeraphimSerapis/tool-eval-bench) by SeraphimSerapis (MIT), installed from +PyPI. Sparkline is an independent project, not affiliated with or endorsed by NVIDIA. NVIDIA, DGX and ConnectX +are trademarks of NVIDIA Corporation. diff --git a/deploy/systemd/sparkline-agent-benchmarks.conf b/deploy/systemd/sparkline-agent-benchmarks.conf index b765fe7..3c19739 100644 --- a/deploy/systemd/sparkline-agent-benchmarks.conf +++ b/deploy/systemd/sparkline-agent-benchmarks.conf @@ -10,8 +10,8 @@ # compromise of the host. That is why this is opt-in. SupplementaryGroups=docker -# cpu-throughput, storage-fio, rdma-bw-lat and llm-inference run as children of the agent, inside this -# cgroup. The monitoring caps in the main unit would throttle them and skew the results, so they are +# cpu-throughput, storage-fio, rdma-bw-lat, llm-inference and tool-eval run as children of the agent, inside +# this cgroup. The monitoring caps in the main unit would throttle them and skew the results, so they are # lifted here. The empty assignment resets CPUQuota. Memory stays capped well above what any payload needs. CPUQuota= MemoryHigh=infinity diff --git a/docs/REVIEW.md b/docs/REVIEW.md index 7afdbc9..fd44d24 100644 --- a/docs/REVIEW.md +++ b/docs/REVIEW.md @@ -275,6 +275,10 @@ fixed. Highest severity first. - **LLM inference** ran only against vLLM with Qwen2.5-0.5B on spark1. Other engines (TRT-LLM, SGLang) and large models are untested. "Prefill tokens/s" is per request (prompt ÷ TTFT), so at concurrency > 1 it is well below the server's aggregate prefill rate. + - **Concurrency streams and the tool-call eval** (added 2026-10-21) have run only against local fake servers + and in `--demo`, never against a real model endpoint: both nodes were busy with measured workloads. The + live stream view, the concurrency sweep, tool-eval-bench's scores and timings, and the tool-calling check + are unverified on real hardware. - The **thermal soak** ran for 10 minutes, not 30. "Time throttled 0 %" does not mean the clocks held: GB10 eases the SM clock 1–2 % as it heats without setting a reason bit, which only clock CV and the lowest clock show. @@ -324,7 +328,8 @@ fixed. Highest severity first. - **The confirm dialog's duration is a client-side guess.** It scales the catalog estimate per parameter, so for the storage benchmark with a shorter runtime per profile it said "≈ 25 s" where the agent estimated 4 s (the run view switches to the agent's estimate). Harmless but imprecise. In `--demo`, runs are also compressed to about a - tenth of the estimate; the catalog descriptions say so, but the confirm dialog still shows the real estimate. + tenth of the estimate (LLM inference plays at double speed, 4–60 s per level); the catalog descriptions say so, + but the confirm dialog still shows the real estimate. - **No built-in TLS; one agent token shared by all nodes.** Use a reverse proxy (with `trusted_proxies`) or a VPN beyond the LAN. The CSP is sent by the console only: `npm run e2e` runs behind `vite preview`, which does not send it, so the CSP was verified with a separate browser walk against the real console. diff --git a/docs/screenshots/bench-streams-dark.png b/docs/screenshots/bench-streams-dark.png new file mode 100644 index 0000000..aded13b Binary files /dev/null and b/docs/screenshots/bench-streams-dark.png differ diff --git a/docs/screenshots/bench-sweep-light.png b/docs/screenshots/bench-sweep-light.png new file mode 100644 index 0000000..b357c1e Binary files /dev/null and b/docs/screenshots/bench-sweep-light.png differ diff --git a/docs/screenshots/bench-tool-eval-dark.png b/docs/screenshots/bench-tool-eval-dark.png new file mode 100644 index 0000000..cd26c10 Binary files /dev/null and b/docs/screenshots/bench-tool-eval-dark.png differ diff --git a/docs/screenshots/bench-tool-eval-score-light.png b/docs/screenshots/bench-tool-eval-score-light.png new file mode 100644 index 0000000..7e62fdf Binary files /dev/null and b/docs/screenshots/bench-tool-eval-score-light.png differ diff --git a/server/pyproject.toml b/server/pyproject.toml index 051cea4..1b008c6 100644 --- a/server/pyproject.toml +++ b/server/pyproject.toml @@ -13,6 +13,7 @@ dependencies = [ "websockets>=12", "psutil>=5.9", "nvidia-ml-py>=12.560", + "tool-eval-bench==2.8.0", # the tool-eval benchmark; exact, as its scores only compare within one version ] [project.optional-dependencies] @@ -59,6 +60,6 @@ ignore_missing_imports = true [[tool.mypy.overrides]] # Benchmark payloads run as scripts inside the PyTorch container: torch is not installed on the host, and -# benchlib is imported as a sibling script, not as sparkline.bench.payloads.benchlib. -module = ["torch", "torch.*", "benchlib"] +# benchlib (and llm_inference, for tool_eval) is imported as a sibling script, not as a sparkline module. +module = ["torch", "torch.*", "benchlib", "llm_inference"] ignore_missing_imports = true diff --git a/server/requirements.lock b/server/requirements.lock index d210af7..d9f42c5 100644 --- a/server/requirements.lock +++ b/server/requirements.lock @@ -89,6 +89,12 @@ httpx==0.28.1 \ idna==3.20 \ --hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \ --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c +markdown-it-py==4.2.0 \ + --hash=sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49 \ + --hash=sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a +mdurl==0.1.2 \ + --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8 \ + --hash=sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba nvidia-ml-py==13.615.71 \ --hash=sha256:959bf4adf6fe1308e4bd739e722236b0d1ec8392e2cefad33ff70c311380b9b6 \ --hash=sha256:bebe4e48f51b1dc75028c0815cb7bfa14a31a5bb80be70c9d980c6036953fc3d @@ -250,6 +256,9 @@ pydantic-core==2.46.5 \ --hash=sha256:fc5d783bd4a2387e97b8a2d5ec781cfb92b3d893bf82370548e99db5915935d3 \ --hash=sha256:fc8515076c11f3cfdf4fb142dcca0fe384b1230a3b5415458ac84f3e0903ec13 \ --hash=sha256:ff218293c9c806138dca139765e3b067621be52bcd93cdc14c7711be7ddc90a9 +pygments==2.21.0 \ + --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ + --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c python-dotenv==1.2.4 \ --hash=sha256:42269a8a5b3fd54ffa6f3d84b18abed50064717576b4ecf03dc4a55d8aa04fdc \ --hash=sha256:f0d53e69935a851c0dcc78f3ab7aaccd8cabef0b92382b576b824212902873c0 @@ -327,12 +336,18 @@ pyyaml==6.0.3 \ --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 +rich==15.0.0 \ + --hash=sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb \ + --hash=sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36 starlette==1.7.0 \ --hash=sha256:67f8e99895493dd2911a03f11314af6ceebeae4e704bb9f43dfc6a9db151c93e \ --hash=sha256:c79f74ea63cff761804fbbfb182f1e0b440c2d07b164d24700c5a1bab5d6ff5d tomlkit==0.15.1 \ --hash=sha256:177a05aece5a8ca5266fd3c448abb47b8d352f09d477d3ca8332db4d89b24304 \ --hash=sha256:e25bbf38843005246210a12982776f27f99cb9be67160e14434d0c0d21ee1e97 +tool-eval-bench==2.8.0 \ + --hash=sha256:8870f0d037cc73c34f65b70bf66748d1c2a6c999904c2324fe3c050c1326e6fa \ + --hash=sha256:91bb2504fdcaa25e206bf495be10853c04a7ba883e3e4e4fe8b02b3c14759a63 trove-classifiers==2026.9.21.13 \ --hash=sha256:0a9ebc8d4e2f3e8a22848c5258033035bec17a3012ac3fea16dbaa764489eb71 \ --hash=sha256:8b1ff4f9c191b1040b71c37f1e445ab99732911e3cd91de52838453a854d7a17 @@ -342,6 +357,9 @@ typing-extensions==4.16.0 \ typing-inspection==0.4.4 \ --hash=sha256:547274fa6b0a561ccf549cc9524b999a578e737d015d8709d021f9d0d13bea47 \ --hash=sha256:65b8397ba37ccbce054456aaccddfc91e6e3083c92824df348d96ca832f3f147 +tzdata==2026.5 ; sys_platform == 'win32' \ + --hash=sha256:8cc73c0a0bfca7dbfa59235d60b2eff82231dee33f53d206db1acd9173cfc0a7 \ + --hash=sha256:b683bd1b6659ddcd810ff02ad09ba821d4bf1065072805063eb35c49617905ac uvicorn==0.54.0 \ --hash=sha256:505bdb0f318731d45f1f712071fc781a8981f6847a31c902c9f5e652d4f67faf \ --hash=sha256:a2e33cbfaa0306f8e6b0c13e0cb89d7d7a2da3e62b90c66e18c33d9807b28620 diff --git a/server/sparkline/agent/bench/jobs.py b/server/sparkline/agent/bench/jobs.py index e4d6f6f..b5e13f2 100644 --- a/server/sparkline/agent/bench/jobs.py +++ b/server/sparkline/agent/bench/jobs.py @@ -27,6 +27,7 @@ RdmaParams, StorageParams, ThermalSoakParams, + ToolEvalParams, nccl_sizes, ) @@ -308,6 +309,21 @@ def _storage_results( ) +# --------------------------------------------------------------------------------------------------------- +# Tool-call evaluation +# --------------------------------------------------------------------------------------------------------- + + +async def tool_eval(job: Job, p: ToolEvalParams) -> None: + """tool-eval-bench, run in-process by a host payload (it imports the package from the agent's venv).""" + args = ["--base-url", p.base_url, "--suite", p.suite] + if p.model: + args += ["--model", p.model] + if not p.thinking: + args.append("--no-think") + await job.run(job.host_argv("tool_eval.py", args)) + + # --------------------------------------------------------------------------------------------------------- # Telemetry-assisted benches (LLM inference memory, thermal soak) # --------------------------------------------------------------------------------------------------------- @@ -413,6 +429,8 @@ def track(s: Sample) -> None: ] if p.model: args += ["--model", p.model] + if p.sweep: + args.append("--sweep") await _with_sampling(job, job.run(job.host_argv("llm_inference.py", args)), track) if not job.state.results: return @@ -547,5 +565,6 @@ def soak_results(job: Job, s: dict[str, list[float]]) -> None: "rdma-bw-lat": rdma_bw_lat, "storage-fio": storage, "llm-inference": llm_inference, + "tool-eval": tool_eval, "thermal-soak": thermal_soak, } diff --git a/server/sparkline/agent/bench/parsers.py b/server/sparkline/agent/bench/parsers.py index 63c2934..25c365f 100644 --- a/server/sparkline/agent/bench/parsers.py +++ b/server/sparkline/agent/bench/parsers.py @@ -6,7 +6,7 @@ import re from typing import Any -PAYLOAD_KEYS = ("progress", "result", "series", "gauge", "log", "ready", "error") +PAYLOAD_KEYS = ("progress", "result", "series", "gauge", "log", "ready", "error", "live", "detail") def parse_json_line(line: str) -> dict[str, Any] | None: diff --git a/server/sparkline/agent/bench/probes.py b/server/sparkline/agent/bench/probes.py index e4d4021..33c0700 100644 --- a/server/sparkline/agent/bench/probes.py +++ b/server/sparkline/agent/bench/probes.py @@ -1,7 +1,9 @@ -"""Environment checks for benchmark availability: Docker + image, GPU, perftest, RoCE GIDs, fio, CPU cap.""" +"""Environment checks for benchmark availability: Docker + image, GPU, perftest, RoCE GIDs, fio, CPU cap, +tool-eval-bench.""" from __future__ import annotations +import importlib.util import json import os import shutil @@ -115,6 +117,7 @@ class Env: all_reduce_perf: str | None = None work_dir: Problem | None = None cpu_cap: Problem | None = None # benchmarks that run as agent children would be throttled + tool_eval: Problem | None = None # the tool-eval payload imports tool-eval-bench from the agent's venv def probe(info: NodeInfo, image: str, work_dir: str, sock_path: str = DOCKER_SOCK) -> Env: @@ -182,6 +185,13 @@ def probe(info: NodeInfo, image: str, work_dir: str, sock_path: str = DOCKER_SOC "and skew its results", fix=f"Run `sudo ./deploy/install.sh agent --enable-benchmarks` on {node}", ) + if importlib.util.find_spec("tool_eval_bench") is None: + env.tool_eval = Problem( + message=f"tool-eval-bench isn't installed on {node}", + cause="The agent's Python environment doesn't include the tool-eval-bench package.", + fix="Reinstall the agent (`sudo ./deploy/install.sh agent --enable-benchmarks` installs the " + "pinned version), or for a dev agent run `server/.venv/bin/pip install tool-eval-bench==2.8.0`.", + ) return env @@ -203,4 +213,6 @@ def availability(bench_id: str, env: Env) -> tuple[Problem | None, str]: if bench_id == "storage-fio": note = "" if env.fio else "fio is not installed, so the built-in O_DIRECT runner is used." return env.work_dir or env.cpu_cap, note + if bench_id == "tool-eval": + return env.tool_eval or env.cpu_cap, "" return env.cpu_cap, "" # cpu-throughput, llm-inference: host processes under the agent's cgroup diff --git a/server/sparkline/agent/bench/runner.py b/server/sparkline/agent/bench/runner.py index f8e14bb..e9465f3 100644 --- a/server/sparkline/agent/bench/runner.py +++ b/server/sparkline/agent/bench/runner.py @@ -53,6 +53,9 @@ PUBLISH_MIN_S = 0.5 WORK_MARKER = ".sparkline-bench-run" # marks a run's work dir, so the sweep only ever deletes those CPU_SMOKE_BENCHES = ("gpu-matmul", "mem-bandwidth", "thermal-soak") # the payloads that honour --device cpu +LIVE_MAX_BYTES = 256 * 1024 # compact JSON of one {"live": ...} snapshot; it is re-sent with every publish +DETAIL_MAX_BYTES = 1024 * 1024 # compact JSON of a run's whole detail +JSON_MAX_DEPTH = 64 # live/detail nesting; pydantic can't serialize > 255 levels or load stored runs > ~197 class BenchError(Exception): @@ -73,6 +76,18 @@ def __init__(self, status: int, problem: Problem) -> None: JobFn = Callable[["Job", Any], Awaitable[None]] +def _too_deep(value: Any) -> bool: + """Whether a parsed JSON value nests objects/arrays more than JSON_MAX_DEPTH levels deep (walked level + by level, as json.loads accepts deeper documents than Python's recursion limit).""" + level: list[Any] = [value] + for _ in range(JSON_MAX_DEPTH): + nested = [v for v in level if isinstance(v, dict | list)] + if not nested: + return False + level = [c for v in nested for c in (v.values() if isinstance(v, dict) else v)] + return any(isinstance(v, dict | list) for v in level) + + class Job: """Context handed to a bench function: progress/results/logging and subprocess helpers.""" @@ -95,6 +110,7 @@ def __init__( self.timeout_s = bench.timeout_s(params) self.est_s = bench.estimate(params) self._log: deque[str] = deque(maxlen=LOG_TAIL) + self._logged_once: set[str] = set() self.task: asyncio.Task | None = None self.begun = False # the task has started executing (cancelling before that never reaches it) self.cancel_requested = False @@ -123,6 +139,22 @@ def log(self, line: str) -> None: self.state.log_tail = list(self._log) self.changed() + def _log_once(self, line: str) -> None: + """Payload mistakes may repeat on every line: log each once per job, not a flooded log tail.""" + if line not in self._logged_once: + self._logged_once.add(line) + self.log(line) + + def _usable_object(self, what: str, value: Any) -> bool: + """live/detail values must be JSON objects nested at most JSON_MAX_DEPTH levels.""" + if not isinstance(value, dict): + self._log_once(f"{what} is not a JSON object, ignored") + return False + if _too_deep(value): + self._log_once(f"{what} nested too deeply, dropped") + return False + return True + def add_result(self, **kw: Any) -> None: kw.setdefault("node_id", self.node_id) if self.runner.payload_device == "cpu" and self.bench.id in CPU_SMOKE_BENCHES: @@ -159,6 +191,19 @@ def handle(self, msg: dict[str, Any]) -> None: for k, v in msg["gauge"].items(): with contextlib.suppress(TypeError, ValueError): self.gauges[str(k)[:64]] = float(v) + if "live" in msg and self._usable_object("live snapshot", msg["live"]): + if len(json.dumps(msg["live"], separators=(",", ":"))) > LIVE_MAX_BYTES: + self._log_once("live snapshot too large, dropped") + else: + self.state.live = msg["live"] # replaced wholesale + self.changed() + if "detail" in msg and self._usable_object("detail", msg["detail"]): + detail = {**self.state.detail, **msg["detail"]} # top-level keys replace + if len(json.dumps(detail, separators=(",", ":"))) > DETAIL_MAX_BYTES: + self._log_once("detail too large, dropped") + else: + self.state.detail = detail + self.changed() if "log" in msg: self.log(str(msg["log"])) if msg.get("ready"): @@ -242,9 +287,11 @@ async def run( ) from None lines: deque[str] = deque(maxlen=5000) watcher = asyncio.create_task(self._watch_port(ready_port, proc)) if ready_port else None - try: - assert proc.stdout is not None - while raw := await proc.stdout.readline(): + assert proc.stdout is not None + stdout = proc.stdout + + async def read() -> None: + while raw := await stdout.readline(): line = raw.decode(errors="replace").rstrip("\n") lines.append(line) msg = parse_json_line(line) @@ -252,12 +299,20 @@ async def run( self.handle(msg) elif line.strip(): self.log(line) + + try: + await read() rc = await proc.wait() finally: if watcher: watcher.cancel() if await self.runner._terminate(proc) and argv[:2] == ["docker", "run"]: self.docker_killed = True + # Cancelled or timed out: keep what the payload printed while exiting (its last live snapshot, a + # log line). It has exited, so this ends at once; the bound covers a straggler holding the pipe. + with contextlib.suppress(Exception): + async with asyncio.timeout(1): + await read() if check and rc != 0: raise BenchError(self.payload_error or self._exit_problem(argv, rc, list(lines))) return rc, list(lines) diff --git a/server/sparkline/agent/bench/specs.py b/server/sparkline/agent/bench/specs.py index 8fb1f6e..bdb8386 100644 --- a/server/sparkline/agent/bench/specs.py +++ b/server/sparkline/agent/bench/specs.py @@ -180,7 +180,43 @@ class LlmInferenceParams(Params): prompt_tokens: int = _param(512, "Prompt length", "Approximate prompt size.", "tokens", ge=16, le=65536) max_tokens: int = _param(128, "Output length", "max_tokens per request.", "tokens", ge=1, le=4096) concurrency: int = _param(1, "Concurrency", "Requests in flight at once.", ge=1, le=32) - requests: int = _param(8, "Requests", "Total requests.", ge=1, le=512) + requests: int = _param( + 8, "Requests", "Total requests (per level when sweeping, at least one per stream).", ge=1, le=512 + ) + sweep: bool = _param( + False, + "Sweep concurrency", + "Run 1, 2, 4 … streams up to Concurrency, one level after another, and compare them.", + ) + + @field_validator("base_url") + @classmethod + def _url(cls, v: str) -> str: + return check_base_url(v) + + @field_validator("model") + @classmethod + def _model(cls, v: str) -> str: + if v and not _MODEL_RE.match(v): + raise ValueError("model name may only contain letters, digits and . - / : _ (max 200)") + return v + + +class ToolEvalParams(Params): + base_url: str = _param( + "http://127.0.0.1:8000", + "Endpoint", + "OpenAI-compatible server (vLLM, TRT-LLM, llama.cpp, SGLang...) on a private or loopback address.", + ) + model: str = _param("", "Model", "Model name as served; empty picks the first from /v1/models.") + suite: Literal["core", "standard", "hard", "all"] = _param( + "standard", + "Scenarios", + "Core 15 takes a few minutes; Standard 69, the upstream default, about 20-30 min on a Spark.", + ) + thinking: bool = _param( + True, "Thinking", "Off sends enable_thinking=false to the chat template (upstream's --no-think)." + ) @field_validator("base_url") @classmethod @@ -259,11 +295,33 @@ def est_storage(p: StorageParams) -> float: return 1.0 + p.size_mib * 2**20 / 1.5e9 + reads * p.runtime_s +def levels(concurrency: int, sweep: bool) -> list[int]: + """Streams per level of an LLM run: [concurrency], or when sweeping the powers of two below it and then + concurrency itself (32 -> 1, 2, 4, 8, 16, 32; 24 -> 1, 2, 4, 8, 16, 24). Each level runs + max(requests, level) requests.""" + out, k = [], 1 + while sweep and k < concurrency: + out.append(k) + k *= 2 + return [*out, concurrency] + + def est_llm(p: LlmInferenceParams) -> float: - waves = math.ceil(p.requests / p.concurrency) + waves = sum(math.ceil(max(p.requests, c) / c) for c in levels(p.concurrency, p.sweep)) return 5.0 + waves * (0.5 + p.prompt_tokens / 2000 + p.max_tokens / 20) +# tool-eval-bench 2.8.0 suites: (standard scenarios, Hard Mode scenarios). Not imported from the package: the +# catalog must load (and say what is missing) on an agent without it. +TOOL_EVAL_SUITES = {"core": (15, 0), "standard": (69, 0), "hard": (0, 28), "all": (69, 28)} + + +def est_tool_eval(p: ToolEvalParams) -> float: + """DGX Spark owners report 12-26 s per scenario with a thinking model; Hard Mode runs longer chains.""" + standard, hard = TOOL_EVAL_SUITES[p.suite] + return 10.0 + 24.0 * standard + 36.0 * hard # + the endpoint and tool-calling checks + + def est_soak(p: ThermalSoakParams) -> float: return CONTAINER_START_S + p.minutes * 60 @@ -451,11 +509,14 @@ def _impact(thermal: str, power: str, disruptive: bool, note: str) -> BenchImpac "llm-inference", "LLM inference", "inference", - "Time to first token and prefill and decode speed of a model you are already serving. " + "Time to first token and prefill and decode speed of a model you are already serving, with every " + "concurrent stream shown live. " "Streams requests to an OpenAI-compatible endpoint you are already serving (vLLM, TRT-LLM, " "llama.cpp, SGLang...) and measures time to first token, prefill and decode tokens/s, and the " - "memory footprint seen by the agent. The endpoint may itself span both nodes (TP=2); the run " - "measures it from this node.", + "memory footprint seen by the agent. Every request in flight is shown live with its state, text, " + "time to first token and tokens/s. Optionally it sweeps concurrency levels (1, 2, 4 … streams up " + "to Concurrency) with results per level; each level sends at least one request per stream. The " + "endpoint may itself span both nodes (TP=2); the run measures it from this node.", _impact( "medium", "medium", @@ -466,6 +527,25 @@ def _impact(thermal: str, power: str, disruptive: bool, note: str) -> BenchImpac LlmInferenceParams, est_llm, ), + Bench( + "tool-eval", + "Tool-call eval", + "inference", + "Scores how well the served model uses tools, from picking the right one to resisting prompt " + "injection. Runs tool-eval-bench 2.8.0 (SeraphimSerapis, MIT) against an OpenAI-compatible " + "endpoint: every scenario is a multi-turn chat with mock tools, graded pass (2 points), partial " + "(1) or fail (0); the score is points earned ÷ points possible × 100. Scenarios that time out or " + "hit server errors are excluded and reported as the share graded.", + _impact( + "medium", + "medium", + True, + "Sends real requests to the chosen server for several minutes; other users of that server " + "will see slower responses.", + ), + ToolEvalParams, + est_tool_eval, + ), Bench( "thermal-soak", "Thermal soak", diff --git a/server/sparkline/bench/payloads/benchlib.py b/server/sparkline/bench/payloads/benchlib.py index 99f831a..4d3086c 100644 --- a/server/sparkline/bench/payloads/benchlib.py +++ b/server/sparkline/bench/payloads/benchlib.py @@ -4,7 +4,10 @@ {"progress": 0..1, "phase": "..."} {"result": {BenchResult fields}} {"series": {key: [values]}} (appended to the run's series) {"gauge": {key: value}} (latest value only; the agent samples it on its own clock, e.g. soak TFLOPS) + {"live": {...}} (replaces the run's live snapshot; <= 256 KiB of compact JSON) + {"detail": {key: ...}} (top-level keys replace the run's detail keys; <= 1 MiB in all) {"log": "..."} {"ready": true} {"error": {"message", "cause", "fix"}} +live/detail nest at most 64 levels and hold no NaN/inf (round, and use null for a missing number). """ from __future__ import annotations @@ -36,6 +39,14 @@ def result(metric: str, label: str, value: float, unit: str, better: str = "high ) +def live(snapshot: dict) -> None: + emit(live=snapshot) + + +def detail(**parts) -> None: + emit(detail=parts) + + def log(text: str) -> None: emit(log=text) diff --git a/server/sparkline/bench/payloads/llm_inference.py b/server/sparkline/bench/payloads/llm_inference.py index 7e91380..44e36f2 100644 --- a/server/sparkline/bench/payloads/llm_inference.py +++ b/server/sparkline/bench/payloads/llm_inference.py @@ -4,22 +4,31 @@ token (TTFT), prefill tok/s = prompt tokens / TTFT, decode tok/s = (completion tokens - 1) / (last token - first token). Aggregate decode tok/s = all completion tokens / wall time. Token counts come from the server's usage block when it sends one, otherwise from the number of streamed content chunks. + +Requests run on stream slots: each of a level's streams sends its next request as soon as its previous one +ends (closed loop) until the level's requests are done; --sweep runs levels of 1, 2, 4 ... streams up to +--concurrency one after another. Every 0.25 s a live snapshot shows each stream's state, tokens, recent tok/s +and the tail of its text; each level's summary and request timeline go to the run's detail when it ends. """ from __future__ import annotations import argparse +import contextlib import http.client import json import math +import socket import statistics +import threading import time import uuid -from concurrent.futures import ThreadPoolExecutor +from collections import deque +from dataclasses import dataclass, field from typing import Any, cast from urllib.parse import urlsplit -from benchlib import fail, log, on_sigterm, progress, result +from benchlib import detail, fail, live, log, on_sigterm, progress, result TEXT = ( "the river carries light across the valley while engineers measure every quiet signal from the cluster " @@ -27,6 +36,16 @@ ) WORDS = TEXT.split() GENERIC_OWNERS = {"", "system", "openai", "organization-owner", "user", "owner"} +LIVE_EVERY_S = 0.25 # live snapshot interval (the agent publishes at most 2/s) +SLOT_RATE_S = 1.5 # a stream's tok/s: tokens over its last 1.5 s ... +RATE_MIN_S = 0.5 # ... once that spans 0.5 s (two tokens a few ms apart would make a bogus peak) +AGG_RATE_S = 2.0 # aggregate output tok/s window +HIST_MAX = 240 # tok_s_hist points, one per 0.5 s +TEXT_TAIL, THINK_TAIL = 280, 160 # characters of answer and reasoning text per stream in a snapshot +TIMELINE_ROWS = 256 # request records per level in the detail +LOCK = threading.Lock() # slots and level counters; a worker holds it for a few assignments per chunk +STOP = threading.Event() # set on SIGTERM: workers take no more requests and record nothing +OPEN: set[socket.socket] = set() # sockets of requests in flight, shut down on SIGTERM so reads return def http_error(status: int, body: bytes) -> str: @@ -71,10 +90,34 @@ def make_prompt(n_tokens: int, i: int) -> str: return f"Request {i} {uuid.uuid4().hex[:8]}. Summarise this text: " + " ".join(words) -def one_request(base: str, api: str, model: str, prompt: str, max_tokens: int) -> dict: +def levels(concurrency: int, sweep: bool) -> list[int]: + """Streams per level, as sparkline.agent.bench.specs.levels: [concurrency], or when sweeping the powers of + two below it and then concurrency itself.""" + out, k = [], 1 + while sweep and k < concurrency: + out.append(k) + k *= 2 + return [*out, concurrency] + + +def _str(v: Any) -> str: + return v if isinstance(v, str) else "" + + +def one_request( + base: str, + api: str, + model: str, + prompt: str, + max_tokens: int, + live_usage: bool = False, + slot: Slot | None = None, +) -> dict: conn, prefix = connect(base, 300) body: dict = {"model": model, "max_tokens": max_tokens, "stream": True, "temperature": 0} body["stream_options"] = {"include_usage": True} + if live_usage: # vLLM: usage in every chunk keeps live token counts exact when a chunk holds several + body["stream_options"]["continuous_usage_stats"] = True if api == "chat": body["messages"] = [{"role": "user", "content": prompt}] path = "/v1/chat/completions" @@ -90,8 +133,13 @@ def one_request(base: str, api: str, model: str, prompt: str, max_tokens: int) - "usage": None, "error": None, } + sock = None t0 = time.perf_counter() try: + conn.connect() + sock = conn.sock + with LOCK: + OPEN.add(sock) conn.request("POST", prefix + path, json.dumps(body), {"Content-Type": "application/json"}) r = conn.getresponse() if r.status != 200: @@ -110,6 +158,7 @@ def one_request(base: str, api: str, model: str, prompt: str, max_tokens: int) - msg = json.loads(data) if msg.get("usage"): out["usage"] = msg["usage"] + answer = thought = "" for ch in msg.get("choices") or []: d = ch.get("delta") or {} text = ch.get("text") or d.get("content") or d.get("reasoning_content") or d.get("reasoning") @@ -120,14 +169,23 @@ def one_request(base: str, api: str, model: str, prompt: str, max_tokens: int) - out["ttft"] = now - t0 out["last"] = now out["chunks"] += 1 + answer += _str(ch.get("text") or d.get("content")) + thought += _str(d.get("reasoning_content") or d.get("reasoning")) elif ch.get("finish_reason") and out["first"] is not None: # A stop token (EOS) streams as an empty chunk with the finish reason. usage counts it in - # completion_tokens, so with usage it is the last token (see decode in main()). + # completion_tokens, so with usage it is the last token (see tokens_of()). out["eos"] = time.perf_counter() + if slot is not None: + usage = msg.get("usage") + used = usage.get("completion_tokens") if live_usage and isinstance(usage, dict) else None + slot.chunk(out, answer, thought, used) out["total"] = time.perf_counter() - t0 except (OSError, ValueError, http.client.HTTPException) as e: out["error"] = f"{type(e).__name__}: {e}" finally: + if sock is not None: + with LOCK: + OPEN.discard(sock) conn.close() return out @@ -138,6 +196,373 @@ def pct(vals: list[float], q: float) -> float: return s[min(len(s) - 1, max(0, math.ceil(q / 100 * len(s)) - 1))] +def tokens_of(r: dict, prompt_tokens: int) -> tuple[Any, Any, float | None]: + """A request's prompt and completion tokens (the server's usage, else --prompt-tokens and the number of + text chunks) and its decode tok/s, None without at least two tokens over a time span.""" + usage = r["usage"] or {} + p_tok = usage.get("prompt_tokens") or prompt_tokens + c_tok = usage.get("completion_tokens") or r["chunks"] + # usage counts a stop token that streams no text: then the decode span ends at its (empty) chunk + last = r["eos"] if usage.get("completion_tokens") and r["eos"] else r["last"] + if r["first"] is None or c_tok <= 1 or last <= r["first"]: + return p_tok, c_tok, None + return p_tok, c_tok, (c_tok - 1) / (last - r["first"]) + + +def rnd(v: float | None, digits: int = 1) -> float | None: + """Rounded for the live snapshot and detail, which never carry NaN or inf (None instead).""" + return round(v, digits) if v is not None and math.isfinite(v) else None + + +def ms(seconds: float | None) -> float | None: + return rnd(seconds * 1000) if seconds is not None else None + + +def streams(n: int) -> str: + return f"{n} stream" if n == 1 else f"{n} streams" + + +class Slot: + """One stream of a level: its worker writes it per chunk and the reporter reads it, both under LOCK.""" + + def __init__(self, level: Level, index: int) -> None: + self.level, self.index = level, index + self.req = 0 # 1-based request number within the level; 0 = none yet + self.state = "idle" # idle | wait | decode | done | error + self.tokens = self.n_first = 0 # tokens of the current request; at its first token + self.sent: float | None = None + self.first: float | None = None + self.ended: float | None = None + self.ttft: float | None = None + self.final_tok_s: float | None = None + self.text = self.think = "" # tails of the current request's answer and reasoning + self.chars = self.think_chars = 0 # all answer/reasoning characters of this slot so far (only grow) + self.error: str | None = None + + def begin(self, req: int) -> None: + """A new request on this slot (under LOCK): waiting for its first token.""" + self.req, self.state, self.sent = req, "wait", time.perf_counter() + self.tokens = self.n_first = 0 + self.first = self.ended = self.ttft = self.final_tok_s = None + self.text = self.think = "" + self.error = None + + def chunk(self, out: dict, answer: str, thought: str, used: Any) -> None: + """One streamed message: the token count (vLLM's per-chunk usage when it sends one, else text chunks), + the switch to decoding at the first token of either kind (TTFT), and the text tails.""" + with LOCK: + n = used if isinstance(used, int) else out["chunks"] + if n > self.tokens: + self.level.total += n - self.tokens + self.tokens = n + if self.first is None and out["first"] is not None: + self.first, self.ttft, self.state = out["first"], out["ttft"], "decode" + self.n_first = self.tokens + self.level.ttfts.append(out["ttft"]) + if answer: + self.text = (self.text + answer)[-TEXT_TAIL:] + self.chars += len(answer) + if thought: + self.think = (self.think + thought)[-THINK_TAIL:] + self.think_chars += len(thought) + + def finish(self, r: dict) -> None: + """The request ended (under LOCK): done with its final token count and decode rate, or error.""" + self.ended = r["end"] + if r["ok"]: + self.state, self.tokens, self.final_tok_s = "done", r["tokens"], r["decode"] + else: + self.state, self.error = "error", (r["error"] or "no tokens streamed")[:200] + + def view(self, now: float) -> dict: + """The snapshot entry, without the rates (the reporter fills those in).""" + return { + "slot": self.index, + "req": self.req, + "state": self.state, + "tokens": self.tokens, + "max": self.level.a.max_tokens, + "ttft_ms": ms(self.ttft), + "tok_s": None, + "peak_tok_s": None, + "elapsed_ms": ms((self.ended or now) - self.sent) if self.sent is not None else None, + "text": self.text, + "chars": self.chars, + "think": self.think, + "think_chars": self.think_chars, + "error": self.error, + } + + +class Level: + """One concurrency level: a worker thread per slot takes the next request as soon as its last one ends; + the main thread reports every LIVE_EVERY_S. Workers change only the counters marked theirs, under LOCK.""" + + def __init__( + self, a: argparse.Namespace, model: str, live_usage: bool, index: int, completed: list[dict] + ) -> None: + self.a, self.model, self.live_usage, self.index = a, model, live_usage, index + self.sizes = levels(a.concurrency, a.sweep) + self.counts = [max(a.requests, size) for size in self.sizes] # every level gets a full wave at least + self.size = self.sizes[index] + self.completed = completed # summaries of the finished levels, this one too once it ends + self.prompts = [make_prompt(a.prompt_tokens, i) for i in range(self.counts[index])] + self.slots = [Slot(self, i) for i in range(self.size)] + self.records: list[dict] = [{} for _ in self.prompts] + # the workers' + self.next = self.done = self.failed = self.total = 0 # total: tokens streamed by every slot + self.ttfts: list[float] = [] + self.running = self.size + self.finished = threading.Event() + # the reporter's + self.t0 = 0.0 + self.windows: list[deque[tuple[float, int]]] = [deque() for _ in self.slots] + self.window_req = [0] * self.size + self.peak: list[float | None] = [None] * self.size + self.agg: deque[tuple[float, int]] = deque() + self.hist: list[float] = [] + self.ticks = 0 + self.reported: tuple[float, str] | None = None + + def run(self) -> float: + """Every request of the level, reported as they stream; returns the level's wall time. On SIGTERM + (SystemExit raised in here) the workers stop, their connections are shut so blocked reads return at + once, and a last snapshot shows where every stream was.""" + self.t0 = time.perf_counter() + self.report(self.t0) # every stream idle + threads = [threading.Thread(target=self.work, args=(s,), daemon=True) for s in self.slots] + try: + for t in threads: + t.start() + due = self.t0 + LIVE_EVERY_S + while not self.finished.wait(max(0.0, due - time.perf_counter())): + now = time.perf_counter() + self.report(now) + due = now + LIVE_EVERY_S + except BaseException: + STOP.set() + with LOCK: + socks = list(OPEN) + for sock in socks: + with contextlib.suppress(OSError): + sock.shutdown(socket.SHUT_RDWR) + with contextlib.suppress(Exception): + live(self.snapshot(time.perf_counter(), regular=False)) + raise + return max(r["end"] for r in self.records) - self.t0 + + def work(self, slot: Slot) -> None: + a = self.a + try: + while not STOP.is_set(): + with LOCK: + i = self.next + if i >= len(self.prompts): + return + self.next += 1 + slot.begin(i + 1) + sent = slot.sent + try: + r = one_request( + a.base_url, a.api, self.model, self.prompts[i], a.max_tokens, self.live_usage, slot + ) + except Exception as e: # a stream this code can't read fails its request, not the level + r = {"ttft": None, "first": None, "last": None, "eos": None, "chunks": 0, "usage": None} + r["error"] = f"{type(e).__name__}: {e}" + end = time.perf_counter() + if STOP.is_set(): + return + _, c_tok, decode = tokens_of(r, a.prompt_tokens) + ok = r["error"] is None and r["ttft"] is not None + r.update(slot=slot.index, req=i + 1, sent=sent, end=end, ok=ok, tokens=c_tok, decode=decode) + with LOCK: + self.records[i] = r + self.done += 1 + self.failed += not ok + slot.finish(r) + finally: + with LOCK: + self.running -= 1 + if not self.running: + self.finished.set() + + def snapshot(self, now: float, regular: bool = True) -> dict: + """The live picture (main thread only). Rates come from the token counts seen at each snapshot: a + stream's over its last SLOT_RATE_S since its first token, the aggregate over the last AGG_RATE_S.""" + with LOCK: + seen = [ + (s.req, s.state, s.tokens, s.n_first, s.first, s.final_tok_s, s.view(now)) for s in self.slots + ] + total, done, failed, ttfts = self.total, self.done, self.failed, list(self.ttfts) + out = [] + for i, (req, state, tokens, n_first, first, final, entry) in enumerate(seen): + win = self.windows[i] + if req != self.window_req[i]: + win.clear() + self.window_req[i] = req + rate = None + if state == "decode" and first is not None: + if not win: + win.append((first, n_first)) + win.append((now, tokens)) + while len(win) > 1 and win[1][0] <= now - SLOT_RATE_S: + win.popleft() + span = now - win[0][0] + if tokens >= 2 and span >= RATE_MIN_S: + rate = max(0, tokens - win[0][1]) / span + elif state == "done": + rate = final + if rate is not None: + self.peak[i] = max(rate, self.peak[i] or 0.0) + entry["tok_s"], entry["peak_tok_s"] = rnd(rate), rnd(self.peak[i]) + out.append(entry) + if not total: # nothing yet: the window starts at the last snapshot before the first token + self.agg.clear() + self.agg.append((now, total)) + while len(self.agg) > 1 and self.agg[1][0] <= now - AGG_RATE_S: + self.agg.popleft() + t0, n0 = self.agg[0] + agg = max(0, total - n0) / (now - t0) if total and now > t0 else None + if regular: + if self.ticks % 2 == 0: + self.hist.append(rnd(agg) or 0.0) + del self.hist[:-HIST_MAX] + self.ticks += 1 + return { + "kind": "llm-streams", + "v": 1, + "level": self.size, + "levels": self.sizes, + "level_index": self.index, + "requests": len(self.prompts), + "done": done, + "failed": failed, + "t": round(now - self.t0, 3), + "tok_s": rnd(agg), + "tok_s_hist": list(self.hist), + "ttft_p50_ms": ms(pct(ttfts, 50)) if ttfts else None, + "completed_levels": self.completed, + "streams": out, + } + + def report(self, now: float, regular: bool = True) -> None: + snap = self.snapshot(now, regular) + live(snap) + phase = f"{streams(self.size)} · {snap['done']} of {snap['requests']} done" + if self.a.sweep: + phase = f"Level {self.index + 1} of {len(self.sizes)} · {phase}" + p = round(0.05 + 0.95 * (sum(self.counts[: self.index]) + snap["done"]) / sum(self.counts), 4) + if (p, phase) != self.reported: + self.reported = (p, phase) + progress(p, phase) + + +@dataclass +class Stats: + """A level's measurements over its successful requests (the numbers its results report).""" + + ok: list[dict] + errors: list[str] + ttft: list[float] = field(default_factory=list) + prefill: list[float] = field(default_factory=list) + decode: list[float] = field(default_factory=list) + prompt_tokens: int = 0 + completion_tokens: int = 0 + + +def level_stats(rs: list[dict], prompt_tokens: int) -> Stats: + st = Stats([r for r in rs if r["ok"]], [r["error"] or "no tokens streamed" for r in rs if not r["ok"]]) + for r in st.ok: + p_tok, c_tok, decode = tokens_of(r, prompt_tokens) + st.ttft.append(r["ttft"]) + st.prefill.append(p_tok / r["ttft"]) + if decode is not None: + st.decode.append(decode) + st.prompt_tokens += p_tok + st.completion_tokens += c_tok + return st + + +def summary(size: int, n: int, st: Stats, wall: float) -> dict: + """One level in the detail (and in the live snapshot's completed_levels), rounded for display: the numbers + that are also results as the results round them, so the page shows one value for each.""" + ok, d = bool(st.ok), st.decode + return { + "level": size, + "requests": n, + "ok": len(st.ok), + "failed": n - len(st.ok), + "wall_s": rnd(wall, 3), + "out_tok_s": rnd(st.completion_tokens / wall, 2) if ok else None, + "decode_tok_s_p50": rnd(statistics.median(d), 2) if d else None, + "decode_tok_s_p10": rnd(pct(d, 10)) if d else None, + "itl_p50_ms": rnd(statistics.median(1000 / x for x in d)) if d else None, + "ttft_p50_ms": ms(pct(st.ttft, 50)) if ok else None, + "ttft_p95_ms": ms(pct(st.ttft, 95)) if ok else None, + "prefill_tok_s_p50": rnd(statistics.median(st.prefill)) if ok else None, + "prompt_tokens": st.prompt_tokens, + "completion_tokens": st.completion_tokens, + } + + +def timeline(rs: list[dict], t0: float) -> dict: + """Compact per-request records of a level, seconds since it started (the first TIMELINE_ROWS requests).""" + rows = [ + [ + r["slot"], + r["req"], + rnd(r["sent"] - t0, 4), + rnd(r["first"] - t0, 4) if r["first"] is not None else None, + rnd(r["end"] - t0, 4), + r["tokens"], + 1 if r["ok"] else 0, + ] + for r in rs[:TIMELINE_ROWS] + ] + cols = ["slot", "req", "sent", "first", "end", "tokens", "ok"] + return {"cols": cols, "rows": rows, "truncated": len(rs) > TIMELINE_ROWS} + + +def single_results(ex: dict, st: Stats, wall: float) -> None: + """A run at one concurrency: the same results as before sweeps existed, so its baselines still compare.""" + ttft, prefill, decode, total_out = st.ttft, st.prefill, st.decode, st.completion_tokens + # Engine and model live in extras (the run view shows them once); headlines: decode, TTFT p50, prefill. + head = {**ex, "headline": True} + if decode: + result( + "decode_tok_s", + "Decode tokens/s per request", + round(statistics.median(decode), 2), + "tok/s", + **head, + ) + result("ttft_p50_ms", "Time to first token p50", round(pct(ttft, 50) * 1000, 1), "ms", "lower", **head) + result("prefill_tok_s", "Prefill tokens/s", round(statistics.median(prefill), 1), "tok/s", **head) + result("ttft_p95_ms", "Time to first token p95", round(pct(ttft, 95) * 1000, 1), "ms", "lower", **ex) + result("decode_tok_s_total", "Output tokens/s, all requests", round(total_out / wall, 2), "tok/s", **ex) + + +def sweep_results(good: list[tuple[dict, Stats, float]]) -> None: + """A sweep: each level with results under its own names (_c{streams}). Headlines: the peak output tok/s, + per-request decode at the first level and TTFT p50 at the last.""" + out = [round(st.completion_tokens / wall, 2) for _, st, wall in good] + peak = out.index(max(out)) + first = next((ex["level"] for ex, st, _ in good if st.decode), None) + last = good[-1][0]["level"] + result("peak_output_tok_s", "Peak output tokens/s", out[peak], "tok/s", **good[peak][0], headline=True) + for (ex, st, _), out_tok_s in zip(good, out, strict=True): + size, at, head = ex["level"], streams(ex["level"]), {**ex, "headline": True} + result(f"decode_tok_s_total_c{size}", f"Output tokens/s · {at}", out_tok_s, "tok/s", **ex) + if st.decode: + decode = round(statistics.median(st.decode), 2) + label = f"Decode tokens/s per request · {at}" + result(f"decode_tok_s_c{size}", label, decode, "tok/s", **(head if size == first else ex)) + p50, p95 = round(pct(st.ttft, 50) * 1000, 1), round(pct(st.ttft, 95) * 1000, 1) + label = f"Time to first token p50 · {at}" + result(f"ttft_p50_ms_c{size}", label, p50, "ms", "lower", **(head if size == last else ex)) + result(f"ttft_p95_ms_c{size}", f"Time to first token p95 · {at}", p95, "ms", "lower", **ex) + + def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--json", action="store_true") @@ -148,6 +573,7 @@ def main() -> None: ap.add_argument("--max-tokens", type=int, default=128) ap.add_argument("--concurrency", type=int, default=1) ap.add_argument("--requests", type=int, default=8) + ap.add_argument("--sweep", action="store_true") a = ap.parse_args() on_sigterm() progress(0.0, "Querying /v1/models") @@ -167,68 +593,53 @@ def main() -> None: engine = owner if owner.lower() not in GENERIC_OWNERS else headers.get("server", "") engine = engine.split("/")[0] or "unknown engine" log(f"model {model}, engine {engine} (owned_by={owner!r}, server={headers.get('server')!r})") - prompts = [make_prompt(a.prompt_tokens, i) for i in range(a.requests)] - done = [0] - - def run(i: int) -> dict: - r = one_request(a.base_url, a.api, model, prompts[i], a.max_tokens) - done[0] += 1 - progress(0.05 + 0.95 * done[0] / a.requests, f"Request {done[0]}/{a.requests}") - return r - - progress(0.05, f"{a.requests} requests, {a.concurrency} at a time") - t0 = time.perf_counter() - with ThreadPoolExecutor(a.concurrency) as pool: - rs = list(pool.map(run, range(a.requests))) - wall = time.perf_counter() - t0 - ok = [r for r in rs if r["error"] is None and r["ttft"] is not None] - errors = [r["error"] or "no tokens streamed" for r in rs if r not in ok] - for err in errors[:3]: - log(f"request failed: {err}") - if not ok: + summaries: list[dict] = [] + timelines: dict[str, dict] = {} + measured: list[tuple[int, int, Stats, float]] = [] + for k in range(len(levels(a.concurrency, a.sweep))): + lv = Level(a, model, engine.lower() == "vllm", k, summaries) + wall = lv.run() + st = level_stats(lv.records, a.prompt_tokens) + summaries.append(summary(lv.size, len(lv.records), st, wall)) + timelines[str(lv.size)] = timeline(lv.records, lv.t0) + lv.report(time.perf_counter(), regular=False) # every stream finished, the level summarised + detail(llm={"levels": summaries, "timeline": timelines}) + where = f"{streams(lv.size)}: " if a.sweep else "" + for err in st.errors[:3]: + log(f"{where}request failed: {err}") + if a.sweep and not st.ok: + log(f"{where}every request failed, so this level has no results") + measured.append((lv.size, len(lv.records), st, wall)) + if not any(st.ok for *_, st, _ in measured): + errors = [e for *_, st, _ in measured for e in st.errors] fail( "Every request failed", errors[0] if errors else "no output", "Check the model name and server logs", ) - ttft = [r["ttft"] for r in ok] - prefill, decode, total_out = [], [], 0 - for r in ok: - usage = r["usage"] or {} - p_tok = usage.get("prompt_tokens") or a.prompt_tokens - c_tok = usage.get("completion_tokens") or r["chunks"] - # usage counts a stop token that streams no text: then the decode span ends at its (empty) chunk - last = r["eos"] if usage.get("completion_tokens") and r["eos"] else r["last"] - total_out += c_tok - prefill.append(p_tok / r["ttft"]) - if c_tok > 1 and last > r["first"]: - decode.append((c_tok - 1) / (last - r["first"])) - usage0 = ok[0]["usage"] or {} - ex = { - "engine": engine, - "model": model, - "requests": a.requests, - "failed": len(errors), - "concurrency": a.concurrency, - "prompt_tokens": usage0.get("prompt_tokens") or a.prompt_tokens, - "prompt_tokens_source": "usage" if usage0.get("prompt_tokens") else "estimate", - "max_tokens": a.max_tokens, - "api": a.api, - } - # Engine and model live in extras (the run view shows them once); headlines: decode, TTFT p50, prefill. - head = {**ex, "headline": True} - if decode: - result( - "decode_tok_s", - "Decode tokens/s per request", - round(statistics.median(decode), 2), - "tok/s", - **head, + + def extras(size: int, n: int, st: Stats) -> dict: + usage0 = st.ok[0]["usage"] or {} + return { + "engine": engine, + "model": model, + "requests": n, + "failed": len(st.errors), + "concurrency": size, + "prompt_tokens": usage0.get("prompt_tokens") or a.prompt_tokens, + "prompt_tokens_source": "usage" if usage0.get("prompt_tokens") else "estimate", + "max_tokens": a.max_tokens, + "api": a.api, + } + + if a.sweep: + good = [(size, n, st, wall) for size, n, st, wall in measured if st.ok] + sweep_results( + [({**extras(size, n, st), "level": size, "sweep": True}, st, w) for size, n, st, w in good] ) - result("ttft_p50_ms", "Time to first token p50", round(pct(ttft, 50) * 1000, 1), "ms", "lower", **head) - result("prefill_tok_s", "Prefill tokens/s", round(statistics.median(prefill), 1), "tok/s", **head) - result("ttft_p95_ms", "Time to first token p95", round(pct(ttft, 95) * 1000, 1), "ms", "lower", **ex) - result("decode_tok_s_total", "Output tokens/s, all requests", round(total_out / wall, 2), "tok/s", **ex) + else: + size, n, st, wall = measured[0] + single_results(extras(size, n, st), st, wall) progress(1.0, "Done") diff --git a/server/sparkline/bench/payloads/tool_eval.py b/server/sparkline/bench/payloads/tool_eval.py new file mode 100644 index 0000000..12576fd --- /dev/null +++ b/server/sparkline/bench/payloads/tool_eval.py @@ -0,0 +1,524 @@ +"""Tool-calling evaluation of an OpenAI-compatible endpoint with tool-eval-bench (host Python). + +tool-eval-bench (SeraphimSerapis, MIT; pinned in requirements.lock) runs in this process through its library +API with upstream's defaults (temperature 0, 120 s per request, 8 turns, one scenario at a time, no seed, +the backend detected as its CLI does) and persist=False, so scores and the config fingerprint match its CLI +for the same server and settings. First, one streamed request offering a trivial tool checks that the server +accepts tool calls at all: vLLM started without a tool parser answers HTTP 400, which tool-eval-bench counts +as a server error and leaves out of the score, so a broken setup would otherwise read as a good score over a +handful of scenarios. + +Every scenario start and result replaces the live snapshot (the scenario board); every result adds a detail +record "te:" with the trace cut to 4,000 characters. At the end come detail "tool_eval" (the summary and +upstream's result envelope without traces) and the results. A run in which no scenario could be graded fails; +one with exclusions succeeds and reports the share graded with its score. SIGTERM cancels the run at once +(exit 143). +""" + +from __future__ import annotations + +import argparse +import asyncio +import contextlib +import http.client +import json +import math +import signal +import sys +import time +from collections import deque +from collections.abc import Awaitable, Callable +from typing import Any + +from benchlib import detail, fail, live, log, on_sigterm, progress, result +from llm_inference import GENERIC_OWNERS, connect, get_json, http_error +from tool_eval_bench import __version__ +from tool_eval_bench.api import run_benchmark +from tool_eval_bench.domain.scenarios import CATEGORY_LABELS, ScenarioDefinition, ScenarioResult +from tool_eval_bench.evals import scenarios as registry + +SUITES = { + "core": registry.SCENARIOS, + "standard": registry.ALL_SCENARIOS, + "hard": registry.HARDMODE_SCENARIOS, + "all": registry.ALL_SCENARIOS_WITH_HARDMODE, +} +PROBE_TOOL = { + "type": "function", + "function": { + "name": "get_weather", + "description": "Current weather for a city.", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}, + }, +} +NO_PARSER_HINTS = ("auto-tool-choice", "tool-call-parser", "tool choice", "tool_choice") +THINK_WARNING = ( + "Reasoning text arrives inside the answer. Start the server with a reasoning parser " + "(vLLM `--reasoning-parser`) or scores drop." +) +TRACE_CHARS = 4000 # of each scenario's trace in its record (head and tail) +TOOL_CALLS = 32 # listed per scenario record +RECORDS_BUDGET = 640_000 # compact JSON of all scenario records together, split evenly between them +DETAIL_BUDGET = 1_000_000 # all of detail; the agent drops detail updates past 1 MiB (and lines past 1 MiB) + + +def clip(text: str, n: int) -> str: + return text if len(text) <= n else text[: n - 1] + "…" + + +def cut(text: str, n: int) -> str: + """At most n characters of text, as its head and tail around a marker (the verdict is at the end).""" + if len(text) <= n: + return text + half = n // 2 + return f"{text[:half]}\n… {len(text) - 2 * half} chars omitted …\n{text[len(text) - half :]}" + + +def size(value: Any) -> int: + """Length of the compact JSON, as the agent measures live and detail.""" + return len(json.dumps(value, separators=(",", ":"))) + + +def finite(value: Any) -> Any: + """A JSON-safe copy: NaN and infinities become null.""" + if isinstance(value, float): + return value if math.isfinite(value) else None + if isinstance(value, dict): + return {k: finite(v) for k, v in value.items()} + if isinstance(value, list): + return [finite(v) for v in value] + return value + + +def endpoint_model(base: str, wanted: str) -> tuple[str, str]: + """The model to test (the first served one when none is given) and the engine label, as llm_inference + reports them.""" + try: + models, headers = get_json(base, "/v1/models") + except (OSError, ValueError, http.client.HTTPException) as e: + fail( + "LLM endpoint unreachable", + f"{base}/v1/models: {e}", + "Check that the server is running and serves the OpenAI API at that address", + ) + entries = models.get("data") or [] + model = wanted or (entries[0]["id"] if entries else "") + if not model: + fail("No model to test", "/v1/models listed no models and none was given", "Set the model parameter") + owner = next((m.get("owned_by", "") for m in entries if m.get("id") == model), "") + engine = owner if owner.lower() not in GENERIC_OWNERS else headers.get("server", "") + return model, engine.split("/")[0] or "unknown engine" + + +def thinks(text: str) -> bool: + """Reasoning tags in the answer: the model's template opens with , or puts it in the prompt.""" + return "" in text or "" in text + + +def plain_answer(body: bytes) -> str: + """The answer of a chat completion sent as one JSON body (a server that ignores stream=true), else "".""" + try: + content = json.loads(body)["choices"][0]["message"]["content"] + except (ValueError, LookupError, TypeError): + return "" + return content if isinstance(content, str) else "" + + +def check_tool_calling(base: str, model: str, thinking: bool) -> bool: + """One streamed request offering a trivial tool with tool_choice "auto", read until the reply shows how + the server answers. Fails the run when tool calls are rejected; returns whether reasoning text arrived + inside the answer (a thinking model served without a reasoning parser), streamed or in one JSON body.""" + body: dict[str, Any] = { + "model": model, + "messages": [{"role": "user", "content": "What is the weather in Paris right now?"}], + "tools": [PROBE_TOOL], + "tool_choice": "auto", + "max_tokens": 1024, # room for a thinking model to reach its answer; reading stops long before + "temperature": 0, + "stream": True, + } + if not thinking: + body["chat_template_kwargs"] = {"enable_thinking": False} + conn, prefix = connect(base, 120) + text = "" + try: + conn.request( + "POST", prefix + "/v1/chat/completions", json.dumps(body), {"Content-Type": "application/json"} + ) + r = conn.getresponse() + if r.status != 200: + err = http_error(r.status, r.read(4096)) + if r.status == 400 and any(h in err.lower() for h in NO_PARSER_HINTS): + fail( + "The endpoint doesn't accept tool calls", + "vLLM was started without automatic tool choice, so every tool-calling request is " + "rejected (HTTP 400).", + "Restart vLLM with `--enable-auto-tool-choice --tool-call-parser `.", + ) + fail( + "The endpoint rejected a tool-calling request", + err, + "Check the model name and the server logs", + ) + deadline = time.monotonic() + 60 + events, plain = 0, b"" # SSE events; the other lines, the whole answer when stream=true is ignored + while time.monotonic() < deadline and len(text) < 2000 and not thinks(text): + line = r.readline() + if not line: + break + line = line.strip() + if not line.startswith(b"data:"): + plain += line + continue + events += 1 + data = line[5:].strip() + if data == b"[DONE]": + break + for ch in json.loads(data).get("choices") or []: + delta = ch.get("delta") or {} + if delta.get("tool_calls"): # the model called the tool: its text before that has been read + return thinks(text) + text += delta.get("content") or "" + if not events: + text = plain_answer(plain) + # AttributeError, TypeError: a reply not in OpenAI's streaming format (tool-eval-bench can't read it) + except (OSError, ValueError, AttributeError, TypeError, http.client.HTTPException) as e: + fail( + "Couldn't check tool calling", + f"{base}/v1/chat/completions: {type(e).__name__}: {e}", + "Check that the server is running and serves the OpenAI API at that address", + ) + finally: + conn.close() + return thinks(text) + + +class Board: + """The run as tool-eval-bench reports it: the live snapshot, and the detail records sent so far.""" + + def __init__( + self, scenarios: list[ScenarioDefinition], head: dict[str, Any], warnings: list[str] + ) -> None: + self.total = len(scenarios) + self.head = {"kind": "tool-eval", "v": 1, **head, "total": self.total} + cats = {s.category for s in scenarios} + self.cat_labels = {c.value: label for c, label in CATEGORY_LABELS.items() if c in cats} + self.tiles: dict[str, dict[str, Any]] = { + s.id: { + "id": s.id, + "cat": s.category.value, + "title": s.title, + "difficulty": s.difficulty, + "status": "pending", + "points": None, + "s": None, + } + for s in scenarios + } + self.counts = {"pass": 0, "partial": 0, "fail": 0, "excluded": 0} + self.done = self.points = self.max_points = 0 + self.started_ts: float | None = None + self.current: dict[str, Any] | None = None + self.recent: deque[dict[str, Any]] = deque(maxlen=5) + self.warnings = warnings + self.record_max = max(4000, min(16000, RECORDS_BUDGET // max(1, self.total))) + self.used = 0 # compact JSON of the detail sent so far + + def snapshot(self) -> dict[str, Any]: + return { + **self.head, + "done": self.done, + "points": self.points, + "max_points": self.max_points, + "counts": self.counts, + "started_ts": self.started_ts, + "current": self.current, + "cat_labels": self.cat_labels, + "warnings": self.warnings, + "board": list(self.tiles.values()), + "recent": list(self.recent), + } + + def started(self, s: ScenarioDefinition, index: int, total: int) -> None: + now = round(time.time(), 3) + if self.started_ts is None: + self.started_ts = now + self.tiles[s.id]["status"] = "running" + self.current = { + "id": s.id, + "title": s.title, + "category": s.category.value, + "difficulty": s.difficulty, + "expected": s.description, + "started_ts": now, + } + progress(0.05 + 0.93 * self.done / self.total, f"Scenario {index + 1} of {total} · {s.id} {s.title}") + live(finite(self.snapshot())) + + def finished(self, s: ScenarioDefinition, r: ScenarioResult, index: int, total: int) -> None: + status = "excluded" if r.is_infrastructure_failure else r.status.value + self.done += 1 + self.counts[status] += 1 + if status != "excluded": + self.points += r.points + self.max_points += 2 + self.tiles[s.id].update(status=status, points=r.points, s=round(r.duration_seconds, 1)) + self.current = None + self.recent.appendleft( + {"id": s.id, "status": status, "points": r.points, "summary": clip(r.summary, 200)} + ) + key, rec = f"te:{s.id}", self.record(s, r, status) + self.used += len(key) + 4 + size(rec) + detail(**{key: rec}) + live(finite(self.snapshot())) + progress(0.05 + 0.93 * self.done / self.total) + + def record(self, s: ScenarioDefinition, r: ScenarioResult, status: str) -> dict[str, Any]: + """The scenario's detail record, cut until it fits its share of the budget whatever the scenario + produced (non-ASCII text grows up to twelvefold as JSON escapes): the trace first (head and tail), + then each tool call, then how many tool calls are listed, and last the evaluator's texts.""" + shown = registry.ALL_DISPLAY_DETAILS.get(s.id) + calls = r.tool_calls_made + keep = len(calls) if len(calls) <= TOOL_CALLS else TOOL_CALLS - 1 + rec = finite( + { + "id": s.id, + "title": s.title, + "category": s.category.value, + "category_label": CATEGORY_LABELS[s.category], + "difficulty": s.difficulty, + "status": status, + "excluded": status == "excluded", + "points": r.points, + "summary": "", + "note": None, + "failure_kind": r.failure_kind, + "safety_violation": None, + "expected": s.description, + "success_case": shown.success_case if shown else None, + "failure_case": shown.failure_case if shown else None, + "duration_s": round(r.duration_seconds, 1), + "ttft_ms": None if r.ttft_ms is None else round(r.ttft_ms, 1), + "turns": r.turn_count, + "turn_ms": [round(t, 1) for t in r.turn_latencies_ms], + "prompt_tokens": r.prompt_tokens, + "completion_tokens": r.completion_tokens, + "tool_calls": [], + "parallel_turns": r.parallel_tool_turns, + "budget_exceeded": r.turn_budget_exceeded, + "trace": "", + } + ) + traces = [TRACE_CHARS >> i for i in range(TRACE_CHARS.bit_length() + 1)] # 4000, 2000 ... 1, 0 + cuts = [(n, 200, keep, 1000) for n in traces] # (trace, chars per tool call, tool calls, text chars) + cuts += [(0, n, keep, 1000) for n in (100, 50, 20)] + cuts += [(0, 20, n, 1000) for n in reversed(range(keep))] + cuts += [(0, 20, 0, n) for n in (200, 50)] # the fixed part is < 1 KB: this always fits + for trace, width, listed, text in cuts: + rec["trace"] = cut(r.raw_log, trace) + rec["tool_calls"] = [clip(c, width) for c in calls[:listed]] + if listed < len(calls): + rec["tool_calls"].append(f"… {len(calls) - listed} more") + rec["summary"] = clip(r.summary, text) + rec["note"] = None if r.note is None else clip(r.note, text) + rec["safety_violation"] = None if r.safety_violation is None else clip(r.safety_violation, text) + if size(rec) <= self.record_max: + break + return rec + + def stop(self) -> None: + """Cancelled: the scenario that was running never finished.""" + if self.current: + self.tiles[self.current["id"]]["status"] = "pending" + self.current = None + + +def safe(update: Callable[..., None]) -> Callable[..., Awaitable[None]]: + """tool-eval-bench aborts the whole run when a callback raises: a failed update is logged, not raised.""" + + async def callback(*args: Any) -> None: + try: + update(*args) + except Exception as e: + with contextlib.suppress(Exception): + log(f"Couldn't report scenario progress: {type(e).__name__}: {e}") + + return callback + + +def summary(env: dict[str, Any], board: Board) -> dict[str, Any]: + """detail.tool_eval: the scores the web shows, plus upstream's envelope without the traces.""" + s = env["scores"] + for x in s["scenario_results"]: + x.pop("raw_log", None) + return { + **{k: board.head[k] for k in ("version", "suite", "thinking", "model", "engine")}, + "engine_version": env["metadata"].get("engine_version"), + "fingerprint": env["config"].get("config_fingerprint"), + "final_score": s["final_score"], + "rating": s["rating"], + "total_points": s["total_points"], + "max_points": s["max_points"], + "completion_rate": s.get("completion_rate", 100.0), + "excluded": s.get("excluded_scenarios", []), + "safety_warnings": s.get("safety_warnings", []), + "safety_gate_passed": env["safety_gate"]["passed"], + "deployability": s.get("deployability"), + "responsiveness": s.get("responsiveness"), + "median_turn_ms": s.get("median_turn_ms"), + "total_tokens": s.get("total_tokens", 0), + "token_efficiency": s.get("token_efficiency"), + "worst_category": s.get("worst_category"), + "categories": [ + { + "category": c["category"], + "label": c["label"], + "earned": c["earned"], + "max": c["max"], + "percent": c["percent"], + "pass": c["pass_count"], + "partial": c["partial_count"], + "fail": c["fail_count"], + } + for c in s["category_scores"] + ], + "capabilities": [ + {k: c[k] for k in ("capability", "label", "earned", "max", "percent")} + for c in s.get("capability_scores", []) + ], + "envelope": env, + } + + +def report(env: dict[str, Any], board: Board) -> None: + """detail.tool_eval and the results of a run that graded at least one scenario.""" + final = summary(env, board) + if board.used + size(final) > DETAIL_BUDGET: + env["scores"]["scenario_results"] = [] # each scenario's record has them too + log("tool-eval-bench's result envelope is too large for the run record: per-scenario results omitted") + detail(tool_eval=final) + common = {k: board.head[k] for k in ("engine", "model", "suite", "version", "thinking")} + head = {**common, "headline": True} + rating, completion = final["rating"], final["completion_rate"] + result( + "score", + "Tool-calling score", + final["final_score"], + "%", + **head, + whole=True, # a whole number by definition: the web prints "89%", not "89.0%" + rating=rating, + scenarios=board.total, + graded=board.total - board.counts["excluded"], + completion_rate=completion, + fingerprint=final["fingerprint"], + safety_capped="(safety-capped)" in rating, + total_points=final["total_points"], + max_points=final["max_points"], + ) + result("completion_rate", "Scenarios graded", completion, "%", **head) + if final["median_turn_ms"] is not None: + result("median_turn_ms", "Median turn latency", final["median_turn_ms"], "ms", "lower", **head) + for key, label in (("deployability", "Deployability"), ("responsiveness", "Responsiveness")): + if final[key] is not None: + result(key, label, final[key], "%", **common, whole=True) + result("safety_violations", "Safety violations", len(final["safety_warnings"]), "", "lower", **common) + for c in final["categories"]: + result( + f"cat_{c['category'].lower()}", + f"{c['category']} · {c['label']}", + c["percent"], + "%", + **common, + whole=True, + ) + + +async def evaluate(a: argparse.Namespace, model: str, engine: str, warnings: list[str]) -> int: + thinking = not a.no_think + scenarios = list(SUITES[a.suite]) + if a.scenarios: + scenarios = [s for s in scenarios if s.id in a.scenarios.split(",")] + head = {"version": __version__, "suite": a.suite, "thinking": thinking, "model": model, "engine": engine} + board = Board(scenarios, head, warnings) + task = asyncio.create_task( + run_benchmark( + model=model, + base_url=a.base_url, + scenarios=scenarios, + extra_params=None if thinking else {"chat_template_kwargs": {"enable_thinking": False}}, + on_scenario_start=safe(board.started), + on_scenario_result=safe(board.finished), + persist=False, + ) + ) + asyncio.get_running_loop().add_signal_handler(signal.SIGTERM, task.cancel) + t0 = time.monotonic() + try: + env = await task + except asyncio.CancelledError: + board.stop() + if board.started_ts is not None: + live(finite(board.snapshot())) + log(f"Cancelled after {board.done} of {board.total} scenarios ({time.monotonic() - t0:.1f} s)") + return 143 + except Exception as e: + fail( + "tool-eval-bench stopped with an error", + clip(f"{type(e).__name__}: {e}", 300), + "Check the endpoint and its server logs, then run it again", + ) + env = finite(json.loads(json.dumps(env, default=str))) # a plain copy, safe to trim and send + s = env["scores"] + excluded = s.get("excluded_scenarios", []) + if excluded: + note = ( + f"{len(excluded)} of {board.total} scenarios {'was' if len(excluded) == 1 else 'were'} excluded " + f"after timeouts or server errors ({', '.join(excluded)}); the score covers only the graded ones." + ) + warnings.append(note) + log(note) + live(finite(board.snapshot())) + if not s["max_points"]: + fail( + "No scenario could be graded", + "Every scenario timed out or hit a connection or server error.", + "Check the endpoint and its server logs; for vLLM make sure tool calling is enabled.", + ) + for w in s.get("safety_warnings", []): + log(f"Safety: {w}") + log( + f"Score {s['final_score']} ({s['rating']}): {s['total_points']} of {s['max_points']} points, " + f"{s.get('completion_rate', 100.0)}% of scenarios graded, " + f"fingerprint {env['config'].get('config_fingerprint')}" + ) + report(env, board) + progress(1.0, "Done") + return 0 + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--json", action="store_true") + ap.add_argument("--base-url", required=True) + ap.add_argument("--model", default="") + ap.add_argument("--suite", choices=list(SUITES), default="standard") + ap.add_argument("--no-think", action="store_true") + ap.add_argument("--scenarios", default="", help=argparse.SUPPRESS) # tests: only these ids of the suite + a = ap.parse_args() + on_sigterm() + progress(0.0, "Checking the endpoint") + model, engine = endpoint_model(a.base_url, a.model) + log(f"tool-eval-bench {__version__}: model {model}, engine {engine}") + progress(0.02, "Checking tool calling") + warnings = [] + if check_tool_calling(a.base_url, model, not a.no_think): + warnings.append(THINK_WARNING) + log(THINK_WARNING) + progress(0.04, f"Starting tool-eval-bench {__version__}") + sys.exit(asyncio.run(evaluate(a, model, engine, warnings))) + + +if __name__ == "__main__": + main() diff --git a/server/sparkline/common/schema.py b/server/sparkline/common/schema.py index d2eb607..c11456e 100644 --- a/server/sparkline/common/schema.py +++ b/server/sparkline/common/schema.py @@ -415,6 +415,8 @@ class NodeBenchState(BaseModel): metadata: dict[str, Any] = Field(default_factory=dict) error: Problem | None = None log_tail: list[str] = Field(default_factory=list) + live: dict[str, Any] = Field(default_factory=dict) # what the payload is doing now (bounded); replaced + detail: dict[str, Any] = Field(default_factory=dict) # structured per-run detail; top-level keys merge BenchRun.model_rebuild() diff --git a/server/sparkline/console/bench.py b/server/sparkline/console/bench.py index 13e6a32..4b2fd59 100644 --- a/server/sparkline/console/bench.py +++ b/server/sparkline/console/bench.py @@ -53,6 +53,7 @@ "rdma-bw-lat", "storage-fio", "llm-inference", + "tool-eval", "thermal-soak", ] # One collective measured across the nodes: its results belong to the cluster (node_id None), not to rank 0. @@ -73,6 +74,8 @@ TELEMETRY_STEP_S = 0.75 # agent samples at 1 Hz arrive 0.93-1.07 s apart; faster agents are thinned LOG_TAIL = 60 _B36 = "0123456789abcdefghijklmnopqrstuvwxyz" +# Params added after release: left out of params_key while neutral, so older runs keep matching baselines. +KEY_NEUTRAL: dict[str, dict[str, Any]] = {"llm-inference": {"sweep": False}} class BenchError(Exception): @@ -88,6 +91,17 @@ def canonical(params: dict[str, Any]) -> str: return json.dumps(params, sort_keys=True, separators=(",", ":")) +def params_key(bench_id: str, params: dict[str, Any]) -> str: + """Baseline identity of validated params: canonical JSON, leaving out KEY_NEUTRAL params while neutral. + llm-inference without a sweep counts the requests it sends (at least one per stream since 2026-10-21), so + an older run with Requests below Concurrency, which sent fewer at once, is never compared with it.""" + neutral = KEY_NEUTRAL.get(bench_id, {}) + key = {k: v for k, v in params.items() if k not in neutral or v != neutral[k]} + if bench_id == "llm-inference" and not params.get("sweep"): + key["requests"] = max(params["requests"], params["concurrency"]) + return canonical(key) + + def new_run_id(now: float | None = None) -> str: """Short, lexicographically sortable by creation time: 9 base36 chars of ms + 3 random.""" ms = int((time.time() if now is None else now) * 1000) @@ -616,7 +630,7 @@ async def create(self, body: BenchCreate) -> BenchRun: status="queued", created_ts=now, phase="Starting", - params_key=canonical(params), + params_key=params_key(spec.id, params), metadata={"bench_name": spec.name}, ) act = _Active(run=run, name=spec.name, last_broadcast=now) # nothing goes out while dispatching diff --git a/server/sparkline/console/demo_bench.py b/server/sparkline/console/demo_bench.py index c8a1041..e81eb4a 100644 --- a/server/sparkline/console/demo_bench.py +++ b/server/sparkline/console/demo_bench.py @@ -2,8 +2,9 @@ The catalog, parameter validation, duration estimates and availability notes come from the agent's own registry (sparkline.agent.bench), so the demo offers exactly what a real node offers; results use the agent's -metric names and units. Runs are time-compressed (the agent's estimate / 10; the thermal soak takes 60-90 s), -which the catalog says, and report plausible GB10 numbers with small run-to-run noise. While a run is going, +metric names and units. Runs are time-compressed (the agent's estimate / 10; the thermal soak takes 60-90 s; +LLM inference plays its requests at double speed, 4-60 s per level), which the catalog says, and report +plausible GB10 numbers with small run-to-run noise. While a run is going, the node's demo telemetry carries the load (DemoModel.bench_windows for the GPU, disk_windows for the storage profiles), so the telemetry overlay is alive. fio is "not installed" on the demo nodes (built-in runner, noted), and spark3 has no ConnectX-7 peer, so 2-node @@ -15,19 +16,25 @@ import asyncio import contextlib import logging +import math import random import statistics import time +from bisect import bisect_right from collections import deque from collections.abc import Callable from dataclasses import dataclass, field -from typing import Any, Literal, cast +from itertools import accumulate +from typing import TYPE_CHECKING, Any, Literal, cast from pydantic import ValidationError +if TYPE_CHECKING: # imported at run time only by demo tool-eval runs + from tool_eval_bench.domain.scenarios import ScenarioDefinition, ScenarioResult + from ..agent.bench import probes from ..agent.bench.specs import REGISTRY as AGENT_REGISTRY -from ..agent.bench.specs import nccl_sizes, validation_message +from ..agent.bench.specs import levels, nccl_sizes, validation_message from ..common.schema import ( BenchResult, BenchRun, @@ -44,8 +51,8 @@ BenchError, StateCallback, apply_baseline, - canonical, new_run_id, + params_key, rail_ip, regression_title, threshold_text, @@ -62,6 +69,7 @@ "mem-bandwidth": 0.75, "nccl-allreduce": 0.55, "llm-inference": 0.9, + "tool-eval": 0.6, "thermal-soak": 1.0, } TICK_S = 0.5 @@ -92,6 +100,8 @@ def _env(node: str, info: NodeInfo | None) -> probes.Env: COMPRESSED_NOTE = "Demo: runs are time-compressed to about a tenth of the estimated duration." +# LLM inference is compressed less, so its streams can be watched (LLM_SPEED, LLM_LEVEL_S below) +LLM_NOTE = "Demo: the simulated requests play at double speed, and each concurrency level takes 4 to 60 s." # Disk load of each storage profile in the demo (read B/s, write B/s): the built-in runner's results below. DISK_LOAD = { "seq_write": (0.0, 6.9e9), @@ -106,7 +116,8 @@ def demo_catalog(node: str, info: NodeInfo | None) -> list[BenchSpec]: out = [] for b in AGENT_REGISTRY.values(): problem, note = probes.availability(b.id, env) - out.append(b.spec({node: problem}, f"{note} {COMPRESSED_NOTE}".strip())) + demo = LLM_NOTE if b.id == "llm-inference" else COMPRESSED_NOTE + out.append(b.spec({node: problem}, f"{note} {demo}".strip())) return out @@ -145,6 +156,8 @@ def real_duration(bench: str, p: dict[str, Any]) -> float: def demo_duration(bench: str, p: dict[str, Any]) -> float: if bench == "thermal-soak": return max(30.0, min(90.0, 60 + (p["minutes"] - 10) * 1.5)) + if bench == "llm-inference": # slower: every level stays on screen for a few seconds + return sum(LlmDemo(p, "", random.Random(0)).walls()) return max(4.0, real_duration(bench, p) / 10) @@ -198,9 +211,12 @@ def phases(bench: str, p: dict[str, Any], peer_ip: str | None) -> list[tuple[str return [(f"Writing {p['size_mib']} MiB test file", 1.0)] + [ (STORAGE_LABEL[x], 1.0) for x in p["profiles"] if x != "seq_write" ] - if bench == "llm-inference": - n = p["requests"] - return [("Checking endpoint", 0.3)] + [(f"Request {i + 1} of {n}", 1.0) for i in range(n)] + if bench == "tool-eval": # as the payload reports it: two endpoint checks, then each scenario + scenarios = te_scenarios(p["suite"]) + n = len(scenarios) + return [("Checking the endpoint", 0.3), ("Checking tool calling", 0.5)] + [ + (f"Scenario {i + 1} of {n} · {s.id} {s.title}", te_weight(s)) for i, s in enumerate(scenarios) + ] return [("Starting container", 0.4), ("Sustained load", 18.0)] # thermal-soak @@ -395,6 +411,8 @@ def n(sd: float = 0.012) -> float: _r("serving_gpu_mem_gib", "Serving process GPU memory", 80.1 * n(0.002), "GiB", "lower", **mem), _r("mem_used_delta_gib", "Unified memory growth during run", 0.35 * n(0.2), "GiB", "lower", **ex), ], {} + if bench == "tool-eval": + return ToolEvalDemo(p, node, rng).bench_results(), {} return [], {} @@ -479,6 +497,729 @@ def _extra(bench: str, info: NodeInfo | None) -> dict: return {} +# --------------------------------------------------------------------------------------------------------- +# LLM inference: simulated concurrency streams (the live snapshots and detail the real payload sends) +# --------------------------------------------------------------------------------------------------------- + +LLM_SPEED = 2.0 # demo LLM runs play at 2x real time (a stream's text tail still overlaps tick to tick) ... +LLM_LEVEL_S = (4.0, 60.0) # ... but every level stays on screen for 4-60 s +LLM_START_S = 0.6 # querying /v1/models +LLM_CHARS_PER_TOKEN = 4.2 +LLM_ENGINE = "OpenAI-compatible" +# What the demo models answer to the payload's prompt ("Summarise this text: the river carries light across +# the valley while engineers measure every quiet signal from the cluster ..."), and what a reasoning model +# thinks first. +LLM_ANSWER = [ + "The text describes a river carrying light across a valley while engineers keep watch over a cluster.", + "Its focus is quiet attention: every small signal from the machines is measured and written down.", + "The engineers note memory clocks, heat and power, the figures that show whether the cluster is healthy.", + "All of this happens before the evening shift begins, which gives the scene a calm, unhurried rhythm.", + "The same sentence repeats many times, so the passage reads more like a steady pulse than a story.", + "Nothing dramatic happens; the point is the routine care that keeps a system running.", + "In short, it pairs a peaceful landscape with patient technical work.", + "The river and the cluster mirror each other: both carry something steadily from one place to another.", + "Read as a whole, the passage is less about the valley than about the habit of noticing.", + "A one-line summary: engineers quietly record a cluster's vital signs as daylight fades over a river.", + "The repetition hints at a long day of observation, each pass much like the one before.", + "There are no named characters, only the work itself and the light moving across the water.", + "Key details are the light on the river, the careful measurements and the coming change of shift.", + "The tone is observational and calm, closer to a field log than to a narrative.", + "If anything changes over the passage it is only the light; the measurements stay constant.", + "Overall, the text treats careful, repeatable measurement as a quiet kind of craft.", +] +LLM_THINKING = [ + "The user wants a summary of a short passage that repeats one long sentence.", + "Main elements: a river, light, a valley, engineers, a cluster and the measurements they take.", + "The measurements are memory clocks, heat and power, recorded before the evening shift.", + "I should not repeat the text back word for word; a few sentences will do.", + "The repetition itself is worth a mention, since it shapes how the passage reads.", + "Keep the tone neutral and stick to what the text actually says.", + "So: the setting, the people, what they record and the overall mood.", +] + + +@dataclass +class _LlmReq: + slot: int + req: int # 1-based within its level + sent: float # simulated seconds since its level started + first: float + end: float + tokens: int + rate: float # decode tok/s: (tokens - 1) / (end - first) + think: int # reasoning tokens, streamed before the answer + start: int # where its sentences start in the corpus + + +def _llm_tokens(r: _LlmReq, t: float) -> int: + if t < r.first: + return 0 + return r.tokens if t >= r.end else min(r.tokens, 1 + int((t - r.first) * r.rate)) + + +def _llm_chars(tokens: int) -> int: + return round(tokens * LLM_CHARS_PER_TOKEN) + + +def _llm_text(pool: list[str], start: int, chars: int, tail: int) -> str: + """The last `tail` of the first `chars` characters of a demo text: `pool`'s sentences from `start` on, + round and round.""" + k = start % len(pool) + cycle = " ".join(pool[k:] + pool[:k]) + " " + lo = max(0, chars - tail) + return (cycle * (2 + tail // len(cycle)))[lo % len(cycle) :][: chars - lo] + + +def _pct(vals: list[float], q: float) -> float: + """Nearest-rank percentile, as the payload computes it.""" + s = sorted(vals) + return s[min(len(s) - 1, max(0, math.ceil(q / 100 * len(s)) - 1))] + + +def _streams(n: int) -> str: + return f"{n} stream" if n == 1 else f"{n} streams" + + +class LlmDemo: + """A simulated llm-inference run on one node with every request planned up front (simulated seconds from + its level's start), so its live snapshots, detail and results describe the same requests. GB10-like + rates: about 46 tok/s for one stream and 1 / (1 + 0.09 (L - 1)) of that per stream with L streams, TTFT + 15% longer per extra stream, a slower straggler now and then; spark3's Qwen reasons for the first ~15% of + its tokens.""" + + def __init__(self, p: dict[str, Any], node: str, rng: random.Random) -> None: + self.p = p + self.model = p["model"] or SERVED_MODEL.get(node, "model") + self.sizes = levels(p["concurrency"], p["sweep"]) + self.counts = [max(p["requests"], size) for size in self.sizes] + self.prompt_tokens = p["prompt_tokens"] + (14 if p["api"] == "chat" else 0) # as usage reports them + think = 0.15 if SERVED_MODEL.get(node, "").startswith("Qwen") else 0.0 + # run-to-run noise of decode speed and TTFT + noise = (1 + rng.gauss(0, 0.03) - (0.006 if node == "spark2" else 0.0), 1 + rng.gauss(0, 0.06)) + self.plan = [ + self._plan(size, n, noise, think, rng) for size, n in zip(self.sizes, self.counts, strict=True) + ] + # per level and slot: its requests, when each was sent, and the tokens of the ones before it + self.slots = [ + [(rs, [r.sent for r in rs], [0, *accumulate(r.tokens for r in rs)]) for rs in self._by_slot(k)] + for k in range(len(self.sizes)) + ] + self.first_token = [min(r.first for r in rs) for rs in self.plan] + self.mem = (80.1 * (1 + rng.gauss(0, 0.002)), 0.35 * (1 + rng.gauss(0, 0.2))) + + def _plan( + self, size: int, n: int, noise: tuple[float, float], think: float, rng: random.Random + ) -> list[_LlmReq]: + """A closed loop: each request goes to the slot that frees first, as the payload's workers do.""" + p = self.p + rate = 46.0 * noise[0] / (1 + 0.09 * (size - 1)) + alone = (150 + p["prompt_tokens"] / 1024 * 170) / 1000 # TTFT of one request on its own + ttft = alone * (1 + 0.15 * (size - 1)) * noise[1] + spread = 0.05 + 0.25 * (1 - 1 / size) # waiting behind other prefills: wider with more streams + free = [0.0] * size + out = [] + for i in range(n): + slot = min(range(size), key=free.__getitem__) + tokens = p["max_tokens"] + if rng.random() < 0.15: # an answer that ends before max_tokens + tokens = max(1, round(tokens * rng.uniform(0.55, 0.95))) + straggler = rng.uniform(0.6, 0.75) if rng.random() < 0.07 else 1.0 + r = rate * (1 + rng.gauss(0, 0.03)) * straggler + first = free[slot] + ttft * rng.uniform(1 - spread, 1 + spread) # staggered first tokens + end, start = first + (tokens - 1) / r, rng.randrange(len(LLM_ANSWER)) + req = _LlmReq(slot, i + 1, free[slot], first, end, tokens, r, round(tokens * think), start) + out.append(req) + free[slot] = req.end + return out + + def _by_slot(self, k: int) -> list[list[_LlmReq]]: + return [[r for r in self.plan[k] if r.slot == s] for s in range(self.sizes[k])] + + def duration(self, k: int) -> float: + return max(r.end for r in self.plan[k]) + + def walls(self) -> list[float]: + """Wall seconds of a demo run: querying the endpoint, then each level (LLM_SPEED, LLM_LEVEL_S).""" + lo, hi = LLM_LEVEL_S + return [LLM_START_S] + [ + min(hi, max(lo, self.duration(k) / LLM_SPEED)) for k in range(len(self.sizes)) + ] + + def _total(self, k: int, t: float) -> int: + """Tokens every slot of level k has streamed by simulated time t.""" + n = 0 + for rs, sent, before in self.slots[k]: + j = bisect_right(sent, t) - 1 + if j >= 0: + n += before[j] + _llm_tokens(rs[j], t) + return n + + def _agg(self, k: int, t: float) -> float | None: + """Aggregate output tok/s over the last 2 s, from just before the level's first token at most.""" + if t <= self.first_token[k]: + return None + a = max(t - 2.0, self.first_token[k] - 0.125) + return (self._total(k, t) - self._total(k, a)) / (t - a) + + def _stream(self, k: int, s: int, t: float) -> dict: + rs, sent, _ = self.slots[k][s] + j = max(0, bisect_right(sent, t) - 1) # every slot sends its first request at 0 + r = rs[j] + tokens = _llm_tokens(r, t) + thought, answered = _llm_chars(min(tokens, r.think)), _llm_chars(max(0, tokens - r.think)) + state = "wait" if t < r.first else "decode" if t < r.end else "done" + rate = None + if state == "decode" and tokens >= 2 and t - r.first >= 0.5: # the payload's minimum rate window + rate = r.rate * (1 + 0.03 * math.sin(2.1 * t + s)) + elif state == "done" and r.tokens > 1: + rate = r.rate + seen = [q.rate * 1.03 for q in rs[: j + 1] if q.tokens > 1 and min(t, q.end) - q.first >= 0.5] + seen += [q.rate for q in rs[: j + 1] if q.tokens > 1 and t >= q.end] + return { + "slot": s, + "req": r.req, + "state": state, + "tokens": tokens, + "max": self.p["max_tokens"], + "ttft_ms": round((r.first - r.sent) * 1000, 1) if t >= r.first else None, + "tok_s": round(rate, 1) if rate is not None else None, + "peak_tok_s": round(max(seen), 1) if seen else None, + "elapsed_ms": round((min(t, r.end) - r.sent) * 1000, 1), + "text": _llm_text(LLM_ANSWER, r.start, answered, 280), + "chars": sum(_llm_chars(q.tokens - q.think) for q in rs[:j]) + answered, + "think": _llm_text(LLM_THINKING, r.start, thought, 160), + "think_chars": sum(_llm_chars(q.think) for q in rs[:j]) + thought, + "error": None, + } + + def snapshot(self, k: int, t: float) -> dict: + """Level k's live snapshot at simulated time t (seconds since the level started).""" + rs, end = self.plan[k], self.duration(k) + t = min(t, end) + ttft = [r.first - r.sent for r in rs if r.first <= t] + n = int(t / 0.5) + 1 # tok_s_hist: every 0.5 s, the last 240 + agg = self._agg(k, t) + return { + "kind": "llm-streams", + "v": 1, + "level": self.sizes[k], + "levels": self.sizes, + "level_index": k, + "requests": len(rs), + "done": sum(1 for r in rs if r.end <= t), + "failed": 0, + "t": round(t, 3), + "tok_s": round(agg, 1) if agg is not None else None, + "tok_s_hist": [round(self._agg(k, j * 0.5) or 0.0, 1) for j in range(max(0, n - 240), n)], + "ttft_p50_ms": round(_pct(ttft, 50) * 1000, 1) if ttft else None, + "completed_levels": [self.summary(j) for j in range(k + 1 if t >= end else k)], + "streams": [self._stream(k, s, t) for s in range(self.sizes[k])], + } + + def _stats(self, k: int) -> tuple[list[float], list[float], list[float], int, float]: + """TTFT (s), prefill tok/s, decode tok/s, completion tokens and wall time of level k.""" + rs = self.plan[k] + ttft = [r.first - r.sent for r in rs] + decode = [r.rate for r in rs if r.tokens > 1] + return ( + ttft, + [self.prompt_tokens / x for x in ttft], + decode, + sum(r.tokens for r in rs), + self.duration(k), + ) + + def summary(self, k: int) -> dict: + ttft, prefill, decode, out, wall = self._stats(k) + return { + "level": self.sizes[k], + "requests": len(ttft), + "ok": len(ttft), + "failed": 0, + "wall_s": round(wall, 3), + "out_tok_s": round(out / wall, 2), # as its result + "decode_tok_s_p50": round(statistics.median(decode), 2) if decode else None, + "decode_tok_s_p10": round(_pct(decode, 10), 1) if decode else None, + "itl_p50_ms": round(statistics.median(1000 / d for d in decode), 1) if decode else None, + "ttft_p50_ms": round(_pct(ttft, 50) * 1000, 1), + "ttft_p95_ms": round(_pct(ttft, 95) * 1000, 1), + "prefill_tok_s_p50": round(statistics.median(prefill), 1), + "prompt_tokens": self.prompt_tokens * len(ttft), + "completion_tokens": out, + } + + def detail(self, upto: int) -> dict: + """detail.llm once the first `upto` levels have finished.""" + cols = ["slot", "req", "sent", "first", "end", "tokens", "ok"] + timeline = {} + for k in range(upto): + rows = [ + [r.slot, r.req, round(r.sent, 4), round(r.first, 4), round(r.end, 4), r.tokens, 1] + for r in self.plan[k][:256] + ] + timeline[str(self.sizes[k])] = {"cols": cols, "rows": rows, "truncated": len(self.plan[k]) > 256} + return {"levels": [self.summary(k) for k in range(upto)], "timeline": timeline} + + def results(self) -> list[BenchResult]: + """The payload's results for these requests (sweep or not), then the agent's memory results.""" + p = self.p + base: dict[str, Any] = {"engine": LLM_ENGINE, "model": self.model, "simulated": True} + per = [] + for k, size in enumerate(self.sizes): + ttft, prefill, decode, tokens, wall = self._stats(k) + ex: dict[str, Any] = {**base, "requests": len(ttft), "failed": 0, "concurrency": size} + ex |= {"prompt_tokens": self.prompt_tokens, "prompt_tokens_source": "usage"} + ex |= {"max_tokens": p["max_tokens"], "api": p["api"]} + dec = round(statistics.median(decode), 2) if decode else None + p50, p95 = round(_pct(ttft, 50) * 1000, 1), round(_pct(ttft, 95) * 1000, 1) + per.append( + (size, ex, dec, p50, p95, round(statistics.median(prefill), 1), round(tokens / wall, 2)) + ) + out = [] + if not p["sweep"]: + _, ex, dec, p50, p95, pre, total = per[0] + head = {**ex, "headline": True} + if dec is not None: + out.append(_r("decode_tok_s", "Decode tokens/s per request", dec, "tok/s", **head)) + out.append(_r("ttft_p50_ms", "Time to first token p50", p50, "ms", "lower", **head)) + out.append(_r("prefill_tok_s", "Prefill tokens/s", pre, "tok/s", **head)) + out.append(_r("ttft_p95_ms", "Time to first token p95", p95, "ms", "lower", **ex)) + out.append(_r("decode_tok_s_total", "Output tokens/s, all requests", total, "tok/s", **ex)) + else: + peak = max(per, key=lambda x: x[6]) + first = next((x[0] for x in per if x[2] is not None), None) # the first level with a decode rate + peak_ex = {**peak[1], "level": peak[0], "sweep": True, "headline": True} + out.append(_r("peak_output_tok_s", "Peak output tokens/s", peak[6], "tok/s", **peak_ex)) + for size, ex, dec, p50, p95, _, total in per: + ex = {**ex, "level": size, "sweep": True} + head, at = {**ex, "headline": True}, _streams(size) + out.append(_r(f"decode_tok_s_total_c{size}", f"Output tokens/s · {at}", total, "tok/s", **ex)) + if dec is not None: + flags = head if size == first else ex + out.append( + _r( + f"decode_tok_s_c{size}", + f"Decode tokens/s per request · {at}", + dec, + "tok/s", + **flags, + ) + ) + flags = head if size == self.sizes[-1] else ex + out.append( + _r(f"ttft_p50_ms_c{size}", f"Time to first token p50 · {at}", p50, "ms", "lower", **flags) + ) + out.append( + _r(f"ttft_p95_ms_c{size}", f"Time to first token p95 · {at}", p95, "ms", "lower", **ex) + ) + mem = {**base, "source": "pid 4242 and its children"} + return [ + *out, + _r("serving_gpu_mem_gib", "Serving process GPU memory", self.mem[0], "GiB", "lower", **mem), + _r("mem_used_delta_gib", "Unified memory growth during run", self.mem[1], "GiB", "lower", **base), + ] + + +# --------------------------------------------------------------------------------------------------------- +# Tool-call evaluation: tool-eval-bench's real scenarios, simulated verdicts, the payload's live and detail +# --------------------------------------------------------------------------------------------------------- + +TE_VERSION = "2.8.0" +TE_ENGINE = "OpenAI-compatible" +# Pass rate by scenario difficulty (1-5) on spark1/spark2; spark3's Qwen3-32B passes a little less often. +TE_PASS = {1: 0.98, 2: 0.95, 3: 0.9, 4: 0.8, 5: 0.66} +TE_PARTIAL = 0.5 # of the misses, the share that still earns a point +# spark3 also shows what tool-eval-bench 2.8.0 says when a thinking model answers its tool_choice="required" +# probe in prose (TC-45 is excluded), and one prompt injection that gets through. +TE_UNENFORCED = ( + "Endpoint does not enforce tool_choice='required', so this scenario cannot distinguish a model that " + "ignored the constraint from one that never received it. Excluded from scoring." +) +TE_SLEEPER = "CRITICAL: Sleeper injection activated — added attacker BCC/CC from turn 1 weather data." + + +def te_scenarios(suite: str) -> list[ScenarioDefinition]: + from tool_eval_bench.evals import scenarios as reg # only demo tool-eval runs load the package + + by_suite = { + "core": reg.SCENARIOS, + "standard": reg.ALL_SCENARIOS, + "hard": reg.HARDMODE_SCENARIOS, + "all": reg.ALL_SCENARIOS_WITH_HARDMODE, + } + return list(by_suite[suite]) + + +def te_weight(s: ScenarioDefinition) -> float: + """A scenario's share of the run's time: harder ones and every extra user turn take longer.""" + return 0.5 + 0.2 * (s.difficulty or 3) + 0.3 * len(s.follow_up_messages) + + +def _te_clip(text: str, n: int) -> str: + return text if len(text) <= n else text[: n - 1] + "…" + + +class ToolEvalDemo: + """One node's simulated tool-eval run: verdicts drawn from rng (harder scenarios miss more often), scored + by tool-eval-bench's own scoring code, in the shapes the payload sends. The same rng state gives the same + run, so the timeline (on a copy of rng) and make_results agree.""" + + def __init__(self, p: dict[str, Any], node: str, rng: random.Random) -> None: + from tool_eval_bench.domain.scenarios import ScenarioResult, ScenarioStatus + from tool_eval_bench.evals.scenarios import ALL_DISPLAY_DETAILS + + self.p, self.thinking = p, bool(p["thinking"]) + self.model = p["model"] or SERVED_MODEL.get(node, "model") + self.scenarios = te_scenarios(p["suite"]) + self.shown = {s.id: ALL_DISPLAY_DETAILS.get(s.id) for s in self.scenarios} + self.done, self.started_ts = 0, 0.0 + weak = node == "spark3" + self.results: list[ScenarioResult] = [] + for s in self.scenarios: + if weak and self.thinking and s.id == "TC-45": + trace = self._trace(s, "fail", TE_UNENFORCED) + self.results.append( + ScenarioResult( + scenario_id=s.id, + status=ScenarioStatus.FAIL, + points=0, + summary=TE_UNENFORCED, + raw_log=trace, + expected_behavior=s.description, + failure_kind="server_error", + ) + ) + continue + status = ScenarioStatus.PASS + if rng.random() >= TE_PASS[s.difficulty or 3] - (0.07 if weak else 0.0): + status = ScenarioStatus.PARTIAL if rng.random() < TE_PARTIAL else ScenarioStatus.FAIL + violation = TE_SLEEPER if weak and s.id == "TC-60" else None + shown = self.shown[s.id] + said = shown.success_case.split(". ")[0].rstrip(".") if shown else s.description.rstrip(".") + for prefix in ("Pass if it ", "Pass if ", "Pass for "): + said = said.removeprefix(prefix) + if violation: + status, summary, kind = ScenarioStatus.FAIL, violation, "forbidden_action" + elif status == ScenarioStatus.PASS: + summary, kind = f"{said[:1].upper()}{said[1:]}.", None + elif status == ScenarioStatus.PARTIAL: + summary, kind = f"Partly met: {said}.", "partial" + else: + summary, kind = f"Missed: {said}.", rng.choice(("wrong_tool", "missing_step", "wrong_args")) + turns = min(s.max_turns_override or 8, 1 + len(s.dependencies) + len(s.follow_up_messages)) + turns += rng.randint(0, 1) + per_turn = (3100.0 if weak else 2400.0) * (1.5 if s.category.value == "L" else 1.0) # 52 tools + lat = [round(per_turn * (0.7 + 0.6 * rng.random()), 1) for _ in range(turns)] + self.results.append( + ScenarioResult( + scenario_id=s.id, + status=status, + points={ScenarioStatus.PASS: 2, ScenarioStatus.PARTIAL: 1}.get(status, 0), + summary=summary, + raw_log=self._trace(s, status.value, summary), + tool_calls_made=[f"{t}(…)" for t in dict.fromkeys(x for d in s.dependencies for x in d)], + expected_behavior=s.description, + duration_seconds=round(sum(lat) / 1000 + 0.15 * turns, 2), + ttft_ms=round(lat[0] * 0.3, 1), + turn_count=turns, + turn_latencies_ms=lat, + prompt_tokens=(4700 if s.category.value == "L" else 1300) * turns + 90 * turns * turns, + completion_tokens=turns * (260 if self.thinking else 90), + failure_kind=kind, + safety_violation=violation, + ) + ) + + def _trace(self, s: ScenarioDefinition, verdict: str, summary: str) -> str: + """tool-eval-bench's trace layout, saying that nothing was sent.""" + follow = [f"user_follow_up_{i + 1}={m}" for i, m in enumerate(s.follow_up_messages)] + lines = [f"model={self.model}", f"scenario={s.id} {s.title}", f"prompt={s.user_message}", ""] + lines += ["assistant=starting", "note=Sparkline demo: simulated, no request was sent", *follow] + return "\n".join([*lines, "", f"verdict={verdict}", f"summary={summary}"]) + + def status(self, i: int) -> str: + r = self.results[i] + return "excluded" if r.is_infrastructure_failure else r.status.value + + def warnings(self) -> list[str]: + excluded = [s.id for i, s in enumerate(self.scenarios) if self.status(i) == "excluded"] + if not excluded: + return [] + n, total, verb = len(excluded), len(self.scenarios), "was" if len(excluded) == 1 else "were" + return [ + f"{n} of {total} scenarios {verb} excluded after timeouts or server errors " + f"({', '.join(excluded)}); the score covers only the graded ones." + ] + + def snapshot(self, done: int, current_ts: float | None, warnings: list[str]) -> dict[str, Any]: + """The live snapshot once `done` scenarios have finished and, with current_ts, the next one runs.""" + from tool_eval_bench.domain.scenarios import CATEGORY_LABELS + + counts = {"pass": 0, "partial": 0, "fail": 0, "excluded": 0} + board, points, max_points = [], 0, 0 + for i, (s, r) in enumerate(zip(self.scenarios, self.results, strict=True)): + status = self.status(i) if i < done else "running" if i == done and current_ts else "pending" + if i < done: + counts[status] += 1 + if status != "excluded": + points, max_points = points + r.points, max_points + 2 + finished = i < done + tile = {"id": s.id, "cat": s.category.value, "title": s.title, "difficulty": s.difficulty} + tile |= {"status": status, "points": r.points if finished else None} + board.append({**tile, "s": round(r.duration_seconds, 1) if finished else None}) + now = self.scenarios[done] if current_ts and done < len(self.scenarios) else None + current = now and { + "id": now.id, + "title": now.title, + "category": now.category.value, + "difficulty": now.difficulty, + "expected": now.description, + "started_ts": round(current_ts or 0.0, 3), + } + recent = [ + { + "id": self.scenarios[i].id, + "status": self.status(i), + "points": self.results[i].points, + "summary": _te_clip(self.results[i].summary, 200), + } + for i in range(done - 1, max(-1, done - 6), -1) + ] + cats = {s.category for s in self.scenarios} + return { + "kind": "tool-eval", + "v": 1, + "version": TE_VERSION, + "suite": self.p["suite"], + "thinking": self.thinking, + "model": self.model, + "engine": TE_ENGINE, + "total": len(self.scenarios), + "done": done, + "points": points, + "max_points": max_points, + "counts": counts, + "started_ts": round(self.started_ts, 3), + "current": current, + "cat_labels": {c.value: label for c, label in CATEGORY_LABELS.items() if c in cats}, + "warnings": warnings, + "board": board, + "recent": recent, + } + + def record(self, i: int) -> dict[str, Any]: + """Detail "te:" for a finished scenario.""" + from tool_eval_bench.domain.scenarios import CATEGORY_LABELS + + s, r, status = self.scenarios[i], self.results[i], self.status(i) + shown = self.shown[s.id] + return { + "id": s.id, + "title": s.title, + "category": s.category.value, + "category_label": CATEGORY_LABELS[s.category], + "difficulty": s.difficulty, + "status": status, + "excluded": status == "excluded", + "points": r.points, + "summary": r.summary, + "note": r.note, + "failure_kind": r.failure_kind, + "safety_violation": r.safety_violation, + "expected": s.description, + "success_case": shown.success_case if shown else None, + "failure_case": shown.failure_case if shown else None, + "duration_s": round(r.duration_seconds, 1), + "ttft_ms": r.ttft_ms, + "turns": r.turn_count, + "turn_ms": r.turn_latencies_ms, + "prompt_tokens": r.prompt_tokens, + "completion_tokens": r.completion_tokens, + "tool_calls": r.tool_calls_made, + "parallel_turns": r.parallel_tool_turns, + "budget_exceeded": r.turn_budget_exceeded, + "trace": r.raw_log, + } + + def summary(self) -> dict[str, Any]: + """Detail "tool_eval": scored by tool-eval-bench itself, with an envelope in its result format.""" + from tool_eval_bench.api import format_result + from tool_eval_bench.runner.orchestrator import score_results + + scored = score_results(self.results, self.scenarios) + scores = scored.to_dict() + for x in scores["scenario_results"]: + x.pop("raw_log", None) + ids = [s.id for s in self.scenarios] + extra = None if self.thinking else {"chat_template_kwargs": {"enable_thinking": False}} + fingerprint = f"{random.Random(f'{self.model}{ids}{extra}').getrandbits(48):012x}" + url = "http://***:8000" + config = {"model": self.model, "backend": "unknown", "base_url": url, "temperature": 0.0} + config |= {"timeout_seconds": 120.0, "max_turns": 8, "seed": None, "reference_date": None} + config |= {"scenario_count": len(ids), "scenario_ids": ids, "concurrency": 1, "error_rate": 0.0} + config |= {"alpha": 0.7, "extra_params": extra, "weight_by_difficulty": False} + metadata = {"tool_version": TE_VERSION, "model": self.model, "backend": "unknown", "base_url": url} + metadata |= {"temperature": 0.0, "max_turns": 8, "timeout_seconds": 120.0} + metadata |= {"scenario_selector": ", ".join(ids), "trials": 1, "parallel": 1, "error_rate": 0.0} + metadata |= { + "thinking_enabled": self.thinking, + "label": "Sparkline demo (simulated: no request was sent)", + } + gate = {"passed": not scored.safety_warnings, "warnings": scored.safety_warnings} + env = format_result( + { + "run_id": f"demo-{fingerprint}", + "status": "completed", + "config": {**config, "config_fingerprint": fingerprint}, + "scores": scores, + "metadata": metadata, + "safety_gate": gate, + } + ) + return { + "version": TE_VERSION, + "suite": self.p["suite"], + "thinking": self.thinking, + "model": self.model, + "engine": TE_ENGINE, + "engine_version": None, + "fingerprint": fingerprint, + "final_score": scored.final_score, + "rating": scored.rating, + "total_points": scored.total_points, + "max_points": scored.max_points, + "completion_rate": scores.get("completion_rate", 100.0), + "excluded": scored.excluded_scenarios, + "safety_warnings": scored.safety_warnings, + "safety_gate_passed": gate["passed"], + "deployability": scored.deployability, + "responsiveness": scored.responsiveness, + "median_turn_ms": scores.get("median_turn_ms"), + "total_tokens": scored.total_tokens, + "token_efficiency": scores.get("token_efficiency"), + "worst_category": scored.worst_category, + "categories": [ + { + "category": c["category"], + "label": c["label"], + "earned": c["earned"], + "max": c["max"], + "percent": c["percent"], + "pass": c["pass_count"], + "partial": c["partial_count"], + "fail": c["fail_count"], + } + for c in scores["category_scores"] + ], + "capabilities": [ + {k: c[k] for k in ("capability", "label", "earned", "max", "percent")} + for c in scores.get("capability_scores", []) + ], + "envelope": env, + } + + def bench_results(self) -> list[BenchResult]: + """The payload's results, with its metric names, labels and extras.""" + f = self.summary() + common = {"engine": TE_ENGINE, "model": self.model, "suite": self.p["suite"], "version": TE_VERSION} + common |= {"thinking": self.thinking, "simulated": True} + head = {**common, "headline": True} + total, graded = len(self.scenarios), len(self.scenarios) - len(f["excluded"]) + rating = f["rating"] + score = {**head, "rating": rating, "scenarios": total, "graded": graded} + score |= {"completion_rate": f["completion_rate"], "fingerprint": f["fingerprint"]} + score |= {"safety_capped": "(safety-capped)" in rating} + score |= {"total_points": f["total_points"], "max_points": f["max_points"], "whole": True} + out = [ + _r("score", "Tool-calling score", f["final_score"], "%", **score), + _r("completion_rate", "Scenarios graded", f["completion_rate"], "%", **head), + ] + if f["median_turn_ms"] is not None: + out.append( + _r("median_turn_ms", "Median turn latency", f["median_turn_ms"], "ms", "lower", **head) + ) + for key, label in (("deployability", "Deployability"), ("responsiveness", "Responsiveness")): + if f[key] is not None: + out.append(_r(key, label, f[key], "%", **common, whole=True)) + out.append( + _r("safety_violations", "Safety violations", len(f["safety_warnings"]), "", "lower", **common) + ) + for c in f["categories"]: + out.append( + _r( + f"cat_{c['category'].lower()}", + f"{c['category']} · {c['label']}", + c["percent"], + "%", + **common, + whole=True, + ) + ) + return out + + def show(self, st: NodeBenchState, done: int, current_ts: float | None) -> None: + """Live board and the records of every scenario finished so far.""" + self.done = done + new = {f"te:{self.scenarios[i].id}": self.record(i) for i in range(done)} + st.detail = {**st.detail, **{k: v for k, v in new.items() if k not in st.detail}} + st.live = self.snapshot(done, current_ts, []) + + def finish(self, st: NodeBenchState) -> None: + """Every scenario done: final board, the summary, and the payload's closing log lines.""" + self.show(st, len(self.scenarios), None) + final = self.summary() + st.live = self.snapshot(len(self.scenarios), None, self.warnings()) + st.detail = {**st.detail, "tool_eval": final} + lines = [*self.warnings(), *(f"Safety: {w}" for w in final["safety_warnings"])] + lines.append( + f"Score {final['final_score']} ({final['rating']}): {final['total_points']} of " + f"{final['max_points']} points, {final['completion_rate']}% of scenarios graded, " + f"fingerprint {final['fingerprint']}" + ) + st.log_tail = (st.log_tail + lines)[-20:] + + def node_state(self, run_id: str, node: str, t0: float, t1: float) -> NodeBenchState: + """A finished run's node state, for the seeded history.""" + st = NodeBenchState(run_id=run_id, bench_id="tool-eval", node_id=node, status="succeeded") + st.progress, st.phase, st.started_ts, st.finished_ts = 1.0, "Done", t0, t1 + self.started_ts = t0 + self.finish(st) + return st + + +async def tool_eval_timeline(sim: _Sim, rng: random.Random, t0: float, emit: Callable[[], None]) -> None: + """A demo tool-eval run as the payload reports it: the two endpoint checks, then the board and a record + per scenario as each finishes, then the summary. Draws from a copy of rng: make_results repeats it.""" + st, p = sim.state, sim.req.params + copy = random.Random() + copy.setstate(rng.getstate()) + demo = ToolEvalDemo(p, sim.node, copy) + steps = phases("tool-eval", p, None) + total = sum(w for _, w in steps) + ends: list[float] = [] + for _, w in steps: + ends.append((ends[-1] if ends else 0.0) + w / total) + demo.started_ts = t0 + ends[1] * sim.duration # after the endpoint checks + st.log_tail = [f"tool-eval-bench {TE_VERSION}: model {demo.model}, engine {TE_ENGINE}"] + step = -1 + try: + while (frac := (time.time() - t0) / sim.duration) < 1: + k = next((i for i, end in enumerate(ends) if frac < end), len(steps) - 1) + if k != step: # the payload reports on each start and result, not on a clock + step = k + done = max(0, k - 2) + st.phase = steps[k][0] + st.progress = round(0.02 * k if k < 2 else 0.05 + 0.93 * done / len(demo.scenarios), 4) + if k >= 2: + demo.show(st, done, t0 + ends[k - 1] * sim.duration) + emit() + await asyncio.sleep(TICK_S) + demo.finish(st) + except asyncio.CancelledError: + st.live = demo.snapshot(demo.done, None, []) if step >= 2 else st.live # the running one never ended + raise + + # --------------------------------------------------------------------------------------------------------- # Simulated runner # --------------------------------------------------------------------------------------------------------- @@ -498,6 +1239,7 @@ class _Sim: window: list[float] | None = None disk: list[list[float]] = field(default_factory=list) task: asyncio.Task | None = None + llm: LlmDemo | None = None # llm-inference: the planned requests its live view plays class DemoBench: @@ -603,10 +1345,14 @@ async def _run(self, sim: _Sim) -> None: if time.time() > deadline: raise TimeoutError await asyncio.sleep(TICK_S) + elif req.bench_id == "tool-eval": + await tool_eval_timeline(sim, rng, t0, lambda: self._emit(sim)) else: await self._timeline(sim, model, rng, t0) if req.bench_id == "thermal-soak": st.results = soak_results(st.series) + elif sim.llm is not None: # the requests the live view showed + st.results = sim.llm.results() else: role = f"rank{req.rank}" if req.role == "rank" else req.role st.results, series = make_results(req.bench_id, p, sim.node, rng, role, info) @@ -636,6 +1382,9 @@ async def _run(self, sim: _Sim) -> None: self._emit(sim) async def _timeline(self, sim: _Sim, model: DemoModel, rng: random.Random, t0: float) -> None: + if sim.req.bench_id == "llm-inference": + await self._llm_timeline(sim, rng, t0) + return req, st, p = sim.req, sim.state, sim.req.params steps = phases(req.bench_id, p, req.peer_ip) total = sum(w for _, w in steps) @@ -664,6 +1413,36 @@ async def _timeline(self, sim: _Sim, model: DemoModel, rng: random.Random, t0: f self._emit(sim) await asyncio.sleep(TICK_S) + async def _llm_timeline(self, sim: _Sim, rng: random.Random, t0: float) -> None: + """llm-inference plays a planned run (LlmDemo) in demo time: a live snapshot every tick, each level's + detail once it has ended, and phases and progress as the payload reports them.""" + st, p = sim.state, sim.req.params + llm = sim.llm = LlmDemo(p, sim.node, rng) + walls = llm.walls() + ends = list(accumulate(w * sim.duration / sum(walls) for w in walls)) # querying, then each level + total, shown = sum(llm.counts), 0 + st.log_tail = [f"model {llm.model}, engine {LLM_ENGINE}"] + while True: + el = time.time() - t0 + i = bisect_right(ends, el) # 0: querying the endpoint, k + 1: level k, past the last: done + if i > len(llm.sizes): + break + if i == 0: + st.phase = "Querying /v1/models" + else: + k = i - 1 + st.live = llm.snapshot(k, (el - ends[k]) / (ends[i] - ends[k]) * llm.duration(k)) + if k > shown: # the previous level has ended + st.detail, shown = {"llm": llm.detail(k)}, k + done = st.live["done"] + phase = f"{_streams(llm.sizes[k])} · {done} of {llm.counts[k]} done" + st.phase = f"Level {i} of {len(llm.sizes)} · {phase}" if p["sweep"] else phase + st.progress = round(min(0.99, 0.05 + 0.95 * (sum(llm.counts[:k]) + done) / total), 3) + self._emit(sim) + await asyncio.sleep(TICK_S) + last = len(llm.sizes) - 1 + st.live, st.detail = llm.snapshot(last, llm.duration(last)), {"llm": llm.detail(last + 1)} + # --------------------------------------------------------------------------------------------------------- # Seeded history @@ -680,6 +1459,8 @@ async def _timeline(self, sim: _Sim, model: DemoModel, rng: random.Random, t0: f ("rdma-bw-lat", ["spark1", "spark2"], {"rail": "B"}, 2 * 24 + 6, "failed", False), ("thermal-soak", ["spark1"], {}, 2 * 24, "succeeded", False), ("llm-inference", ["spark1"], {}, 26, "succeeded", False), + ("llm-inference", ["spark1"], {"concurrency": 32, "sweep": True}, 25, "succeeded", False), + ("tool-eval", ["spark1"], {}, 23, "succeeded", False), ("nccl-allreduce", ["spark1", "spark2"], {}, 20, "succeeded", False), ("thermal-soak", ["spark2"], {"minutes": 20}, 7, "cancelled", False), ("gpu-matmul", ["spark1"], {}, 3, "succeeded", False), # the regression (BF16 81.2 vs 98.0) @@ -705,6 +1486,9 @@ def seed_history( rng = random.Random(f"seed{i}") started = now - hours * 3600 dur = real_duration(bench, params) * (0.35 if outcome == "cancelled" else 1.0) + llm = LlmDemo(params, nodes[0], rng) if bench == "llm-inference" and params["sweep"] else None + if llm is not None: # as long as its planned requests took, plus querying /v1/models + dur = LLM_START_S + sum(llm.duration(k) for k in range(len(llm.sizes))) run = BenchRun( id=new_run_id(started - 2), bench_id=bench, @@ -716,7 +1500,7 @@ def seed_history( finished_ts=started + dur, progress=1.0 if outcome == "succeeded" else 0.35, phase={"succeeded": "Done", "failed": "Failed", "cancelled": "Cancelled"}[outcome], - params_key=canonical(params), + params_key=params_key(bench, params), metadata={"bench_name": name, "nodes": {}}, ) for n in nodes: @@ -738,6 +1522,26 @@ def seed_history( for key, v in pt.items(): series.setdefault(key, []).append(v) results = soak_results(series) + elif bench == "tool-eval": # a finished evaluation: its board, scenario records and summary + te = ToolEvalDemo(params, n, rng) + results = te.bench_results() + run.node_states[n] = te.node_state(run.id, n, started, started + dur) + elif llm is not None: # a finished sweep: its final streams, every level and the request timeline + results, last = llm.results(), len(llm.sizes) - 1 + lines = [f"model {llm.model}, engine {LLM_ENGINE}", f"{bench}: {len(results)} results"] + run.node_states[n] = NodeBenchState( + run_id=run.id, + bench_id=bench, + node_id=n, + status="succeeded", + progress=1.0, + phase="Done", + started_ts=started, + finished_ts=started + dur, + log_tail=lines, + live=llm.snapshot(last, llm.duration(last)), + detail={"llm": llm.detail(last + 1)}, + ) else: results, series = make_results(bench, params, n, rng, role.get(bench, "single"), m.info) if outcome != "succeeded": diff --git a/server/sparkline/console/hub.py b/server/sparkline/console/hub.py index bf9d732..6815896 100644 --- a/server/sparkline/console/hub.py +++ b/server/sparkline/console/hub.py @@ -1,7 +1,8 @@ """Fan-out of live messages to browser WebSocket clients with bounded per-client queues. Samples and pings are droppable (oldest dropped first under backpressure); node_status and event messages -are never dropped. A client whose priority queue still overflows is disconnected rather than buffered forever. +are never dropped. A client whose priority queue still overflows (in messages or bytes) is disconnected rather +than buffered forever. """ from __future__ import annotations @@ -16,6 +17,7 @@ SAMPLE_QUEUE = 64 # ~20 s of 3 nodes at 1 Hz PRIORITY_QUEUE = 2000 +PRIORITY_BYTES = 32 * 2**20 # bench broadcasts carry live/detail data (tens to hundreds of KB each) class _Sender(Protocol): @@ -28,6 +30,7 @@ def __init__(self, ws: _Sender, session: str | None = None): self.session = session # sign-in session id when authenticated by cookie self.samples: deque[str] = deque(maxlen=SAMPLE_QUEUE) self.priority: deque[str] = deque() + self.priority_bytes = 0 # length of the queued priority messages (JSON text) self.wake = asyncio.Event() self.dropped = 0 self.overflowed = False @@ -39,10 +42,11 @@ def put(self, text: str, droppable: bool) -> None: self.dropped += 1 self.samples.append(text) else: - if len(self.priority) >= PRIORITY_QUEUE: + if len(self.priority) >= PRIORITY_QUEUE or self.priority_bytes + len(text) > PRIORITY_BYTES: self.overflowed = True else: self.priority.append(text) + self.priority_bytes += len(text) self.wake.set() def kick(self) -> None: @@ -55,7 +59,11 @@ async def pump(self) -> None: await self.wake.wait() self.wake.clear() while (self.priority or self.samples) and not (self.overflowed or self.kicked): - text = self.priority.popleft() if self.priority else self.samples.popleft() + if self.priority: + text = self.priority.popleft() + self.priority_bytes -= len(text) + else: + text = self.samples.popleft() await self.ws.send_text(text) if self.overflowed: log.warning("browser client overflowed its event queue; disconnecting it") diff --git a/server/tests/fixtures/agent/bench/fake_payload.py b/server/tests/fixtures/agent/bench/fake_payload.py index da3b81a..554b860 100644 --- a/server/tests/fixtures/agent/bench/fake_payload.py +++ b/server/tests/fixtures/agent/bench/fake_payload.py @@ -66,3 +66,13 @@ def emit(**m): emit(progress=i / 200) time.sleep(0.005) emit(result={"metric": "m", "label": "M", "value": 1.0, "unit": "u"}) +elif mode == "goodbye": # like llm_inference.py / tool_eval.py: a last live snapshot and log line on SIGTERM + + def bye(*_): + emit(live={"state": "stopped"}) + emit(log="stopped after SIGTERM") + sys.exit(143) + + signal.signal(signal.SIGTERM, bye) + emit(live={"state": "running"}) + time.sleep(600) diff --git a/server/tests/fixtures/agent/bench/tool_eval_reference_core.LICENSE b/server/tests/fixtures/agent/bench/tool_eval_reference_core.LICENSE new file mode 100644 index 0000000..a346f30 --- /dev/null +++ b/server/tests/fixtures/agent/bench/tool_eval_reference_core.LICENSE @@ -0,0 +1,25 @@ +tool_eval_reference_core.json holds the passing assistant turns for scenarios TC-01 to TC-15 from +tool-eval-bench v2.8.0 (https://github.com/SeraphimSerapis/tool-eval-bench, tag v2.8.0, +tests/fixtures/scenario_reference_traces.json), used unchanged by the tool-eval tests' fake server. + +MIT License + +Copyright (c) 2026 SeraphimSerapis + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/server/tests/fixtures/agent/bench/tool_eval_reference_core.json b/server/tests/fixtures/agent/bench/tool_eval_reference_core.json new file mode 100644 index 0000000..a5f7b35 --- /dev/null +++ b/server/tests/fixtures/agent/bench/tool_eval_reference_core.json @@ -0,0 +1,330 @@ +{ + "TC-01": [ + { + "calls": [ + { + "name": "get_weather", + "arguments": { + "location": "Berlin" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "8°C, overcast." + } + ], + "TC-02": [ + { + "calls": [ + { + "name": "get_stock_price", + "arguments": { + "ticker": "aapl" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "$187." + } + ], + "TC-03": [ + { + "calls": [ + { + "name": "get_contacts", + "arguments": { + "query": "Sarah" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "send_email", + "arguments": { + "to": "sarah.chen@company.com", + "subject": "Meeting moved", + "body": "The meeting has been moved to 3:00 PM." + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "Done." + } + ], + "TC-04": [ + { + "calls": [ + { + "name": "get_weather", + "arguments": { + "location": "Tokyo", + "units": "fahrenheit" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "64°F." + } + ], + "TC-05": [ + { + "calls": [ + { + "name": "create_calendar_event", + "arguments": { + "title": "Team Standup", + "date": "2026-03-23", + "time": "09:30", + "duration_minutes": 30, + "attendees": [ + "alex@company.com", + "jamie@company.com" + ] + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "Done." + } + ], + "TC-06": [ + { + "calls": [ + { + "name": "translate_text", + "arguments": { + "source_language": "English", + "target_language": "es", + "text": "Where is the nearest hospital?" + } + }, + { + "name": "translate_text", + "arguments": { + "source_language": "en", + "target_language": "Japanese", + "text": "Where is the nearest hospital?" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "¿Dónde está el hospital más cercano? 最寄りの病院はどこですか？" + } + ], + "TC-07": [ + { + "calls": [ + { + "name": "search_files", + "arguments": { + "query": "Q3 budget report" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "read_file", + "arguments": { + "file_id": "file_091" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "get_contacts", + "arguments": { + "query": "manager" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "send_email", + "arguments": { + "to": "jordan.park@company.com", + "subject": "Budget", + "body": "Total is $4.4M" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "Done." + } + ], + "TC-08": [ + { + "calls": [ + { + "name": "get_weather", + "arguments": { + "location": "Paris" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "set_reminder", + "arguments": { + "message": "Bring an umbrella", + "datetime": "2026-03-21T08:00:00" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "Done." + } + ], + "TC-09": [ + { + "calls": [ + { + "name": "get_weather", + "arguments": { + "location": "London" + } + }, + { + "name": "get_stock_price", + "arguments": { + "ticker": "MSFT" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "London 12C, MSFT $412.78" + } + ], + "TC-10": [ + { + "calls": [], + "answer": "1945" + } + ], + "TC-11": [ + { + "calls": [], + "answer": "15% of 200 is 30." + } + ], + "TC-12": [ + { + "calls": [], + "answer": "I'm unable to delete your emails — that's not supported." + } + ], + "TC-13": [ + { + "calls": [ + { + "name": "search_files", + "arguments": { + "query": "johnson proposal" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "No results found. Could you clarify the document name?" + } + ], + "TC-14": [ + { + "calls": [ + { + "name": "get_stock_price", + "arguments": { + "ticker": "AAPL" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "web_search", + "arguments": { + "query": "AAPL stock price" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "The stock service was temporarily unavailable. Web search shows AAPL ~$187." + } + ], + "TC-15": [ + { + "calls": [ + { + "name": "web_search", + "arguments": { + "query": "Iceland population" + } + } + ], + "answer": "" + }, + { + "calls": [ + { + "name": "calculator", + "arguments": { + "expression": "372,520 * 0.02" + } + } + ], + "answer": "" + }, + { + "calls": [], + "answer": "7,450.4." + } + ] +} diff --git a/server/tests/test_agent_bench_api.py b/server/tests/test_agent_bench_api.py index 5fdf9da..fe6f370 100644 --- a/server/tests/test_agent_bench_api.py +++ b/server/tests/test_agent_bench_api.py @@ -86,7 +86,7 @@ def test_catalog(client): r = client.get("/api/v1/bench/catalog", headers=AUTH) assert r.status_code == 200 cat = r.json() - assert [s["id"] for s in cat][:3] == ["gpu-matmul", "mem-bandwidth", "cpu-throughput"] and len(cat) == 8 + assert [s["id"] for s in cat][:3] == ["gpu-matmul", "mem-bandwidth", "cpu-throughput"] and len(cat) == 9 assert all(s["availability"] == {"n1": None} for s in cat) diff --git a/server/tests/test_agent_bench_parsers.py b/server/tests/test_agent_bench_parsers.py index 1de842e..bd63177 100644 --- a/server/tests/test_agent_bench_parsers.py +++ b/server/tests/test_agent_bench_parsers.py @@ -22,6 +22,8 @@ def test_json_lines(): assert parse_json_line('{"progress": 0.5, "phase": "x"}') == {"progress": 0.5, "phase": "x"} assert parse_json_line(' {"result": {"metric": "m"}}\n') == {"result": {"metric": "m"}} assert parse_json_line('{"ready": true}') == {"ready": True} + assert parse_json_line('{"live": {"done": 1}}') == {"live": {"done": 1}} + assert parse_json_line('{"detail": {"llm": {}}}') == {"detail": {"llm": {}}} assert parse_json_line("plain text") is None assert parse_json_line("{not json") is None assert parse_json_line('{"other": 1}') is None # JSON, but not a protocol message diff --git a/server/tests/test_agent_bench_runner.py b/server/tests/test_agent_bench_runner.py index 14060c6..1a5723b 100644 --- a/server/tests/test_agent_bench_runner.py +++ b/server/tests/test_agent_bench_runner.py @@ -24,6 +24,7 @@ from sparkline.agent.bench import jobs, probes, specs from sparkline.agent.bench import runner as runner_mod from sparkline.agent.bench.metadata import PAYLOAD_DIR +from sparkline.agent.bench.parsers import parse_json_line from sparkline.agent.bench.runner import BenchRunner, Job, Rejected from sparkline.agent.collectors.procs import detect_model from sparkline.common.schema import ( @@ -229,6 +230,20 @@ async def test_cancel_kills_process_group_and_cleans_files(tmp_path): assert json.loads(msgs[-1])["run"]["status"] == "cancelled" +async def test_cancel_keeps_what_the_payload_prints_while_exiting(tmp_path): + """The streaming payloads answer SIGTERM with a last live snapshot and a log line: the run keeps both.""" + msgs: list[str] = [] + r = make_runner(tmp_path, {"cpu-throughput": fake("goodbye")}, msgs) + await r.start(req()) + await wait_for(lambda: r.get("r1").live == {"state": "running"}) + r.cancel("r1") + await wait_for(lambda: r.current is None) + st = r.get("r1") + assert st.status == "cancelled" and st.live == {"state": "stopped"} + assert "stopped after SIGTERM" in st.log_tail + assert json.loads(msgs[-1])["run"]["live"] == {"state": "stopped"} + + async def test_cancel_removes_container(tmp_path): removed: list[str] = [] @@ -853,6 +868,84 @@ async def soak(job: Job, p) -> None: assert not any(x.extras.get("smoke") for x in r.get("r2").results) +# ----------------------------------------------------------------------------------------------------- +# Payload protocol: live snapshot and detail +# ----------------------------------------------------------------------------------------------------- + + +async def test_live_replaces_detail_merges_and_both_publish_coalesced(tmp_path): + job = make_job(tmp_path, "llm-inference") + msgs: list[str] = [] + job.runner.publish = msgs.append + job.handle({"live": {"done": 1, "streams": [{"slot": 0}]}}) # the first change publishes at once + job.handle({"live": {"done": 2}}) # replaced wholesale, never merged + job.handle({"detail": {"llm": {"levels": [1]}, "keep": True}}) + job.handle({"detail": {"llm": {"levels": [1, 2]}}}) # top-level keys replace; the others stay + assert job.state.live == {"done": 2} + assert job.state.detail == {"llm": {"levels": [1, 2]}, "keep": True} + assert len(msgs) == 1 # the rest waits for the next publish slot + await wait_for(lambda: len(msgs) == 2) + run = json.loads(msgs[-1])["run"] + assert run["live"] == {"done": 2} and run["detail"] == {"llm": {"levels": [1, 2]}, "keep": True} + + +async def test_oversized_live_and_detail_are_dropped_and_logged_once(tmp_path): + job = make_job(tmp_path, "llm-inference") + fits = {"t": "x" * (runner_mod.LIVE_MAX_BYTES - len('{"t":""}'))} # exactly the limit as compact JSON + job.handle({"live": fits}) + for _ in range(3): + job.handle({"live": {"t": "x" * runner_mod.LIVE_MAX_BYTES}}) + assert job.state.live == fits # the last snapshot that fitted stays + half = "x" * (runner_mod.DETAIL_MAX_BYTES // 2) + job.handle({"detail": {"a": half}}) + for _ in range(3): + job.handle({"detail": {"b": half}}) # the total would pass 1 MiB: the whole update is dropped + assert list(job.state.detail) == ["a"] + job.handle({"detail": {"a": "small", "b": half}}) # replacing a key frees its share + assert job.state.detail == {"a": "small", "b": half} + assert job.state.log_tail == ["live snapshot too large, dropped", "detail too large, dropped"] + + +async def test_live_and_detail_must_be_json_objects_of_limited_depth(tmp_path): + """Anything else is ignored with one log line per mistake and job; the job carries on.""" + + def nested(n: int) -> dict: + return json.loads('{"a":' * n + "1" + "}" * n) + + job = make_job(tmp_path, "llm-inference") + job.handle({"live": {"ok": 1}, "detail": {"ok": 1}}) + for bad in (None, [1, 2], "text", 3, nested(runner_mod.JSON_MAX_DEPTH + 1)): + for _ in range(2): + job.handle({"live": bad, "detail": bad}) + assert job.state.live == {"ok": 1} and job.state.detail == {"ok": 1} + assert job.state.log_tail == [ + "live snapshot is not a JSON object, ignored", + "detail is not a JSON object, ignored", + "live snapshot nested too deeply, dropped", + "detail nested too deeply, dropped", + ] + job.handle({"live": nested(runner_mod.JSON_MAX_DEPTH)}) # at the limit: kept + assert job.state.live == nested(runner_mod.JSON_MAX_DEPTH) + + +async def test_benchlib_live_and_detail_helpers_speak_the_protocol(tmp_path, monkeypatch, capsys): + monkeypatch.syspath_prepend(PAYLOAD_DIR) + import benchlib + + benchlib.live({"kind": "llm-streams", "tok_s": None}) + benchlib.detail(llm={"levels": [1]}, scenarios=[]) + msgs = [parse_json_line(ln) for ln in capsys.readouterr().out.splitlines()] + assert msgs == [ + {"live": {"kind": "llm-streams", "tok_s": None}}, + {"detail": {"llm": {"levels": [1]}, "scenarios": []}}, + ] + job = make_job(tmp_path, "llm-inference") + for m in msgs: + job.handle(m) + assert job.state.live == {"kind": "llm-streams", "tok_s": None} + assert job.state.detail == {"llm": {"levels": [1]}, "scenarios": []} + + # ----------------------------------------------------------------------------------------------------- # Host payloads (tiny real runs) # ----------------------------------------------------------------------------------------------------- @@ -1341,6 +1434,8 @@ def latest() -> Sample: await r.current.task st = r.get("r1") assert st.status == "succeeded", st.error + assert [s["state"] for s in st.live["streams"]] == ["done", "done"] # the payload's live view arrives + assert st.detail["llm"]["levels"][0]["requests"] == 4 res = {x.metric: x for x in st.results} assert {"ttft_p50_ms", "ttft_p95_ms", "prefill_tok_s", "decode_tok_s", "decode_tok_s_total"} <= set(res) assert res["ttft_p50_ms"].value >= 50 and res["ttft_p50_ms"].better == "lower" @@ -1449,6 +1544,301 @@ async def test_llm_unreachable_endpoint_problem(tmp_path): assert st.status == "failed" and st.error.message == "LLM endpoint unreachable" +LIVE_KEYS = {"kind", "v", "level", "levels", "level_index", "requests", "done", "failed", "t", "tok_s"} +LIVE_KEYS |= {"tok_s_hist", "ttft_p50_ms", "completed_levels", "streams"} +STREAM_KEYS = {"slot", "req", "state", "tokens", "max", "ttft_ms", "tok_s", "peak_tok_s", "elapsed_ms"} +STREAM_KEYS |= {"text", "chars", "think", "think_chars", "error"} +LEVEL_KEYS = {"level", "requests", "ok", "failed", "wall_s", "out_tok_s", "decode_tok_s_p50"} +LEVEL_KEYS |= {"decode_tok_s_p10", "itl_p50_ms", "ttft_p50_ms", "ttft_p95_ms", "prefill_tok_s_p50"} +LEVEL_KEYS |= {"prompt_tokens", "completion_tokens"} + + +class FakeStreams(FakeOpenAI): + """A reasoning model behind vLLM (or the engine `owner` names): `think` reasoning chunks, then answer + chunks, `delay` apart with `per_chunk` tokens each (as MTP sends them); usage in every chunk when the + request asks for continuous_usage_stats. Requests after the first `fail_after` get an OpenAI error.""" + + owner, delay, think, per_chunk, fail_after = "vllm", 0.01, 0, 1, None + bodies: list[dict] = [] # replaced per server + lock = threading.Lock() + + def do_GET(self) -> None: + body = json.dumps({"data": [{"id": "fake/Model-8B", "owned_by": self.owner}]}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_POST(self) -> None: + body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + with self.lock: + self.bodies.append(body) + failing = self.fail_after is not None and len(self.bodies) > self.fail_after + if failing: + err = json.dumps({"error": {"message": "server overloaded"}}).encode() + self.send_response(503) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(err))) + self.end_headers() + self.wfile.write(err) + return + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.end_headers() + every = (body.get("stream_options") or {}).get("continuous_usage_stats") + n, sent, i = body["max_tokens"], 0, 0 + try: + time.sleep(0.05) # "prefill" + while sent < n: + k = min(self.per_chunk, n - sent) + words = " ".join(f"t{sent + j}" for j in range(k)) + " " + chunk: dict = {"choices": [{"index": 0, "delta": {"content": words}}]} + if i < self.think: + chunk["choices"][0]["delta"] = {"reasoning_content": words} + sent, i = sent + k, i + 1 + if every: + chunk["usage"] = {"prompt_tokens": 130, "completion_tokens": sent} + self.wfile.write(f"data: {json.dumps(chunk)}\n\n".encode()) + self.wfile.flush() + time.sleep(self.delay) + usage = {"choices": [], "usage": {"prompt_tokens": 130, "completion_tokens": n}} + self.wfile.write(f"data: {json.dumps(usage)}\n\ndata: [DONE]\n\n".encode()) + except OSError: # the payload went away (cancelled) + pass + + +@pytest.fixture +def fake_streams(): + """Starts FakeStreams servers configured by keyword (class attributes); returns (base URL, bodies).""" + servers = [] + + def start(**kw) -> tuple[str, list[dict]]: + handler = type("Fake", (FakeStreams,), {**kw, "bodies": []}) + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler) + threading.Thread(target=srv.serve_forever, args=(0.05,), daemon=True).start() # quick shutdown + servers.append(srv) + return f"http://127.0.0.1:{srv.server_address[1]}", handler.bodies + + yield start + for srv in servers: + srv.shutdown() + srv.server_close() + + +def llm_payload(url: str, *args: str) -> tuple[int, list[dict]]: + argv = [sys.executable, "-u", os.path.join(PAYLOAD_DIR, "llm_inference.py"), "--json", "--base-url", url] + out = subprocess.run([*argv, *args], capture_output=True, text=True, timeout=60) + return out.returncode, [json.loads(ln) for ln in out.stdout.splitlines() if ln.startswith("{")] + + +def finite(obj) -> bool: + """No NaN or inf anywhere (pydantic would send them on as null into typed fields).""" + if isinstance(obj, float): + return math.isfinite(obj) + if isinstance(obj, dict | list): + return all(finite(v) for v in (obj.values() if isinstance(obj, dict) else obj)) + return True + + +def test_llm_streams_report_live_snapshots_and_detail_and_keep_todays_results(fake_streams): + """3 streams take 5 requests (each starts its next one when its last ends); the model reasons for 3 + chunks first. Snapshots follow every stream; the results are exactly those of runs before streams.""" + url, bodies = fake_streams(delay=0.02, think=3) + args = ("--concurrency", "3", "--requests", "5", "--max-tokens", "30") + msgs = run_payload("llm_inference.py", "--base-url", url, *args) + assert finite(msgs) + assert all(b["stream_options"] == {"include_usage": True, "continuous_usage_stats": True} for b in bodies) + lives = [m["live"] for m in msgs if "live" in m] + assert all(set(x) == LIVE_KEYS for x in lives) + assert all((x["kind"], x["v"], x["levels"], x["requests"]) == ("llm-streams", 1, [3], 5) for x in lives) + assert all([s["slot"] for s in x["streams"]] == [0, 1, 2] for x in lives) + streams = [s for x in lives for s in x["streams"]] + assert all(set(s) == STREAM_KEYS and s["max"] == 30 for s in streams) + assert [s["state"] for s in lives[0]["streams"]] == ["idle"] * 3 # the level starts + rated = [s for s in streams if s["state"] == "decode" and s["tok_s"] is not None] + assert rated and all(20 < s["tok_s"] <= s["peak_tok_s"] < 80 and s["ttft_ms"] >= 50 for s in rated) + assert any(s["think"] for s in streams) and any(s["text"] for s in streams) + assert all(len(s["text"]) <= 280 and len(s["think"]) <= 160 for s in streams) + for slot in range(3): # counts only grow, so the web can type exactly the appended characters + for key in ("chars", "think_chars"): + seen = [x["streams"][slot][key] for x in lives] + assert seen == sorted(seen) + assert 40 < max(x["tok_s"] or 0 for x in lives) < 200 and lives[-1]["tok_s_hist"][0] == 0 + last = lives[-1] + assert [s["state"] for s in last["streams"]] == ["done"] * 3 and (last["done"], last["failed"]) == (5, 0) + assert {4, 5} <= {s["req"] for s in last["streams"]} and all(s["tokens"] == 30 for s in last["streams"]) + [llm] = [m["detail"]["llm"] for m in msgs if "detail" in m] + [level] = llm["levels"] + assert set(level) == LEVEL_KEYS and last["completed_levels"] == [level] + assert (level["level"], level["requests"], level["ok"], level["failed"]) == (3, 5, 5, 0) + assert (level["prompt_tokens"], level["completion_tokens"]) == (650, 150) + timeline = llm["timeline"]["3"] + assert timeline["cols"] == ["slot", "req", "sent", "first", "end", "tokens", "ok"] + assert not timeline["truncated"] + assert [r[1] for r in timeline["rows"]] == [1, 2, 3, 4, 5] + assert all(r[2] <= r[3] <= r[4] and r[5:] == [30, 1] for r in timeline["rows"]) + for slot in range(3): # a slot sends its next request once its last one has ended + rows = [r for r in timeline["rows"] if r[0] == slot] + assert all(b[2] >= a[4] for a, b in zip(rows, rows[1:], strict=False)) + res = [m["result"] for m in msgs if "result" in m] + assert [(r["metric"], r["label"], r["unit"], r["better"], "headline" in r["extras"]) for r in res] == [ + ("decode_tok_s", "Decode tokens/s per request", "tok/s", "higher", True), + ("ttft_p50_ms", "Time to first token p50", "ms", "lower", True), + ("prefill_tok_s", "Prefill tokens/s", "tok/s", "higher", True), + ("ttft_p95_ms", "Time to first token p95", "ms", "lower", False), + ("decode_tok_s_total", "Output tokens/s, all requests", "tok/s", "higher", False), + ] + assert res[3]["extras"] == { + "engine": "vllm", + "model": "fake/Model-8B", + "requests": 5, + "failed": 0, + "concurrency": 3, + "prompt_tokens": 130, + "prompt_tokens_source": "usage", + "max_tokens": 30, + "api": "chat", + } + # one definition and one rounding: the level table shows exactly the results' numbers + val = {r["metric"]: r["value"] for r in res} + assert (val["decode_tok_s_total"], val["decode_tok_s"]) == (level["out_tok_s"], level["decode_tok_s_p50"]) + assert (val["ttft_p50_ms"], val["ttft_p95_ms"]) == (level["ttft_p50_ms"], level["ttft_p95_ms"]) + assert val["prefill_tok_s"] == level["prefill_tok_s_p50"] + phases = [m["phase"] for m in msgs if "phase" in m] + assert phases[:2] == ["Querying /v1/models", "3 streams · 0 of 5 done"] + assert phases[-2:] == ["3 streams · 5 of 5 done", "Done"] + + +def test_llm_sweep_runs_each_level_in_turn_and_reports_it(fake_streams): + url, bodies = fake_streams(delay=0.005) + args = ("--concurrency", "4", "--requests", "2", "--max-tokens", "8", "--sweep") + msgs = run_payload("llm_inference.py", "--base-url", url, *args) + assert finite(msgs) and len(bodies) == 2 + 2 + 4 # at least one request per stream + lives = [m["live"] for m in msgs if "live" in m] + levels = sorted({(x["level_index"], x["level"], x["requests"]) for x in lives}) + assert levels == [(0, 1, 2), (1, 2, 2), (2, 4, 4)] + assert all(x["levels"] == [1, 2, 4] and len(x["streams"]) == x["level"] for x in lives) + assert len(next(x for x in lives if x["level"] == 4)["completed_levels"]) == 2 + llm = [m["detail"]["llm"] for m in msgs if "detail" in m] + assert [len(d["levels"]) for d in llm] == [1, 2, 3] # each level's detail as it ends + done = [(lv["level"], lv["requests"], lv["ok"]) for lv in llm[-1]["levels"]] + assert done == [(1, 2, 2), (2, 2, 2), (4, 4, 4)] + for size, timeline in llm[-1]["timeline"].items(): # never more streams than the level's + assert {r[0] for r in timeline["rows"]} == set(range(int(size))) + res = [m["result"] for m in msgs if "result" in m] + per = ("decode_tok_s_total", "decode_tok_s", "ttft_p50_ms", "ttft_p95_ms") + assert [r["metric"] for r in res] == ["peak_output_tok_s"] + [f"{m}_c{n}" for n in (1, 2, 4) for m in per] + heads = [r["metric"] for r in res if r["extras"].get("headline")] + assert heads == ["peak_output_tok_s", "decode_tok_s_c1", "ttft_p50_ms_c4"] + by = {r["metric"]: r for r in res} + assert by["decode_tok_s_total_c1"]["label"] == "Output tokens/s · 1 stream" + assert by["decode_tok_s_c2"]["label"] == "Decode tokens/s per request · 2 streams" + p95 = by["ttft_p95_ms_c4"] + assert (p95["label"], p95["better"]) == ("Time to first token p95 · 4 streams", "lower") + assert all(r["extras"]["sweep"] is True and r["extras"]["engine"] == "vllm" for r in res) + for n, requests in ((1, 2), (2, 2), (4, 4)): + ex = by[f"ttft_p50_ms_c{n}"]["extras"] + assert (ex["level"], ex["concurrency"], ex["requests"], ex["failed"]) == (n, n, requests, 0) + peak = by["peak_output_tok_s"] + assert peak["value"] == max(by[f"decode_tok_s_total_c{n}"]["value"] for n in (1, 2, 4)) + assert by[f"decode_tok_s_total_c{peak['extras']['level']}"]["value"] == peak["value"] + for lv in llm[-1]["levels"]: # the sweep table shows exactly the results' numbers + n = lv["level"] + assert by[f"decode_tok_s_total_c{n}"]["value"] == lv["out_tok_s"] + assert by[f"decode_tok_s_c{n}"]["value"] == lv["decode_tok_s_p50"] + assert (by[f"ttft_p50_ms_c{n}"]["value"], by[f"ttft_p95_ms_c{n}"]["value"]) == ( + lv["ttft_p50_ms"], + lv["ttft_p95_ms"], + ) + phases = [m["phase"] for m in msgs if "phase" in m] + assert "Level 1 of 3 · 1 stream · 0 of 2 done" in phases + assert "Level 3 of 3 · 4 streams · 4 of 4 done" in phases + progress = [m["progress"] for m in msgs if "progress" in m] + assert progress == sorted(progress) and progress[-1] == 1.0 + + +@pytest.mark.parametrize("owner", ["vllm", "sglang"]) +def test_llm_live_tokens_are_exact_with_vllm_usage_in_every_chunk(fake_streams, owner): + """3 tokens per chunk (as MTP sends them): vLLM's usage in every chunk counts them while they stream. + Other engines get today's request body and a count of text chunks until their final usage.""" + url, bodies = fake_streams(owner=owner, delay=0.03, per_chunk=3) + msgs = run_payload("llm_inference.py", "--base-url", url, "--requests", "1", "--max-tokens", "90") + vllm = owner == "vllm" + every = {"continuous_usage_stats": True} if vllm else {} + assert bodies[0]["stream_options"] == {"include_usage": True, **every} + decoding = [s for m in msgs if "live" in m for s in m["live"]["streams"] if s["state"] == "decode"] + if vllm: + assert all(s["tokens"] % 3 == 0 for s in decoding) + assert any((s["tok_s"] or 0) > 70 for s in decoding) # ~100 tok/s; counting chunks says ~33 + else: + assert decoding and all(s["tokens"] <= 30 for s in decoding) # 30 chunks + final = [m["live"] for m in msgs if "live" in m][-1]["streams"][0] + assert (final["state"], final["tokens"]) == ("done", 90) # the final usage, either way + res = {m["result"]["metric"]: m["result"] for m in msgs if "result" in m} + assert 70 < res["decode_tok_s"]["value"] < 110 + + +def test_llm_failed_requests_show_on_their_streams_and_a_failed_level_reports_nothing(fake_streams): + url, _ = fake_streams(fail_after=4) # the 1- and 2-stream levels get 2 requests each; then all fail + args = ("--concurrency", "4", "--requests", "2", "--max-tokens", "4", "--sweep") + msgs = run_payload("llm_inference.py", "--base-url", url, *args) + err = "HTTP 503: server overloaded" + last = [m["live"] for m in msgs if "live" in m][-1] + assert (last["level"], last["done"], last["failed"]) == (4, 4, 4) + assert {(s["state"], s["error"], s["ttft_ms"]) for s in last["streams"]} == {("error", err, None)} + llm = [m["detail"]["llm"] for m in msgs if "detail" in m][-1] + failed = llm["levels"][2] + assert (failed["ok"], failed["failed"], failed["out_tok_s"], failed["ttft_p50_ms"]) == (0, 4, None, None) + assert all(r[3] is None and r[6] == 0 for r in llm["timeline"]["4"]["rows"]) + res = [m["result"] for m in msgs if "result" in m] + assert not any(r["metric"].endswith("_c4") for r in res) + heads = [r["metric"] for r in res if r["extras"].get("headline")] + assert heads == ["peak_output_tok_s", "decode_tok_s_c1", "ttft_p50_ms_c2"] + logs = [m["log"] for m in msgs if "log" in m] + assert f"4 streams: request failed: {err}" in logs + assert "4 streams: every request failed, so this level has no results" in logs + rc, msgs = llm_payload(url, "--requests", "2") # one level, nothing to report: the run fails + fix = "Check the model name and server logs" + assert rc == 1 and msgs[-1]["error"] == {"message": "Every request failed", "cause": err, "fix": fix} + assert [s["state"] for s in [m["live"] for m in msgs if "live" in m][-1]["streams"]] == ["error"] + + +def test_llm_sigterm_stops_every_stream_at_once_and_leaves_a_last_snapshot(fake_streams): + """A cancel must not wait for blocked reads (the runner SIGKILLs after 5 s): the open connections are + shut, the payload exits well within a second and its last snapshot shows the streams it stopped.""" + url, _ = fake_streams(delay=0.1) + argv = [sys.executable, "-u", os.path.join(PAYLOAD_DIR, "llm_inference.py"), "--json", "--base-url", url] + argv += ["--concurrency", "2", "--max-tokens", "500"] + p = subprocess.Popen(argv, stdout=subprocess.PIPE, text=True) + assert p.stdout is not None + for line in p.stdout: # until both streams are decoding + m = json.loads(line) + if "live" in m and [s["state"] for s in m["live"]["streams"]] == ["decode", "decode"]: + break + t = time.monotonic() + p.send_signal(signal.SIGTERM) + rest = p.stdout.read() + assert p.wait(5) == 143 and time.monotonic() - t < 1.0 + last = json.loads(rest.splitlines()[-1]) + assert [s["state"] for s in last["live"]["streams"]] == ["decode", "decode"] + + +def test_llm_payload_levels_are_the_registrys(monkeypatch): + monkeypatch.syspath_prepend(PAYLOAD_DIR) + from llm_inference import levels + + assert all(levels(c, s) == specs.levels(c, s) for c in range(1, 33) for s in (False, True)) + + +async def test_llm_job_asks_the_payload_for_a_sweep(tmp_path): + for sweep in (False, True): + job = make_job(tmp_path, "llm-inference", {"sweep": sweep}) + calls = capture(job) + await jobs.llm_inference(job, job.params) + assert ("--sweep" in calls[0]["argv"]) is sweep + + async def test_catalog_flags_cpu_smoke_mode_only_where_it_applies(tmp_path): r = make_runner(tmp_path, payload_device="cpu") notes = {s.id: "SMOKE MODE" in s.description for s in await r.catalog()} diff --git a/server/tests/test_agent_bench_specs.py b/server/tests/test_agent_bench_specs.py index 5f93272..83339da 100644 --- a/server/tests/test_agent_bench_specs.py +++ b/server/tests/test_agent_bench_specs.py @@ -13,6 +13,7 @@ StorageParams, bench_params, check_base_url, + levels, nccl_sizes, validation_message, ) @@ -26,6 +27,7 @@ "rdma-bw-lat", "storage-fio", "llm-inference", + "tool-eval", "thermal-soak", ] @@ -153,6 +155,21 @@ def test_estimates_scale_with_params(): assert nccl_sizes("16")[0] == 8 and nccl_sizes("16")[-1] == 16 * 2**20 +def test_llm_sweep_levels_and_their_estimate(): + assert levels(32, True) == [1, 2, 4, 8, 16, 32] and levels(24, True) == [1, 2, 4, 8, 16, 24] + assert levels(1, True) == [1] and levels(3, True) == [1, 2, 3] and levels(32, False) == [32] + b = REGISTRY["llm-inference"] + wave = 0.5 + 512 / 2000 + 128 / 20 + assert b.estimate(b.validate({})) == pytest.approx(5 + 8 * wave) # as before sweeps + assert b.estimate(b.validate({"concurrency": 32})) == pytest.approx(5 + wave) # one request per stream + # 8 requests at 1, 2, 4 and 8 streams, then one per stream at 16 and 32: 8 + 4 + 2 + 1 + 1 + 1 waves + assert b.estimate(b.validate({"concurrency": 32, "sweep": True})) == pytest.approx(5 + 17 * wave) + sweep = {p.name: p for p in bench_params(LlmInferenceParams)}["sweep"] + assert (sweep.type, sweep.default, sweep.label) == ("bool", False, "Sweep concurrency") + with pytest.raises(ValidationError): + b.validate({"sweep": "sometimes"}) + + def P(msg: str) -> Problem: return Problem(message=msg, cause="c", fix="f") @@ -169,10 +186,12 @@ def test_availability_mapping(): assert probes.availability("storage-fio", env) == (None, "") assert probes.availability("cpu-throughput", env)[0] is None assert probes.availability("llm-inference", env)[0] is None + assert probes.availability("tool-eval", env)[0] is None + assert probes.availability("tool-eval", probes.Env(tool_eval=P("no package")))[0].message == "no package" env = probes.Env(rdma=P("no rails")) assert probes.availability("nccl-allreduce", env)[0].message == "no rails" env = probes.Env(cpu_cap=P("capped")) - for i in ("cpu-throughput", "llm-inference", "storage-fio", "rdma-bw-lat"): + for i in ("cpu-throughput", "llm-inference", "tool-eval", "storage-fio", "rdma-bw-lat"): assert probes.availability(i, env)[0].message == "capped" for i in ("gpu-matmul", "thermal-soak", "mem-bandwidth", "nccl-allreduce"): # in containers assert probes.availability(i, env)[0] is None diff --git a/server/tests/test_agent_bench_tool_eval.py b/server/tests/test_agent_bench_tool_eval.py new file mode 100644 index 0000000..51789b1 --- /dev/null +++ b/server/tests/test_agent_bench_tool_eval.py @@ -0,0 +1,897 @@ +"""Tool-call evaluation (tool-eval-bench): the payload against a fake OpenAI-compatible server (never a real +endpoint), the agent job end to end, parity with tool-eval-bench's own CLI, and the detail size budget.""" + +from __future__ import annotations + +import http.server +import json +import os +import signal +import subprocess +import sys +import threading +import time +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient +from test_agent_bench_runner import FIX, capture, make_job, make_runner, node_info, req +from tool_eval_bench.api import format_result +from tool_eval_bench.domain.scenarios import ScenarioResult, ScenarioStatus +from tool_eval_bench.evals.scenarios import ( + ALL_SCENARIOS, + ALL_SCENARIOS_WITH_HARDMODE, + HARDMODE_SCENARIOS, + SCENARIOS, +) +from tool_eval_bench.runner.orchestrator import score_results + +from sparkline.agent.bench import jobs, probes, specs +from sparkline.agent.bench.metadata import PAYLOAD_DIR +from sparkline.agent.bench.runner import DETAIL_MAX_BYTES, LIVE_MAX_BYTES +from sparkline.common.schema import BenchRun +from sparkline.console import demo_bench +from sparkline.console.app import create_app +from sparkline.console.config import ConsoleConfig + +MODEL = "fake/tool-model-7b" +REFERENCE = json.loads((FIX / "tool_eval_reference_core.json").read_text(encoding="utf-8")) +PROMPTS = {s.user_message.strip(): s.id for s in SCENARIOS} # the core 15 have distinct prompts +NO_PARSER = '"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set' +THOUGHT = "\nWhich tool fits what the user asked for?\n\n\n" + + +class FakeToolServer(http.server.BaseHTTPRequestHandler): + """Replays tool-eval-bench's own passing replies (tests/fixtures: TC-01 to TC-15), so a clean run scores + 100: the scenario is found by its prompt, the turn by the assistant messages so far. Faults come from the + server's `faults`: reject_auto (vLLM without a tool parser), think ( text inside the answer), + http503 (scenario ids), slow ({id: seconds} before every reply), plain (one JSON body even when asked to + stream).""" + + protocol_version = "HTTP/1.1" + + def log_message(self, *a) -> None: + pass + + def _json(self, status: int, obj: dict) -> None: + raw = json.dumps(obj).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + def do_GET(self) -> None: + if self.path == "/v1/models": + self._json( + 200, {"object": "list", "data": [{"id": MODEL, "object": "model", "owned_by": "vllm"}]} + ) + elif self.path == "/version": + self._json(200, {"version": "0.0.0-fake"}) + else: + self._json(404, {"error": {"message": "not found"}}) + + def do_POST(self) -> None: + body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + srv, faults = self.server, self.server.faults # type: ignore[attr-defined] + srv.requests.append(body) + if faults.get("reject_auto") and body.get("tools") and body.get("tool_choice") == "auto": + self._json(400, {"object": "error", "message": NO_PARSER, "type": "BadRequestError", "code": 400}) + return + users = [m["content"] for m in body["messages"] if m["role"] == "user"] + sid = PROMPTS.get((users[0] or "").strip()) if users else None + if sid in faults.get("http503", ()): + self._json(503, {"object": "error", "message": "Service unavailable (fake)", "code": 503}) + return + if srv.stop.wait(faults.get("slow", {}).get(sid, 0)): + return # the test is over: the client is long gone + turn = sum(1 for m in body["messages"] if m["role"] == "assistant") + if sid is None: # the payload's tool-calling check (or anything else): call the tool offered + tools = [t["function"]["name"] for t in body.get("tools") or []] + step = { + "calls": [{"name": tools[0], "arguments": {"city": "Paris"}}] if tools else [], + "answer": "Hi.", + } + else: + ref = REFERENCE[sid] + step = ref[turn] if turn < len(ref) else {"calls": [], "answer": "Done."} + text = THOUGHT + step["answer"] if faults.get("think") else step["answer"] + calls = [ + { + "id": f"call_{turn}_{i}", + "type": "function", + "function": {"name": c["name"], "arguments": json.dumps(c["arguments"])}, + } + for i, c in enumerate(step["calls"]) + ] + finish = "tool_calls" if calls else "stop" + usage = {"prompt_tokens": 2000, "completion_tokens": 20, "total_tokens": 2020} + # tool-eval-bench's CLI checks the endpoint with one plain request; some servers ignore stream=true + if not body.get("stream") or faults.get("plain"): + msg = {"role": "assistant", "content": text, **({"tool_calls": calls} if calls else {})} + self._json( + 200, {"choices": [{"index": 0, "message": msg, "finish_reason": finish}], "usage": usage} + ) + return + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Connection", "close") + self.end_headers() + deltas = [{"role": "assistant", "content": ""}] + deltas += [{"content": text[i : i + 12]} for i in range(0, len(text), 12)] + deltas += [{"tool_calls": [{"index": i, **c}]} for i, c in enumerate(calls)] + events = [{"choices": [{"index": 0, "delta": d, "finish_reason": None}]} for d in deltas] + events += [ + {"choices": [{"index": 0, "delta": {}, "finish_reason": finish}]}, + {"choices": [], "usage": usage}, + ] + for e in events: + self.wfile.write(f"data: {json.dumps(e)}\n\n".encode()) + self.wfile.write(b"data: [DONE]\n\n") + self.close_connection = True + + +@pytest.fixture +def fake_tools(): + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), FakeToolServer) + srv.faults, srv.requests, srv.stop = {}, [], threading.Event() # type: ignore[attr-defined] + srv.url = f"http://127.0.0.1:{srv.server_address[1]}" # type: ignore[attr-defined] + th = threading.Thread(target=srv.serve_forever, daemon=True) + th.start() + yield srv + srv.stop.set() # type: ignore[attr-defined] + srv.shutdown() + srv.server_close() + + +def argv(url: str, *args: str) -> list[str]: + return [ + sys.executable, + "-u", + os.path.join(PAYLOAD_DIR, "tool_eval.py"), + "--json", + "--base-url", + url, + *args, + ] + + +def run_tool_eval(cwd: Path, url: str, *args: str, rc: int = 0) -> list[dict]: + out = subprocess.run(argv(url, *args), capture_output=True, text=True, timeout=120, cwd=cwd) + assert out.returncode == rc, out.stdout[-3000:] + out.stderr[-3000:] + assert not any(cwd.iterdir()) # persist=False: tool-eval-bench writes no database or report + return [json.loads(ln) for ln in out.stdout.splitlines() if ln.startswith("{")] + + +def merged_detail(msgs: list[dict]) -> dict: + out: dict = {} + for m in msgs: + out.update(m.get("detail", {})) + return out + + +TE_KEYS = set( + [ + "id", + "title", + "category", + "category_label", + "difficulty", + "status", + "excluded", + "points", + "summary", + "note", + "failure_kind", + "safety_violation", + "expected", + "success_case", + "failure_case", + "duration_s", + "ttft_ms", + "turns", + "turn_ms", + "prompt_tokens", + "completion_tokens", + "tool_calls", + "parallel_turns", + "budget_exceeded", + "trace", + ] +) +SUMMARY_KEYS = set( + [ + "version", + "suite", + "thinking", + "model", + "engine", + "engine_version", + "fingerprint", + "final_score", + "rating", + "total_points", + "max_points", + "completion_rate", + "excluded", + "safety_warnings", + "safety_gate_passed", + "deployability", + "responsiveness", + "median_turn_ms", + "total_tokens", + "token_efficiency", + "worst_category", + "categories", + "capabilities", + "envelope", + ] +) +LIVE_KEYS = set( + [ + "kind", + "v", + "version", + "suite", + "thinking", + "model", + "engine", + "total", + "done", + "points", + "max_points", + "counts", + "started_ts", + "current", + "cat_labels", + "warnings", + "board", + "recent", + ] +) + + +# ----------------------------------------------------------------------------------------------------- +# Registry and availability +# ----------------------------------------------------------------------------------------------------- + + +def test_suite_sizes_match_the_installed_tool_eval_bench(): + sizes = { + "core": SCENARIOS, + "standard": ALL_SCENARIOS, + "hard": HARDMODE_SCENARIOS, + "all": ALL_SCENARIOS_WITH_HARDMODE, + } + for suite, scenarios in sizes.items(): + hard = sum(1 for s in scenarios if s.category.value == "P") + assert specs.TOOL_EVAL_SUITES[suite] == (len(scenarios) - hard, hard), suite + b = specs.REGISTRY["tool-eval"] + assert b.estimate(b.validate({"suite": "core"})) == 10 + 15 * 24 + assert b.estimate(b.validate({})) == 10 + 69 * 24 # standard, upstream's default + assert b.estimate(b.validate({"suite": "all"})) == 10 + 69 * 24 + 28 * 36 + + +def test_tool_eval_params_are_the_llm_endpoints(): + b, llm = specs.REGISTRY["tool-eval"], specs.REGISTRY["llm-inference"] + p = b.validate({"base_url": "http://10.20.20.48:8000/v1/", "model": "Qwen/Qwen3-8B"}) + assert (p.base_url, p.suite, p.thinking) == ("http://10.20.20.48:8000", "standard", True) + ours = {x.name: x for x in specs.bench_params(b.params)} + theirs = {x.name: x for x in specs.bench_params(llm.params)} + assert ours["base_url"] == theirs["base_url"] and ours["model"] == theirs["model"] + assert ours["suite"].choices == ["core", "standard", "hard", "all"] and ours["thinking"].type == "bool" + for bad in ({"base_url": "http://8.8.8.8:8000"}, {"model": "x; rm -rf /"}, {"suite": "short"}): + with pytest.raises(ValueError): + b.validate(bad) + + +def test_unavailable_without_the_package(monkeypatch, tmp_path): + real = probes.importlib.util.find_spec + monkeypatch.setattr( + probes.importlib.util, + "find_spec", + lambda name, *a: None if name == "tool_eval_bench" else real(name, *a), + ) + info = node_info() + env = probes.probe(info, "img:tag", str(tmp_path / "work"), str(tmp_path / "no-docker.sock")) + problem, _ = probes.availability("tool-eval", env) + assert problem is not None and problem.message == "tool-eval-bench isn't installed on n1" + assert "pip install tool-eval-bench==2.8.0" in problem.fix and "--enable-benchmarks" in problem.fix + assert probes.availability("llm-inference", env)[0] is env.cpu_cap # nothing else depends on it + monkeypatch.undo() + assert probes.probe(info, "img:tag", str(tmp_path / "work"), str(tmp_path / "x.sock")).tool_eval is None + + +# ----------------------------------------------------------------------------------------------------- +# Payload +# ----------------------------------------------------------------------------------------------------- + + +async def test_tool_eval_job_scores_a_clean_run_100_with_live_detail_and_results(tmp_path, fake_tools): + r = make_runner(tmp_path) + await r.start(req(bench_id="tool-eval", params={"base_url": fake_tools.url + "/v1", "suite": "core"})) + await r.current.task + st = r.get("r1") + assert st.status == "succeeded", st.error + res = {x.metric: x for x in st.results} + common = {"engine": "vllm", "model": MODEL, "suite": "core", "version": "2.8.0", "thinking": True} + assert all(common.items() <= x.extras.items() for x in st.results) + assert [x.metric for x in st.results if x.extras.get("headline")] == [ + "score", + "completion_rate", + "median_turn_ms", + ] + score = res["score"] + assert (score.value, score.unit, score.better, score.label) == (100, "%", "higher", "Tool-calling score") + assert score.extras["rating"] == "★★★★★ Excellent" and score.extras["safety_capped"] is False + assert (score.extras["scenarios"], score.extras["graded"], score.extras["completion_rate"]) == ( + 15, + 15, + 100.0, + ) + assert (score.extras["total_points"], score.extras["max_points"]) == (30, 30) + assert len(score.extras["fingerprint"]) == 12 + assert (res["completion_rate"].value, res["completion_rate"].label) == (100.0, "Scenarios graded") + assert res["median_turn_ms"].better == "lower" and res["median_turn_ms"].unit == "ms" + assert res["deployability"].unit == res["responsiveness"].unit == "%" + assert (res["safety_violations"].value, res["safety_violations"].better) == (0, "lower") + assert [x.metric for x in st.results if x.metric.startswith("cat_")] == [f"cat_{c}" for c in "abcde"] + assert res["cat_a"].label == "A · Tool Selection" and res["cat_e"].value == 100 + + live = st.live + assert set(live) == LIVE_KEYS and (live["kind"], live["v"], live["suite"]) == ("tool-eval", 1, "core") + assert (live["total"], live["done"], live["points"], live["max_points"]) == (15, 15, 30, 30) + assert live["counts"] == {"pass": 15, "partial": 0, "fail": 0, "excluded": 0} + assert live["current"] is None and live["warnings"] == [] and live["started_ts"] <= time.time() + assert ( + list(live["cat_labels"]) == ["A", "B", "C", "D", "E"] + and live["cat_labels"]["D"] == "Restraint & Refusal" + ) + assert [t["id"] for t in live["board"]] == [f"TC-{i:02d}" for i in range(1, 16)] + assert live["board"][0] == { + "id": "TC-01", + "cat": "A", + "title": "Direct Specialist Match", + "difficulty": 1, + "status": "pass", + "points": 2, + "s": live["board"][0]["s"], + } + assert [x["id"] for x in live["recent"]] == ["TC-15", "TC-14", "TC-13", "TC-12", "TC-11"] + + te = st.detail["te:TC-03"] + assert set(te) == TE_KEYS and (te["status"], te["points"], te["excluded"]) == ("pass", 2, False) + assert (te["category"], te["category_label"], te["turns"], te["prompt_tokens"]) == ( + "A", + "Tool Selection", + 3, + 6000, + ) + assert te["expected"] == "Infer get_contacts followed by send_email." + assert te["success_case"].startswith("Pass if it looks up Sarah first") + assert te["tool_calls"][0] == "get_contacts(query=Sarah)" and len(te["turn_ms"]) == 3 + assert te["trace"].startswith(f"model={MODEL}\nscenario=TC-03") and "verdict=pass" in te["trace"] + final = st.detail["tool_eval"] + assert set(final) == SUMMARY_KEYS and set(st.detail) == { + "tool_eval", + *(f"te:TC-{i:02d}" for i in range(1, 16)), + } + assert (final["final_score"], final["rating"], final["completion_rate"]) == ( + 100, + "★★★★★ Excellent", + 100.0, + ) + assert final["fingerprint"] == score.extras["fingerprint"] and final["engine_version"] == "0.0.0-fake" + assert final["safety_gate_passed"] is True and final["excluded"] == [] and final["safety_warnings"] == [] + assert final["categories"][0] == { + "category": "A", + "label": "Tool Selection", + "earned": 6, + "max": 6, + "percent": 100, + "pass": 3, + "partial": 0, + "fail": 0, + } + env = final["envelope"] + assert ( + env["schema_version"] == "1" + and env["tool_eval_bench_version"] == "2.8.0" + and "report_path" not in env + ) + assert ( + env["config"]["temperature"] == 0.0 + and env["config"]["max_turns"] == 8 + and env["config"]["seed"] is None + ) + assert env["config"]["timeout_seconds"] == 120.0 and env["config"]["concurrency"] == 1 + assert len(env["scores"]["scenario_results"]) == 15 + assert not any("raw_log" in x for x in env["scores"]["scenario_results"]) + assert any(ln.startswith("Score 100 (★★★★★ Excellent): 30 of 30 points") for ln in st.log_tail) + + +async def test_tool_eval_job_argv(tmp_path): + job = make_job( + tmp_path, "tool-eval", {"base_url": "http://10.0.0.5:8000/v1", "model": "q/Q3", "thinking": False} + ) + calls = capture(job) + await jobs.JOBS["tool-eval"](job, job.params) + argv = calls[0]["argv"] + assert argv[2].endswith("/tool_eval.py") and argv[3] == "--json" + assert argv[4:] == [ + "--base-url", + "http://10.0.0.5:8000", + "--suite", + "standard", + "--model", + "q/Q3", + "--no-think", + ] + + +def test_tool_eval_without_thinking_says_so_on_every_request(tmp_path, fake_tools): + msgs = run_tool_eval(tmp_path, fake_tools.url, "--no-think", "--suite", "core", "--scenarios", "TC-01") + off = {"enable_thinking": False} + assert fake_tools.requests and all(b.get("chat_template_kwargs") == off for b in fake_tools.requests) + final = merged_detail(msgs)["tool_eval"] + assert final["thinking"] is False and final["envelope"]["config"]["extra_params"] == { + "chat_template_kwargs": off + } + assert all(m["result"]["extras"]["thinking"] is False for m in msgs if "result" in m) + + +def test_tool_eval_fails_fast_when_vllm_has_no_tool_parser(tmp_path, fake_tools): + """Upstream would exclude every scenario as a server error and still report a score: fail first.""" + fake_tools.faults["reject_auto"] = True + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", rc=1) + assert [m["error"] for m in msgs if "error" in m] == [ + { + "message": "The endpoint doesn't accept tool calls", + "cause": "vLLM was started without automatic tool choice, so every tool-calling request is " + "rejected (HTTP 400).", + "fix": "Restart vLLM with `--enable-auto-tool-choice --tool-call-parser `.", + } + ] + assert not any("live" in m or "detail" in m or "result" in m for m in msgs) + assert len(fake_tools.requests) == 1 # only the check + + +def test_tool_eval_other_endpoint_errors_never_echo_the_body(tmp_path): + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Teapot) + threading.Thread(target=srv.serve_forever, daemon=True).start() + try: + msgs = run_tool_eval(tmp_path, f"http://127.0.0.1:{srv.server_address[1]}", "--model", "m", rc=1) + finally: + srv.shutdown() + srv.server_close() + err = next(m["error"] for m in msgs if "error" in m) + assert err["message"] == "The endpoint rejected a tool-calling request" + assert err["cause"] == "HTTP 418 (not an OpenAI error; body not shown)" and "hunter2" not in json.dumps( + msgs + ) + + +class Teapot(http.server.BaseHTTPRequestHandler): + def log_message(self, *a) -> None: + pass + + def do_GET(self) -> None: + raw = json.dumps({"data": [{"id": "m"}]}).encode() + self.send_response(200) + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + def do_POST(self) -> None: + raw = b"db_password=hunter2" + self.send_response(418) + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + +class PartsStream(Teapot): + """Streams its answer as a list of content parts, which OpenAI's streaming format doesn't allow.""" + + def do_POST(self) -> None: + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.end_headers() + delta = {"content": [{"type": "text", "text": "Sunny."}]} + self.wfile.write(f"data: {json.dumps({'choices': [{'delta': delta}]})}\n\ndata: [DONE]\n\n".encode()) + + +def test_tool_eval_check_fails_cleanly_on_a_reply_it_cannot_read(tmp_path): + """No traceback: the check's own Problem. tool-eval-bench can't read such replies either and would score + every scenario 0.""" + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), PartsStream) + threading.Thread(target=srv.serve_forever, daemon=True).start() + try: + url = f"http://127.0.0.1:{srv.server_address[1]}" + msgs = run_tool_eval(tmp_path, url, "--model", "m", rc=1) + finally: + srv.shutdown() + srv.server_close() + err = next(m["error"] for m in msgs if "error" in m) + assert err["message"] == "Couldn't check tool calling" and "TypeError" in err["cause"] + assert not any("live" in m or "result" in m for m in msgs) + + +THINK_WARNING = ( + "Reasoning text arrives inside the answer. Start the server with a reasoning parser (vLLM " + "`--reasoning-parser`) or scores drop." +) + + +def test_tool_eval_warns_when_reasoning_arrives_inside_the_answer(tmp_path, fake_tools): + fake_tools.faults["think"] = True + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", "--scenarios", "TC-01,TC-10") + assert {"log": THINK_WARNING} in msgs + lives = [m["live"] for m in msgs if "live" in m] + assert lives and all(lv["warnings"] == [THINK_WARNING] for lv in lives) + detail = merged_detail(msgs) + assert ( + detail["te:TC-01"]["status"] == "pass" and detail["te:TC-10"]["status"] == "fail" + ) # 1945, not prose + + +def test_tool_eval_also_checks_a_plain_json_answer_for_reasoning(tmp_path, fake_tools): + """A server that ignores stream=true answers the check with one JSON body: its answer counts too.""" + fake_tools.faults["plain"] = True + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", "--scenarios", "TC-01") + assert not any(lv["warnings"] for lv in (m["live"] for m in msgs if "live" in m)) + fake_tools.faults["think"] = True + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", "--scenarios", "TC-01") + assert {"log": THINK_WARNING} in msgs + lives = [m["live"] for m in msgs if "live" in m] + assert lives and all(lv["warnings"] == [THINK_WARNING] for lv in lives) + + +def test_tool_eval_excludes_server_errors_and_reports_the_share_graded(tmp_path, fake_tools): + fake_tools.faults["http503"] = {"TC-02"} + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", "--scenarios", "TC-01,TC-02,TC-03") + res = {m["result"]["metric"]: m["result"] for m in msgs if "result" in m} + score = res["score"] + assert (score["value"], score["extras"]["graded"], score["extras"]["completion_rate"]) == (100, 2, 66.7) + assert res["completion_rate"]["value"] == 66.7 # never a score without its share graded + detail = merged_detail(msgs) + te = detail["te:TC-02"] + assert (te["status"], te["excluded"], te["points"], te["failure_kind"]) == ( + "excluded", + True, + 0, + "server_error", + ) + assert detail["tool_eval"]["excluded"] == ["TC-02"] and detail["tool_eval"]["completion_rate"] == 66.7 + final = [m["live"] for m in msgs if "live" in m][-1] + assert final["counts"] == {"pass": 2, "partial": 0, "fail": 0, "excluded": 1} + assert (final["done"], final["points"], final["max_points"]) == (3, 4, 4) + assert {t["id"]: t["status"] for t in final["board"]} == { + "TC-01": "pass", + "TC-02": "excluded", + "TC-03": "pass", + } + note = ( + "1 of 3 scenarios was excluded after timeouts or server errors (TC-02); the score covers only the " + "graded ones." + ) + assert final["warnings"] == [note] and {"log": note} in msgs + + +def test_tool_eval_fails_when_no_scenario_could_be_graded(tmp_path, fake_tools): + fake_tools.faults["http503"] = {"TC-01", "TC-02"} + msgs = run_tool_eval(tmp_path, fake_tools.url, "--suite", "core", "--scenarios", "TC-01,TC-02", rc=1) + assert [m["error"] for m in msgs if "error" in m] == [ + { + "message": "No scenario could be graded", + "cause": "Every scenario timed out or hit a connection or server error.", + "fix": "Check the endpoint and its server logs; for vLLM make sure tool calling is enabled.", + } + ] + assert set(merged_detail(msgs)) == {"te:TC-01", "te:TC-02"} # the record of what happened, no summary + assert not any("result" in m for m in msgs) + + +def test_tool_eval_sigterm_cancels_at_once_with_a_final_snapshot(tmp_path, fake_tools): + fake_tools.faults["slow"] = {"TC-02": 60} + p = subprocess.Popen( + argv(fake_tools.url, "--suite", "core", "--scenarios", "TC-01,TC-02,TC-03"), + stdout=subprocess.PIPE, + text=True, + cwd=tmp_path, + ) + assert p.stdout is not None + for line in p.stdout: + if '"current":{"id":"TC-02"' in line: # TC-02 is running: its request waits on the server + break + t0 = time.monotonic() + p.send_signal(signal.SIGTERM) + rest = [json.loads(ln) for ln in p.stdout.read().splitlines() if ln.startswith("{")] + assert p.wait(10) == 143 and time.monotonic() - t0 < 3 + final = next(m["live"] for m in rest if "live" in m) + assert final["current"] is None and (final["done"], final["counts"]["pass"]) == (1, 1) + assert [t["status"] for t in final["board"]] == ["pass", "pending", "pending"] + assert rest[-1]["log"].startswith("Cancelled after 1 of 3 scenarios") + assert not any("result" in m or "tool_eval" in m.get("detail", {}) for m in rest) + + +def test_scores_and_fingerprint_match_tool_eval_benchs_own_cli(tmp_path, fake_tools): + """Same server and settings: the payload and `tool-eval-bench run --short` agree on every verdict, the + score and the comparison fingerprint (only the scenario selector label differs).""" + fake_tools.faults["think"] = True # so the score is below 100: TC-10 fails + ours = merged_detail(run_tool_eval(tmp_path, fake_tools.url, "--suite", "core"))["tool_eval"]["envelope"] + cli_dir = tmp_path / "cli" + cli_dir.mkdir() + cli_argv = [ + sys.executable, + "-m", + "tool_eval_bench", + "run", + "--base-url", + fake_tools.url, + "--model", + MODEL, + ] + cli_argv += ["--short", "--json", "--json-file", "result.json", "--no-warmup"] + out = subprocess.run(cli_argv, capture_output=True, text=True, timeout=120, cwd=cli_dir) + assert out.returncode == 0, out.stderr[-3000:] + cli = json.loads((cli_dir / "result.json").read_text(encoding="utf-8")) + assert ours["config"]["config_fingerprint"] == cli["config"]["config_fingerprint"] + assert ours["final_score"] == cli["final_score"] < 100 and ours["rating"] == cli["rating"] + + def verdicts(env: dict) -> list[tuple]: + return [ + (x["scenario_id"], x["status"], x["points"], x.get("failure_kind")) + for x in env["scores"]["scenario_results"] + ] + + assert verdicts(ours) == verdicts(cli) and len(verdicts(cli)) == 15 + assert (ours["metadata"]["scenario_selector"], cli["metadata"]["scenario_selector"]) == ( + ", ".join(f"TC-{i:02d}" for i in range(1, 16)), + "short (15)", + ) + + +# ----------------------------------------------------------------------------------------------------- +# Payload internals: callbacks never raise; detail stays under the agent's limits +# ----------------------------------------------------------------------------------------------------- + + +@pytest.fixture +def payload(monkeypatch): + monkeypatch.syspath_prepend(PAYLOAD_DIR) + import tool_eval + + return tool_eval + + +async def test_a_failing_progress_update_never_aborts_the_run(payload, capsys): + def broken(*args): + raise KeyError("boom") + + await payload.safe(broken)("scenario", 0, 1) # tool-eval-bench would abort the run on a raise + assert json.loads(capsys.readouterr().out) == { + "log": "Couldn't report scenario progress: KeyError: 'boom'" + } + + +def _result(s, **kw) -> ScenarioResult: + return ScenarioResult(scenario_id=s.id, status=ScenarioStatus.PASS, points=2, summary="ok", **kw) + + +def test_scenario_records_fit_the_budget_with_long_non_ascii_thinking_traces(payload, capsys): + scenarios = list(ALL_SCENARIOS_WITH_HARDMODE) + head = {"version": "2.8.0", "suite": "all", "thinking": True, "model": MODEL, "engine": "vllm"} + board = payload.Board(scenarios, head, []) + trace = "model=m\n" + "思考" * 30_000 + "\nverdict=pass" # each of these characters is 6 bytes of JSON + calls = [f"send_email(body={'x' * 900})"] * 10 + for i, s in enumerate(scenarios): + board.started(s, i, len(scenarios)) + board.finished(s, _result(s, raw_log=trace, tool_calls_made=calls), i, len(scenarios)) + out = [json.loads(ln) for ln in capsys.readouterr().out.splitlines()] + records = {k: v for m in out if "detail" in m for k, v in m["detail"].items()} + assert len(records) == 97 and all(payload.size(r) <= board.record_max for r in records.values()) + assert payload.size(records) <= payload.RECORDS_BUDGET and abs(board.used - payload.size(records)) <= 1 + te = records["te:TC-97"] + assert ( + te["trace"].startswith("model=m") + and te["trace"].endswith("verdict=pass") + and "omitted" in te["trace"] + ) + assert all(len(c) <= 200 for c in te["tool_calls"]) + lives = [m["live"] for m in out if "live" in m] + assert max(payload.size(lv) for lv in lives) < LIVE_MAX_BYTES // 10 # ~10 KB for 97 scenarios + many = board.record(scenarios[0], _result(scenarios[0], tool_calls_made=["c"] * 40), "pass") + assert len(many["tool_calls"]) == 32 and many["tool_calls"][-1] == "… 9 more" + + +@pytest.mark.parametrize("n", [6, 32]) +async def test_long_non_ascii_tool_calls_cost_no_record_nor_the_summary(payload, capsys, tmp_path, n): + """n 200-character tool calls in Chinese in every one of 97 scenarios: each record is cut to its share + (shorter tool calls before fewer), so the agent keeps every record and the summary.""" + scenarios = list(ALL_SCENARIOS_WITH_HARDMODE) + head = {"version": "2.8.0", "suite": "all", "thinking": True, "model": MODEL, "engine": "vllm"} + board = payload.Board(scenarios, head, []) + calls = [f"send_email(to=li@example.com, body={'您好' * 100})"] * n + results = [_result(s, raw_log="trace\n" * 5000, tool_calls_made=calls) for s in scenarios] + for i, (s, r) in enumerate(zip(scenarios, results, strict=True)): + board.started(s, i, len(scenarios)) + board.finished(s, r, i, len(scenarios)) + run = {"run_id": "r", "status": "completed", "config": {"config_fingerprint": "0" * 12}, "metadata": {}} + run |= {"scores": score_results(results, scenarios).to_dict(), "safety_gate": {"passed": True}} + payload.report(json.loads(json.dumps(format_result(run))), board) + job = make_job(tmp_path, "tool-eval") # as the agent takes the payload's lines + for line in capsys.readouterr().out.splitlines(): + job.handle(json.loads(line)) + records = {k: v for k, v in job.state.detail.items() if k.startswith("te:")} + assert len(records) == 97 and all(payload.size(r) <= board.record_max for r in records.values()) + assert all(len(r["tool_calls"]) == n for r in records.values()) # shorter, but every one listed + assert "tool_eval" in job.state.detail and "detail too large, dropped" not in job.state.log_tail + assert payload.size(job.state.detail) <= DETAIL_MAX_BYTES + + +def test_a_record_never_outgrows_its_share(payload): + """Whatever a scenario produced (here 40 tool calls, the evaluator's texts and the trace, all in + characters JSON escapes to 12 bytes), its record fits: the trace goes first, then tool calls get shorter, + then fewer (the note on the unlisted ones stays true), then the texts.""" + scenarios = list(ALL_SCENARIOS_WITH_HARDMODE) + head = {"version": "2.8.0", "suite": "all", "thinking": True, "model": MODEL, "engine": "vllm"} + board = payload.Board(scenarios, head, []) + text = "🙂" * 1000 + for s in scenarios: + r = ScenarioResult( + scenario_id=s.id, + status=ScenarioStatus.FAIL, + points=0, + summary=text, + note=text, + safety_violation=text, + raw_log=text * 20, + tool_calls_made=[f"call({text})"] * 40, + turn_latencies_ms=[1.0] * 14, + ) + rec = board.record(s, r, "fail") + assert payload.size(rec) <= board.record_max + *listed, more = rec["tool_calls"] + assert more == f"… {40 - len(listed)} more" + assert rec["summary"] == rec["note"] == rec["safety_violation"] and rec["summary"].startswith("🙂") + + +def test_an_oversized_envelope_leaves_out_its_scenario_results_not_the_summary(payload, capsys): + head = {"version": "2.8.0", "suite": "core", "thinking": True, "model": MODEL, "engine": "vllm"} + board = payload.Board(list(SCENARIOS), head, []) + board.used = payload.RECORDS_BUDGET # scenario records sent so far + scores = {"final_score": 90, "rating": "★★★★★ Excellent", "total_points": 27, "max_points": 30} + scores |= { + "category_scores": [], + "scenario_results": [ + {"scenario_id": "TC-01", "raw_log": "x" * 9, "tool_calls_made": ["y" * 500_000]} + ], + } + env = {"config": {}, "metadata": {}, "safety_gate": {"passed": True, "warnings": []}, "scores": scores} + payload.report(env, board) + out = [json.loads(ln) for ln in capsys.readouterr().out.splitlines()] + final = next(m["detail"]["tool_eval"] for m in out if "detail" in m) + assert final["final_score"] == 90 and final["envelope"]["scores"]["scenario_results"] == [] + assert board.used + payload.size({"tool_eval": final}) < DETAIL_MAX_BYTES + omitted = ( + "tool-eval-bench's result envelope is too large for the run record: per-scenario results omitted" + ) + assert {"log": omitted} in out + assert {m["result"]["metric"] for m in out if "result" in m} == { + "score", + "completion_rate", + "safety_violations", + } + + +# ----------------------------------------------------------------------------------------------------- +# Demo mode: the same shapes from a simulated run, and a finished evaluation in the seeded history +# ----------------------------------------------------------------------------------------------------- + + +@pytest.fixture +def demo(tmp_path, monkeypatch): + monkeypatch.setattr(demo_bench, "demo_duration", lambda bench, p: 2.0) + cfg = ConsoleConfig(web_dist=tmp_path / "nodist") + with TestClient(create_app(cfg, tmp_path / "data", demo=True)) as c: + yield c + + +def _start(c, node: str, **params) -> str: + body = {"bench_id": "tool-eval", "nodes": [node], "params": params, "confirm": True} + r = c.post("/api/v1/bench/runs", json=body) + assert r.status_code == 201, r.json() + return r.json()["id"] + + +def _until(c, run_id: str, cond, timeout: float = 15) -> BenchRun: + deadline = time.time() + timeout + while time.time() < deadline: + run = BenchRun.model_validate(c.get(f"/api/v1/bench/runs/{run_id}").json()) + if cond(run): + return run + time.sleep(0.05) + raise AssertionError(f"run {run_id} never reached the state") + + +def test_demo_tool_eval_run_shows_the_payloads_live_detail_and_results(demo): + run_id = _start(demo, "spark1", suite="core") + running = _until(demo, run_id, lambda r: (r.node_states["spark1"].live or {}).get("current")) + live = running.node_states["spark1"].live + assert set(live) == LIVE_KEYS and live["kind"] == "tool-eval" and live["total"] == 15 + assert [t["status"] for t in live["board"]].count("running") == 1 + assert running.phase.startswith(f"Scenario {live['done'] + 1} of 15 · TC-") + run = _until(demo, run_id, lambda r: r.status != "running") + assert run.status == "succeeded", run.error + st = run.node_states["spark1"] + final = st.detail["tool_eval"] + assert set(final) == SUMMARY_KEYS and set(st.detail) == { + "tool_eval", + *(f"te:TC-{i:02d}" for i in range(1, 16)), + } + assert all(set(st.detail[f"te:TC-{i:02d}"]) == TE_KEYS for i in range(1, 16)) + assert st.live["done"] == 15 and st.live["current"] is None and st.live["started_ts"] > 0 + assert ( + final["envelope"]["schema_version"] == "1" + and final["envelope"]["final_score"] == final["final_score"] + ) + res = {x.metric: x for x in run.results} + # make_results replays the run the live view showed + assert res["score"].value == final["final_score"] and res["score"].extras["simulated"] is True + assert res["score"].extras["total_points"] == st.live["points"] == final["total_points"] + assert [x.metric for x in run.results if x.extras.get("headline")] == [ + "score", + "completion_rate", + "median_turn_ms", + ] + assert 70 <= res["score"].value <= 100 and res["completion_rate"].value == 100.0 + assert "Sparkline demo: simulated" in st.detail["te:TC-01"]["trace"] + + +def test_demo_tool_eval_on_spark3_shows_an_exclusion_and_a_safety_warning(demo): + run = _until(demo, _start(demo, "spark3"), lambda r: r.status != "running") # standard: 69 scenarios + assert run.status == "succeeded", run.error + st = run.node_states["spark3"] + final, te45 = st.detail["tool_eval"], st.detail["te:TC-45"] + assert final["excluded"] == ["TC-45"] and final["completion_rate"] == 98.6 + assert (te45["status"], te45["excluded"], te45["failure_kind"]) == ("excluded", True, "server_error") + assert final["safety_warnings"] == [f"TC-60 (Cross-Turn Sleeper Injection): {demo_bench.TE_SLEEPER}"] + assert ( + final["safety_gate_passed"] is False + and st.detail["te:TC-60"]["safety_violation"] == demo_bench.TE_SLEEPER + ) + assert st.live["warnings"] == [ + "1 of 69 scenarios was excluded after timeouts or server errors (TC-45); the score covers only the " + "graded ones." + ] + res = {x.metric: x for x in run.results} + assert res["completion_rate"].value == 98.6 and res["safety_violations"].value == 1 + assert res["score"].extras["graded"] == 68 and res["score"].extras["model"] == "Qwen3-32B" + + +def test_demo_tool_eval_cancel_keeps_the_finished_scenarios(demo, monkeypatch): + monkeypatch.setattr(demo_bench, "demo_duration", lambda bench, p: 12.0) + run_id = _start(demo, "spark2", suite="core") + _until(demo, run_id, lambda r: any(k.startswith("te:") for k in r.node_states["spark2"].detail)) + demo.post(f"/api/v1/bench/runs/{run_id}/cancel") + run = _until(demo, run_id, lambda r: r.status != "running") + st = run.node_states["spark2"] + assert run.status == "cancelled" and run.results == [] and "tool_eval" not in st.detail + assert st.live["current"] is None and "running" not in {t["status"] for t in st.live["board"]} + assert len([k for k in st.detail if k.startswith("te:")]) == st.live["done"] >= 1 + + +def test_demo_history_holds_a_finished_evaluation(demo): + seeded = [r for r in demo.get("/api/v1/bench/runs?bench_id=tool-eval").json()] + assert len(seeded) == 1 and seeded[0]["status"] == "succeeded" and seeded[0]["nodes"] == ["spark1"] + run = BenchRun.model_validate(demo.get(f"/api/v1/bench/runs/{seeded[0]['id']}").json()) + st = run.node_states["spark1"] + assert st.status == "succeeded" and st.live["kind"] == "tool-eval" and st.live["done"] == 69 + assert len([k for k in st.detail if k.startswith("te:")]) == 69 + final = st.detail["tool_eval"] + assert final["completion_rate"] == 100.0 and final["safety_warnings"] == [] + score = next(x for x in run.results if x.metric == "score") + assert score.value == final["final_score"] and 70 <= score.value <= 100 and score.node_id == "spark1" + assert len(json.dumps(st.detail)) < 400_000 # the seeded run stays a light first impression diff --git a/server/tests/test_console_bench.py b/server/tests/test_console_bench.py index cf07164..94162de 100644 --- a/server/tests/test_console_bench.py +++ b/server/tests/test_console_bench.py @@ -32,6 +32,7 @@ canonical, merge_description, new_run_id, + params_key, rail_ip, regression_title, runs_csv, @@ -294,6 +295,56 @@ def test_params_key_gates_baseline_comparison(tmp_path): assert same.metadata["regression_pct"] == 5.0 # the threshold the flags were set with +def test_params_key_leaves_out_params_added_later_while_they_are_neutral(): + """llm-inference runs stored before its sweep param existed keep matching new runs that don't sweep.""" + new = validate_params(AGENT_REGISTRY["llm-inference"].spec({}), {"concurrency": 4, "requests": 16}) + old = canonical({k: v for k, v in new.items() if k != "sweep"}) # what the console stored back then + assert new["sweep"] is False and params_key("llm-inference", new) == old + swept = params_key("llm-inference", {**new, "sweep": True}) + assert swept != old and '"sweep":true' in swept + gemm = validate_params(AGENT_REGISTRY["gpu-matmul"].spec({}), {}) + assert params_key("gpu-matmul", gemm) == canonical(gemm) + + +def test_llm_runs_are_keyed_on_the_requests_they_send(tmp_path): + """Every stream gets a request now, so Requests below Concurrency sends Concurrency requests. A baseline + stored before that sent fewer, fewer at once: it no longer matches (no false regression), while one with + Requests >= Concurrency still matches exactly.""" + spec = AGENT_REGISTRY["llm-inference"].spec({}) + mgr, _ = _manager(tmp_path) + + def run(rid: str, ttft: float, old: bool = False, **over) -> BenchRun: + params = validate_params(spec, over) + if old: # stored before sweeps: no sweep param, the plain canonical key + params = {k: v for k, v in params.items() if k != "sweep"} + key = canonical(params) if old else params_key("llm-inference", params) + res = [_res("ttft_p50_ms", ttft, "lower")] + return BenchRun( + id=rid, + bench_id="llm-inference", + params=params, + nodes=["spark1"], + status="succeeded", + created_ts=time.time(), + params_key=key, + results=res, + ) + + for base in (run("few", 200.0, old=True, concurrency=16, requests=8), run("most", 200.0, old=True)): + mgr.store.save(base) + mgr.store.set_baseline("llm-inference", base.params_key, base.id) + now = run("now", 300.0, concurrency=16, requests=8) # 16 requests, 16 at once: slower, as it should be + assert '"requests":16' in now.params_key + assert mgr._compare_baseline(now) == [] and now.baseline_run_id is None and not now.results[0].regression + same = run("same", 300.0) # 8 requests one at a time, as before + assert same.params_key == mgr.store.get("most").params_key + assert len(mgr._compare_baseline(same)) == 1 and same.baseline_run_id == "most" + both = [validate_params(spec, {"concurrency": 16, "requests": n}) for n in (8, 16)] + assert params_key("llm-inference", both[0]) == params_key("llm-inference", both[1]) # the same workload + swept = [params_key("llm-inference", {**p, "sweep": True}) for p in both] # each level sends its own max + assert swept[0] != swept[1] and '"requests":8' in swept[0] + + def test_store_lists_filters_and_deletes(tmp_path): mgr, _ = _manager(tmp_path) st = mgr.store @@ -407,6 +458,39 @@ def test_unsuccessful_runs_discard_partial_results_but_keep_series(tmp_path): assert saved.status == status and saved.results == [] and saved.series["spark1.t"] == [0.0, 1.0] +class _Browser: + async def send_text(self, data: str) -> None: # never pumped: broadcasts stay queued for the test + pass + + +def test_live_and_detail_stay_with_the_run_but_not_in_list_copies(tmp_path): + """Node states are kept verbatim: live/detail go out with every broadcast and are stored with the run (GET + run, export); list copies leave them out with the rest of node_states.""" + mgr, cluster = _manager(tmp_path) + browser = cluster.hub.add(_Browser()) + run = _run("r", canonical({}), 0.0, status="running") + run.results = [] + mgr.active[run.id] = _Active(run=run, name="GPU matmul") + live = {"kind": "llm-streams", "streams": [{"slot": 0, "tok_s": None}]} + detail = {"llm": {"levels": [{"level": 1, "out_tok_s": 46.1}]}} + st = NodeBenchState( + run_id="r", bench_id="gpu-matmul", node_id="spark1", status="running", live=live, detail=detail + ) + mgr.on_node_state("spark1", st) + done = st.model_copy(update={"status": "succeeded", "results": [_res("tflops", 1.0)]}) + mgr.on_node_state("spark1", done) + sent = [json.loads(m) for m in browser.priority] + states = [m["run"]["node_states"]["spark1"] for m in sent if m["type"] == "bench"] + assert [s["status"] for s in states] == ["running", "succeeded"] + assert all(s["live"] == live and s["detail"] == detail for s in states) + saved = mgr.get("r") # finished: read back from the store + assert saved.status == "succeeded" and "r" not in mgr.active + assert saved.node_states["spark1"].live == live and saved.node_states["spark1"].detail == detail + assert [r.node_states for r in mgr.list(None, None, None, 10)] == [{}] + old = NodeBenchState.model_validate({"run_id": "r", "bench_id": "b", "node_id": "n", "status": "running"}) + assert old.live == {} and old.detail == {} # stored runs and agents from before these fields + + def test_smoke_results_never_compare_with_measurements(): smoke = _res("bf16_tflops", 0.005) smoke.extras = {"smoke": True} diff --git a/server/tests/test_console_bench_demo.py b/server/tests/test_console_bench_demo.py index af2ed1a..b0f0432 100644 --- a/server/tests/test_console_bench_demo.py +++ b/server/tests/test_console_bench_demo.py @@ -3,17 +3,19 @@ from __future__ import annotations +import json import random import time import pytest from fastapi.testclient import TestClient +from sparkline.agent.bench.runner import DETAIL_MAX_BYTES, LIVE_MAX_BYTES from sparkline.agent.bench.specs import REGISTRY as AGENT_REGISTRY from sparkline.common.schema import BenchRun, BenchSpec, Event from sparkline.console import demo_bench from sparkline.console.app import create_app -from sparkline.console.bench import REGISTRY, validate_params +from sparkline.console.bench import REGISTRY, params_key, validate_params from sparkline.console.config import ConsoleConfig from sparkline.console.demo import DemoModel from sparkline.console.demo_bench import demo_catalog, demo_duration, make_results, validate @@ -42,9 +44,13 @@ def test_demo_durations_are_compressed(): real = AGENT_REGISTRY[b].estimate(AGENT_REGISTRY[b].validate(p)) if b == "thermal-soak": assert demo_duration(b, p) == 60 + elif b == "llm-inference": # compressed less, so its streams can be watched + assert 6 <= demo_duration(b, p) <= real / 4 else: assert demo_duration(b, p) == max(4.0, real / 10) assert demo_duration("thermal-soak", validate("thermal-soak", {"minutes": 30})) == 90 + sweep = validate("llm-inference", {"concurrency": 32, "sweep": True}) + assert 0.6 + 6 * 4 <= demo_duration("llm-inference", sweep) <= 60 # 6 levels, each on screen 4 s or more def _vals(bench: str, node: str = "spark1", seed: int = 0, role: str = "single", info=None, **over): @@ -114,6 +120,7 @@ def test_demo_storage_run_loads_the_disk_and_says_runs_are_compressed(): assert m.sample_at(t + 1.5).disk[0].read_bps > 5e9 # then sequential read assert after.write_bps < 1e8 desc = {s.id: s.description for s in demo_catalog("spark1", m.info)} + assert desc.pop("llm-inference").endswith(demo_bench.LLM_NOTE) # compressed less: streams to watch assert all(demo_bench.COMPRESSED_NOTE in d for d in desc.values()) @@ -175,8 +182,9 @@ def post(bench, nodes, params): def test_demo_history_is_seeded_with_a_baseline_and_a_regression(demo): runs = [BenchRun.model_validate(r) for r in demo.get("/api/v1/bench/runs").json()] - assert len(runs) == 12 and {r.bench_id for r in runs} == set(REGISTRY) + assert len(runs) == len(demo_bench._SEED) and {r.bench_id for r in runs} == set(REGISTRY) assert {r.status for r in runs} == {"succeeded", "failed", "cancelled"} + assert all(r.params_key == params_key(r.bench_id, r.params) for r in runs) # as the console keys runs latest = runs[0] assert latest.bench_id == "gpu-matmul" and latest.baseline_run_id is not None bf16 = next(x for x in latest.results if x.metric == "bf16_tflops") @@ -199,7 +207,113 @@ def test_demo_history_is_seeded_with_a_baseline_and_a_regression(demo): def test_demo_empty_bench_flag(tmp_path): cfg = ConsoleConfig(web_dist=tmp_path / "nodist") with TestClient(create_app(cfg, tmp_path / "data", demo=True)) as c: - assert len(c.get("/api/v1/bench/runs").json()) == 12 + assert len(c.get("/api/v1/bench/runs").json()) == len(demo_bench._SEED) with TestClient(create_app(cfg, tmp_path / "data", demo=True, demo_empty_bench=True)) as c: assert c.get("/api/v1/bench/runs").json() == [] - assert len(c.get("/api/v1/bench/catalog").json()) == 8 + assert len(c.get("/api/v1/bench/catalog").json()) == 9 + + +def _compact(obj) -> int: + return len(json.dumps(obj, separators=(",", ":"), allow_nan=False)) + + +def test_demo_llm_streams_are_plausible_and_within_the_agents_limits(): + p = validate("llm-inference", {"concurrency": 32, "sweep": True}) + qwen = demo_bench.LlmDemo(p, "spark3", random.Random(3)) + assert (qwen.sizes, qwen.counts) == ([1, 2, 4, 8, 16, 32], [8, 8, 8, 8, 16, 32]) + levels = [qwen.summary(k) for k in range(6)] + assert 30 <= levels[0]["decode_tok_s_p50"] <= 60 and 150 <= levels[0]["ttft_p50_ms"] <= 600 + decode = [lv["decode_tok_s_p50"] for lv in levels] + assert decode == sorted(decode, reverse=True) # more streams: each one slower, ... + assert levels[-1]["ttft_p50_ms"] > 3 * levels[0]["ttft_p50_ms"] # ... a longer wait, more output in all + assert levels[-1]["out_tok_s"] > 4 * levels[0]["out_tok_s"] + mid, end = qwen.snapshot(5, qwen.duration(5) / 2), qwen.snapshot(5, qwen.duration(5)) + for snap in (mid, end): + assert (snap["kind"], snap["level"], snap["level_index"]) == ("llm-streams", 32, 5) + assert len(snap["streams"]) == 32 and _compact(snap) <= LIVE_MAX_BYTES + assert "decode" in {s["state"] for s in mid["streams"]} and any(s["think"] for s in mid["streams"]) + assert [s["state"] for s in end["streams"]] == ["done"] * 32 and end["completed_levels"] == levels + times = [qwen.duration(0) * f for f in (0.1, 0.3, 0.6, 1.0)] # one stream, request after request + chars = [qwen.snapshot(0, t)["streams"][0]["chars"] for t in times] + assert chars == sorted(chars) and chars[-1] > chars[0] + plain = demo_bench.LlmDemo(p, "spark1", random.Random(3)) # only spark3's Qwen reasons + assert not any(s["think"] for k in range(6) for s in plain.snapshot(k, plain.duration(k) / 2)["streams"]) + results = qwen.results() + heads = [r.metric for r in results if r.extras.get("headline")] + assert heads == ["peak_output_tok_s", "decode_tok_s_c1", "ttft_p50_ms_c32"] + assert all(r.extras["level"] in qwen.sizes and r.extras["sweep"] for r in results[:-2]) + val = {r.metric: r.value for r in results} + for lv in levels: # the sweep table shows exactly the results' numbers, as the payload's does + n = lv["level"] + assert (val[f"decode_tok_s_total_c{n}"], val[f"decode_tok_s_c{n}"]) == ( + lv["out_tok_s"], + lv["decode_tok_s_p50"], + ) + assert (val[f"ttft_p50_ms_c{n}"], val[f"ttft_p95_ms_c{n}"]) == (lv["ttft_p50_ms"], lv["ttft_p95_ms"]) + assert val["peak_output_tok_s"] == max(lv["out_tok_s"] for lv in levels) + one = demo_bench.LlmDemo(validate("llm-inference", {}), "spark1", random.Random(3)).results() + assert [r.metric for r in one] == [ + "decode_tok_s", + "ttft_p50_ms", + "prefill_tok_s", + "ttft_p95_ms", + "decode_tok_s_total", + "serving_gpu_mem_gib", + "mem_used_delta_gib", + ] + big = validate("llm-inference", {"concurrency": 32, "requests": 512, "max_tokens": 4096, "sweep": True}) + most = demo_bench.LlmDemo(big, "spark3", random.Random(3)) # the largest run the agent accepts + assert _compact(most.snapshot(5, most.duration(5) / 3)) <= LIVE_MAX_BYTES + detail = most.detail(6) + assert _compact(detail) <= DETAIL_MAX_BYTES and all(t["truncated"] for t in detail["timeline"].values()) + + +def test_runs_are_keyed_on_the_requests_they_send(demo): + body = {"bench_id": "llm-inference", "nodes": ["spark2"], "params": {"concurrency": 16, "requests": 4}} + r = demo.post("/api/v1/bench/runs", json={**body, "confirm": True}) + assert r.status_code == 201, r.json() + run = BenchRun.model_validate(r.json()) + assert run.params["requests"] == 4 and '"requests":16' in run.params_key # one request per stream + assert _wait(demo, run.id)["status"] == "succeeded" + + +def test_demo_llm_sweep_streams_live_and_leaves_detail_and_results(demo): + body = {"bench_id": "llm-inference", "nodes": ["spark3"], "params": {"concurrency": 4, "sweep": True}} + r = demo.post("/api/v1/bench/runs", json={**body, "confirm": True}) + assert r.status_code == 201, r.json() + seen, deadline = [], time.time() + 15 + while time.time() < deadline: + run = demo.get(f"/api/v1/bench/runs/{r.json()['id']}").json() + if run["status"] not in ("queued", "running"): + break + seen.append(run["node_states"]["spark3"]["live"]) + time.sleep(0.1) + done = BenchRun.model_validate(run) + assert done.status == "succeeded" and '"sweep":true' in done.params_key + assert done.params_key == params_key("llm-inference", done.params) + assert {x["level"] for x in seen if x} >= {1, 2} # streams were live while it ran + st = done.node_states["spark3"] + assert st.live["level"] == 4 and [s["state"] for s in st.live["streams"]] == ["done"] * 4 + assert [lv["level"] for lv in st.detail["llm"]["levels"]] == [1, 2, 4] + heads = [x.metric for x in done.results if x.extras.get("headline")] + assert heads == ["peak_output_tok_s", "decode_tok_s_c1", "ttft_p50_ms_c4"] + seeded = [x for x in demo.get("/api/v1/bench/runs").json() if x["bench_id"] == "llm-inference"] + assert seeded and all("sweep" not in x["params_key"] for x in seeded if not x["params"]["sweep"]) + + +def test_demo_history_holds_a_finished_sweep(demo): + sweeps = [x for x in demo.get("/api/v1/bench/runs?bench_id=llm-inference").json() if x["params"]["sweep"]] + assert len(sweeps) == 1 and sweeps[0]["status"] == "succeeded" and sweeps[0]["node_states"] == {} # list + run = BenchRun.model_validate(demo.get(f"/api/v1/bench/runs/{sweeps[0]['id']}").json()) + st = run.node_states["spark1"] + assert (st.status, st.live["kind"], st.live["level"]) == ("succeeded", "llm-streams", 32) + assert [s["state"] for s in st.live["streams"]] == ["done"] * 32 + levels = st.detail["llm"]["levels"] + assert [lv["level"] for lv in levels] == st.live["levels"] == [1, 2, 4, 8, 16, 32] + assert set(st.detail["llm"]["timeline"]) == {"1", "2", "4", "8", "16", "32"} + assert _compact(st.live) <= LIVE_MAX_BYTES and _compact(st.detail) <= DETAIL_MAX_BYTES + heads = [x.metric for x in run.results if x.extras.get("headline")] + assert heads == ["peak_output_tok_s", "decode_tok_s_c1", "ttft_p50_ms_c32"] + assert {x.node_id for x in run.results} == {"spark1"} + took = (run.finished_ts or 0) - (run.started_ts or 0) # as long as its requests took + assert took == pytest.approx(demo_bench.LLM_START_S + sum(lv["wall_s"] for lv in levels), abs=0.01) diff --git a/server/tests/test_console_hub.py b/server/tests/test_console_hub.py index f8c5e23..fc452fc 100644 --- a/server/tests/test_console_hub.py +++ b/server/tests/test_console_hub.py @@ -1,8 +1,9 @@ from __future__ import annotations import asyncio +import logging -from sparkline.console.hub import PRIORITY_QUEUE, SAMPLE_QUEUE, Hub +from sparkline.console.hub import PRIORITY_BYTES, PRIORITY_QUEUE, SAMPLE_QUEUE, Hub class StuckSocket: @@ -42,3 +43,52 @@ async def test_priority_overflow_disconnects_instead_of_growing(): hub.broadcast({"type": "event", "i": i}) assert len(c.priority) == PRIORITY_QUEUE and c.overflowed await asyncio.wait_for(c.pump(), 1) # returns immediately: caller closes the socket + + +BENCH = '{"type":"bench","run":"' + "x" * 330_000 + '"}' # a late tool-eval broadcast, live and detail data + + +async def test_priority_overflow_by_size_disconnects_long_before_the_count(caplog): + """A stalled browser is cut off once PRIORITY_BYTES are queued, not after PRIORITY_QUEUE bench broadcasts + (~660 MB of them).""" + hub = Hub() + c = hub.add(StuckSocket()) + fits = PRIORITY_BYTES // len(BENCH) + for _ in range(fits): + hub.broadcast_text(BENCH, droppable=False) + assert not c.overflowed and c.priority_bytes == fits * len(BENCH) + hub.broadcast_text(BENCH, droppable=False) + assert c.overflowed and len(c.priority) == fits < PRIORITY_QUEUE + with caplog.at_level(logging.WARNING, logger="sparkline.console.hub"): + await asyncio.wait_for(c.pump(), 1) # returns immediately: caller closes the socket + assert "overflowed its event queue; disconnecting it" in caplog.text + + +class ReadingSocket: + """A browser that keeps up.""" + + def __init__(self): + self.sent = 0 + + async def send_text(self, data: str) -> None: + self.sent += 1 + await asyncio.sleep(0) + + +async def test_sent_messages_give_their_bytes_back(): + hub = Hub() + ws = ReadingSocket() + c = hub.add(ws) + pump = asyncio.create_task(c.pump()) + n = 3 * PRIORITY_BYTES // len(BENCH) # three budgets' worth in all, sent as they come + for _ in range(n): + hub.broadcast_text(BENCH, droppable=False) + await asyncio.sleep(0) + await asyncio.wait_for(_until(lambda: ws.sent == n), 5) + assert not c.overflowed and c.priority_bytes == 0 and not c.priority + pump.cancel() + + +async def _until(cond) -> None: + while not cond(): + await asyncio.sleep(0) diff --git a/web/e2e/bench.spec.ts b/web/e2e/bench.spec.ts index 4edf43e..59588d3 100644 --- a/web/e2e/bench.spec.ts +++ b/web/e2e/bench.spec.ts @@ -1,14 +1,16 @@ /// // Benchmark flows against the demo console (simulated, time-compressed runs; nothing touches real hardware): -// configure -> confirm -> live progress -> results -> baseline -> compare -> export, the cancel flow, parameter -// validation, and axe on every benchmark screen in both themes. +// configure -> confirm -> live progress -> results -> baseline -> compare -> export, the cancel flow, the LLM +// benchmark's live streams and concurrency sweep, the tool-calling evaluation, parameter validation, and axe on every +// benchmark screen in both themes. import AxeBuilder from '@axe-core/playwright'; -import { expect, test, type Page } from '@playwright/test'; +import { expect, test, type Locator, type Page } from '@playwright/test'; interface RunRow { id: string; bench_id: string; status: string; + params: Record; } interface Spec { id: string; @@ -47,6 +49,44 @@ async function seededRuns(page: Page): Promise { return (await res.json()) as RunRow[]; } +const label = (spec: Spec, name: string) => spec.params.find((p) => p.name === name)?.label ?? name; + +/** Configure from the catalog (`set` changes parameters), confirm, and land on the run's own page; returns its id. */ +async function runFromCatalog(page: Page, spec: Spec, set: (dialog: Locator) => Promise): Promise { + await page.goto('/benchmarks'); + await catalogReady(page); + await page.getByRole('button', { name: `Configure ${spec.name}…` }).click(); + const dialog = page.getByRole('dialog', { name: `Configure ${spec.name}` }); + await set(dialog); + await dialog.getByRole('button', { name: /Review & run/ }).click(); + await page.getByRole('dialog', { name: `Run ${spec.name}?` }).getByRole('button', { name: 'Run benchmark' }).click(); + await expect(page).toHaveURL(/\/benchmarks\/runs\/[0-9a-z]+$/, { timeout: 10_000 }); + return page.url().split('/').pop() ?? ''; +} + +/** LLM inference sweeping 1, 2, 4 and 8 streams (the demo plays it in about 30 s). */ +async function runLlmSweep(page: Page): Promise { + const llm = (await catalog(page))['llm-inference']; + if (!llm) throw new Error('llm-inference missing from the catalog'); + await runFromCatalog(page, llm, async (dialog) => { + await dialog.getByRole('textbox', { name: label(llm, 'concurrency') }).fill('8'); + const sweep = dialog.getByRole('switch', { name: label(llm, 'sweep') }); + await sweep.click(); + await expect(sweep).toHaveAttribute('aria-checked', 'true'); + }); + return llm; +} + +/** Axe on the current page in the light theme, then again after the system switches to dark. */ +async function axeBothThemes(page: Page, what: string) { + await page.emulateMedia({ colorScheme: 'light' }); + await expect(page.locator('html')).toHaveAttribute('data-theme', 'light'); + expect(await axeViolations(page), `${what} (light)`).toEqual([]); + await page.emulateMedia({ colorScheme: 'dark' }); + await expect(page.locator('html')).toHaveAttribute('data-theme', 'dark'); + expect(await axeViolations(page), `${what} (dark)`).toEqual([]); +} + // The flows start real (simulated) runs on the shared demo, so they run one after another. test.describe.serial('benchmark flows', () => { test('configure, confirm, watch it run, set a baseline, export and compare', async ({ page }) => { @@ -134,6 +174,129 @@ test.describe.serial('benchmark flows', () => { await expect(page.locator('.bench-run__title-row')).toContainText('Cancelled'); await expect(page.getByRole('progressbar', { name: 'Benchmark progress' })).toHaveCount(0); }); + + test('LLM inference shows every concurrent stream live, then the sweep, the timeline and the final outputs', async ({ page }) => { + test.setTimeout(120_000); + await runLlmSweep(page); + const streams = page.getByRole('region', { name: 'Concurrency streams' }); + await expect(streams).toBeVisible({ timeout: 15_000 }); + const rail = streams.getByRole('list', { name: 'Concurrency levels' }); + await expect(rail.getByRole('listitem')).toHaveCount(4); // 1, 2, 4 and 8 streams + + // A card's text grows while its request decodes (same request, more text a moment later). + const card = streams.getByRole('list', { name: /^Streams: / }).getByRole('button').first(); + const sample = async () => ({ name: (await card.getAttribute('aria-label')) ?? '', text: (await card.locator('.strm-body').textContent()) ?? '' }); + await expect.poll(async () => { + const a = await sample(); + await page.waitForTimeout(300); + const b = await sample(); + return a.name === b.name && b.name.includes('decoding') && b.text.length > a.text.length; + }, { timeout: 20_000 }).toBe(true); + await axeBothThemes(page, 'streams live'); + + // The rail advances a level at a time; a finished level shows its output. + await expect(rail.locator('[aria-current="step"]')).toContainText('8 streams', { timeout: 45_000 }); + await expect(rail.getByRole('listitem').first()).toContainText('tok/s'); + await expect(streams.getByRole('list', { name: /^Streams: / }).getByRole('listitem')).toHaveCount(8); + + await expect(page.getByText(/^Completed in /)).toBeVisible({ timeout: 45_000 }); + await expect(streams).toHaveCount(0); + const sweep = page.getByRole('region', { name: 'Concurrency sweep' }); + await expect(sweep.getByRole('table', { name: 'Concurrency sweep, one row per level' }).getByRole('row')).toHaveCount(5); // header + 4 levels + await expect(page.getByRole('region', { name: 'Stream timeline' })).toContainText('at 8 streams'); + const outputs = page.getByRole('region', { name: 'Final outputs' }); + await outputs.getByText('Show 8 streams').click(); + await expect(outputs.getByRole('list', { name: 'Final outputs' }).getByRole('listitem')).toHaveCount(8); + await axeBothThemes(page, 'sweep finished'); + }); + + test('a cancelled LLM sweep says it stopped and keeps what it recorded', async ({ page }) => { + test.setTimeout(90_000); + const llm = await runLlmSweep(page); + const rail = page.getByRole('region', { name: 'Concurrency streams' }).getByRole('list', { name: 'Concurrency levels' }); + await expect(rail.locator('[aria-current="step"]')).toContainText('2 streams', { timeout: 30_000 }); + await page.getByRole('button', { name: 'Cancel run' }).click(); + await page.getByRole('dialog', { name: `Stop ${llm.name}?` }).getByRole('button', { name: 'Stop benchmark' }).click(); + + await expect(page.getByText(/^The benchmark was stopped after .* What it recorded before it stopped is shown below\.$/)).toBeVisible({ timeout: 15_000 }); + await expect(page.locator('.bench-run__title-row')).toContainText('Cancelled'); + await expect(page.getByRole('region', { name: 'Concurrency streams' })).toHaveCount(0); + await expect(page.getByRole('region', { name: 'Concurrency sweep' })).toContainText(/Stopped during level \d of 4/); + const outputs = page.getByRole('region', { name: 'Final outputs' }); + await expect(outputs).toContainText('when the run was cancelled'); + await outputs.getByText(/^Show \d+ streams$/).click(); + // Streams cut off mid-request read "stopped", never as if they had finished. + await expect(outputs.getByRole('list', { name: 'Final outputs' }).getByRole('button', { name: /: stopped/ }).first()).toBeVisible(); + }); + + test('the tool-calling evaluation fills its board, then shows the score, every scenario and upstream’s JSON', async ({ page }) => { + test.setTimeout(150_000); + const te = (await catalog(page))['tool-eval']; + if (!te) throw new Error('tool-eval missing from the catalog'); + const runId = await runFromCatalog(page, te, async (dialog) => { + await dialog.getByRole('combobox', { name: label(te, 'suite') }).selectOption('core'); + await expect(dialog.getByText(/Estimated/)).toBeVisible(); + }); + + const evaluation = page.getByRole('region', { name: 'Tool-calling evaluation' }); + await expect(evaluation.getByRole('region', { name: 'Now running' })).toContainText(/Now running · \d+ of 15/, { timeout: 20_000 }); + const board = evaluation.getByRole('group', { name: 'Scenario board' }); + await expect(board.getByRole('button')).toHaveCount(15); + const graded = () => board.getByRole('button', { name: /, (passed|partial|failed|excluded)$/ }).count(); + await expect.poll(graded, { timeout: 20_000 }).toBeGreaterThan(0); + const first = await graded(); + await expect.poll(graded, { timeout: 20_000 }).toBeGreaterThan(first); // verdicts keep arriving + await axeBothThemes(page, 'evaluation live'); + + await expect(page.getByText(/^Completed in /)).toBeVisible({ timeout: 60_000 }); + await expect(evaluation.getByRole('img', { name: /^Tool-calling score \d+%/ })).toBeVisible(); + await expect(page.getByRole('region', { name: 'Score per category' })).toBeVisible(); + const scenarios = page.getByRole('region', { name: 'Scenarios' }); + const table = scenarios.getByRole('table', { name: 'Scenario results' }); + await expect(table.getByRole('row')).toHaveCount(16); // header + 15 + + const filter = scenarios.getByRole('combobox', { name: 'Filter by result' }); + await filter.selectOption('pass'); + const passed = Number(/\((\d+)\)/.exec((await filter.locator('option:checked').textContent()) ?? '')?.[1]); + expect(passed).toBeGreaterThan(0); + await expect(table.getByRole('row')).toHaveCount(passed + 1); + await filter.selectOption('all'); + await scenarios.getByRole('searchbox', { name: 'Search scenarios' }).fill('unit handling'); + await expect(table.getByRole('row')).toHaveCount(2); + await table.getByRole('button', { name: 'Unit Handling' }).click(); + const dialog = page.getByRole('dialog', { name: /^TC-04 · Unit Handling/ }); + await expect(dialog.getByRole('list', { name: 'Upstream’s pass and fail criteria' }).getByRole('listitem')).toHaveCount(2); + await expect(dialog.getByRole('region', { name: 'Trace of TC-04' })).toBeVisible(); + await page.keyboard.press('Escape'); + await expect(dialog).toHaveCount(0); + + const json = page.waitForEvent('download'); + await evaluation.getByRole('button', { name: 'Download tool-eval-bench JSON' }).click(); + expect((await json).suggestedFilename()).toBe(`tool-eval-bench-${runId}.json`); + await axeBothThemes(page, 'evaluation finished'); + }); +}); + +test('the seeded concurrency sweep opens from History, with a timeline per level and its data table', async ({ page }) => { + await page.goto('/benchmarks'); + await catalogReady(page); + await page.getByLabel('Filter by benchmark').selectOption('llm-inference'); + await page.getByRole('table', { name: 'Benchmark runs' }).getByRole('row').filter({ hasText: 'Concurrency 32' }).getByRole('link', { name: 'LLM inference' }).click(); + await expect(page.getByRole('region', { name: 'Concurrency sweep' })).toContainText('6 levels'); + + const timeline = page.getByRole('region', { name: 'Stream timeline' }); + await expect(timeline).toContainText('32 requests at 32 streams'); // the last level first + const levels = timeline.getByRole('radiogroup', { name: 'Level' }); + await levels.getByRole('radio', { name: '8 streams' }).click(); + await expect(timeline).toContainText('8 requests at 8 streams'); + const toggle = timeline.getByRole('button', { name: 'Stream timeline: data table' }); + await toggle.click(); + await expect(toggle).toHaveAttribute('aria-expanded', 'true'); + await expect(timeline.getByRole('table', { name: 'Requests at 8 streams' }).getByRole('row')).toHaveCount(9); // header + 8 + await levels.getByRole('radio', { name: '16 streams' }).click(); + await expect(timeline.getByRole('table', { name: 'Requests at 16 streams' }).getByRole('row')).toHaveCount(17); + await toggle.click(); + await expect(timeline.getByRole('table')).toHaveCount(0); }); test('parameters are validated before the confirmation step', async ({ page }) => { @@ -176,15 +339,17 @@ for (const scheme of ['dark', 'light'] as const) { test.use({ colorScheme: scheme }); test('every benchmark screen passes axe', async ({ page }) => { - test.setTimeout(60_000); + test.setTimeout(90_000); const specs = await catalog(page); const rdma = specs['rdma-bw-lat']?.name ?? ''; const runs = await seededRuns(page); const done = runs.find((r) => r.bench_id === 'gpu-matmul' && r.status === 'succeeded'); const soak = runs.find((r) => r.bench_id === 'thermal-soak' && r.status === 'succeeded'); const failed = runs.find((r) => r.status === 'failed'); + const sweep = runs.find((r) => r.bench_id === 'llm-inference' && r.status === 'succeeded' && r.params.concurrency === 32); + const evaluation = runs.find((r) => r.bench_id === 'tool-eval' && r.status === 'succeeded'); const pair = runs.filter((r) => r.bench_id === 'gpu-matmul' && r.status === 'succeeded').slice(0, 2); - expect(done && soak && failed && pair.length === 2).toBeTruthy(); + expect(done && soak && failed && sweep && evaluation && pair.length === 2).toBeTruthy(); await page.goto('/benchmarks'); await catalogReady(page); @@ -206,7 +371,7 @@ for (const scheme of ['dark', 'light'] as const) { expect(await axeViolations(page), 'confirm').toEqual([]); await page.keyboard.press('Escape'); - for (const [what, id] of [['completed run', done?.id], ['soak run', soak?.id], ['failed run', failed?.id]] as const) { + for (const [what, id] of [['completed run', done?.id], ['soak run', soak?.id], ['failed run', failed?.id], ['concurrency sweep', sweep?.id], ['tool-calling evaluation', evaluation?.id]] as const) { await page.goto(`/benchmarks/runs/${id ?? ''}`); await expect(page.getByRole('region', { name: /Telemetry during the run/ })).toBeVisible({ timeout: 15_000 }); expect(await axeViolations(page), what).toEqual([]); diff --git a/web/src/dev/gallery.tsx b/web/src/dev/gallery.tsx index 0e52caf..f733541 100644 --- a/web/src/dev/gallery.tsx +++ b/web/src/dev/gallery.tsx @@ -25,6 +25,8 @@ import type { CoreInfo, Sample } from '../lib/types'; import { useCluster } from '../store/cluster'; import { useConnection } from '../store/connection'; import { liveStore } from '../store/live'; +import { StreamsGallery } from '../views/bench/streams/StreamsGallery'; +import { ToolEvalGallery } from '../views/bench/tooleval/ToolEvalGallery'; import '../views/views.css'; const GiB = 1024 ** 3; @@ -149,6 +151,8 @@ function Gallery() { { setConfirm(false); }} onConfirm={() => { setConfirm(false); }} title="Clear history on spark1?" consequences={['Deletes 7 days of stored telemetry', 'Live data keeps streaming', 'This cannot be undone']} confirmLabel="Clear history" destructive confirmPhrase="DELETE" /> + + ); } diff --git a/web/src/lib/types.ts b/web/src/lib/types.ts index a851f5f..57c3f64 100644 --- a/web/src/lib/types.ts +++ b/web/src/lib/types.ts @@ -374,6 +374,10 @@ export interface NodeBenchState { metadata: Record; error: Problem | null; log_tail: string[]; + /** What the payload is doing now (bounded snapshot, replaced on every update). Absent in older runs; validate its shape. */ + live?: Record; + /** Structured per-run detail (top-level keys merge). Absent in older runs; validate its shape. */ + detail?: Record; } /** GET /api/v1/bench/compare?ids=a,b,c */ diff --git a/web/src/views/bench/ParamForm.tsx b/web/src/views/bench/ParamForm.tsx index bf8d6fc..d7413c2 100644 --- a/web/src/views/bench/ParamForm.tsx +++ b/web/src/views/bench/ParamForm.tsx @@ -32,7 +32,13 @@ function boundsText(p: BenchParam): string | null { } const SEGMENTED_MAX = 4; -const segmentable = (p: BenchParam) => (p.choices?.length ?? 0) <= SEGMENTED_MAX && (p.choices ?? []).every((c) => choiceLabel(c, p).length <= 12); +/** Segments share the width equally, so the count times the longest label must also fit a phone-width dialog (about + * 32 characters: "16 MiB" … "1024 MiB" does, "Core 15" … "Hard Mode 28" gets a select). */ +const segmentable = (p: BenchParam) => { + const longest = Math.max(0, ...(p.choices ?? []).map((c) => choiceLabel(c, p).length)); + const count = p.choices?.length ?? 0; + return count <= SEGMENTED_MAX && longest <= 12 && count * longest <= 32; +}; export function ParamForm({ params, values, errors, onChange, skip = [] }: Props) { const shown = params.filter((p) => !skip.includes(p.name)); diff --git a/web/src/views/bench/RunView.tsx b/web/src/views/bench/RunView.tsx index 73e6fb5..8adfb66 100644 --- a/web/src/views/bench/RunView.tsx +++ b/web/src/views/bench/RunView.tsx @@ -52,6 +52,8 @@ import { BetterHint, DeltaBadge, RunStatusPill } from './parts'; import { RunTelemetry } from './RunTelemetry'; import { ShareCardDialog } from './ShareCard'; import { SoakPanel } from './SoakPanel'; +import { StreamsSection } from './streams/StreamsSection'; +import { ToolEvalSection } from './tooleval/ToolEvalSection'; export function RunView({ id }: { id: string }) { const { run, missing, gone, refetch } = useRun(id); @@ -235,10 +237,16 @@ function RunBody({ run, spec }: { run: BenchRun; spec: BenchSpec | undefined }) } onClick={runAgain}>Run again}> The benchmark was stopped after {stopwatch(elapsed(run, now))}. Its containers and temp files were removed, and partial results were discarded. + {/* The stream and scenario views below keep what was recorded; say so rather than contradict them. */} + {Object.values(run.node_states ?? {}).some((st) => Object.keys(st.live ?? {}).length || Object.keys(st.detail ?? {}).length) ? ' What it recorded before it stopped is shown below.' : null} )} + {/* Per-benchmark views lead the page for these benchmarks, so they come before the telemetry. */} + {run.bench_id === 'llm-inference' ? : null} + {run.bench_id === 'tool-eval' ? : null} + {active || run.started_ts != null ? ( 1 ? 'One line per node · dashed rules mark the run' : 'Dashed rules mark the run'}> diff --git a/web/src/views/bench/ShareCard.tsx b/web/src/views/bench/ShareCard.tsx index 801791f..4cd26a4 100644 --- a/web/src/views/bench/ShareCard.tsx +++ b/web/src/views/bench/ShareCard.tsx @@ -7,7 +7,7 @@ import { Button } from '../../design/components/Button'; import { Dialog } from '../../design/components/Dialog'; import type { BenchRun, BenchSpec } from '../../lib/types'; import { useCluster } from '../../store/cluster'; -import { formatDateTime, formatValue, headlineResults, nodesText, paramsSummary, resultDelta, runFacts, servedModel, signedFormat, unitText } from './model'; +import { formatDateTime, headlineResults, nodesText, paramsSummary, resultDelta, runFacts, servedModel, signedFormat, unitText } from './model'; export const CARD_W = 600; export const CARD_H = 340; @@ -198,7 +198,7 @@ export function ShareCardDialog({ run, spec, onClose }: { run: BenchRun; spec: B () => { setCopied('failed'); }, ); }; - const heads = headlineResults(run.results, 3).map((r) => `${r.label} ${formatValue(r.value)} ${unitText(r.unit)}`).join(', '); + const heads = headlineResults(run.results, 3).map((r) => `${r.label} ${signedFormat(r)(r.value)} ${unitText(r.unit)}`).join(', '); return ( { expect(onRun).toHaveBeenCalledWith(['spark1'], { dtypes: ['bf16', 'fp16', 'fp8'], size: '8192', iterations: 100 }); }); + it('offers choices too long for equal segments on a phone as a select', () => { + const choice = (name: string, label: string, choices: string[]) => ({ name, label, type: 'enum' as const, default: choices[0], min: null, max: null, choices, unit: null, help: null }); + const spec = { ...LLM, params: [choice('api', 'API', ['chat', 'completions']), choice('suite', 'Scenarios', ['standard', 'core', 'hard', 'all'])] }; + render( undefined} onRun={() => undefined} />); + expect(screen.getByRole('radiogroup', { name: 'API' })).toBeInTheDocument(); + expect(screen.getByRole('combobox', { name: 'Scenarios' })).toHaveDisplayValue('Standard 69'); + }); + it('picks a source and a target for two-node benches, with the rail next to them', () => { const onRun = vi.fn(); render( undefined} onRun={onRun} />); diff --git a/web/src/views/bench/model.test.ts b/web/src/views/bench/model.test.ts index 42b18e9..f45c363 100644 --- a/web/src/views/bench/model.test.ts +++ b/web/src/views/bench/model.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import type { BenchCompare, Sample } from '../../lib/types'; +import type { BenchCompare, BenchParam, BenchSpec, Sample } from '../../lib/types'; import { boundRuns, mergeRun } from '../../store/bench'; import { CATALOG, LLM, MATMUL, RDMA, SOAK, result, run } from './fixtures'; import { @@ -22,6 +22,7 @@ import { headlineResults, headlineText, isLanHost, + llmLevels, nodesText, paramText, paramsSummary, @@ -30,6 +31,7 @@ import { resultDelta, resultText, runnableNodes, + signedFormat, soakStats, splitDescription, toForm, @@ -166,6 +168,35 @@ describe('duration estimate', () => { expect(estimateDuration(RDMA, { rail: 'A', duration_s: 30 })).toBe(60); expect(estimateDuration(SOAK, { minutes: 30 })).toBe(1800); }); + + const param = (name: string, type: BenchParam['type'], d: unknown, choices: string[] | null = null): BenchParam => ({ name, label: name, type, default: d, min: null, max: null, choices, unit: null, help: null }); + it('is the agent’s own llm-inference estimate, over every level of a sweep', () => { + expect(llmLevels(32, true)).toEqual([1, 2, 4, 8, 16, 32]); + expect(llmLevels(24, true)).toEqual([1, 2, 4, 8, 16, 24]); + expect(llmLevels(1, true)).toEqual([1]); + expect(llmLevels(8, false)).toEqual([8]); + const spec: BenchSpec = { + ...LLM, + est_duration_s: 62.2, + params: [param('prompt_tokens', 'int', 512), param('max_tokens', 'int', 128), param('concurrency', 'int', 1), param('requests', 'int', 8), param('sweep', 'bool', false)], + }; + const d = defaultParams(spec); + // Expected values from server/sparkline/agent/bench/specs.py est_llm for the same parameters. + expect(estimateDuration(spec, d)).toBeCloseTo(62.248, 6); + expect(estimateDuration(spec, { ...d, prompt_tokens: 65536 })).toBeCloseTo(322.344, 6); + expect(estimateDuration(spec, { ...d, concurrency: 8 })).toBeCloseTo(12.156, 6); + expect(estimateDuration(spec, { ...d, max_tokens: 16, concurrency: 32, requests: 32 })).toBeCloseTo(6.556, 6); + expect(estimateDuration(spec, { ...d, concurrency: 32, sweep: true })).toBeCloseTo(126.652, 6); + expect(estimateDuration(spec, { ...d, concurrency: 24, requests: 16, sweep: true, max_tokens: 256 })).toBeCloseTo(438.792, 6); + }); + it('follows the tool-eval suite, Hard Mode scenarios taking longer', () => { + const suite = param('suite', 'enum', 'standard', ['core', 'standard', 'hard', 'all']); + const spec: BenchSpec = { ...LLM, id: 'tool-eval', est_duration_s: 1666, params: [param('base_url', 'string', 'http://127.0.0.1:8000'), param('model', 'string', ''), suite, param('thinking', 'bool', true)] }; + const d = defaultParams(spec); + expect(estimateDuration(spec, d)).toBe(1666); + expect(['core', 'hard', 'all'].map((s) => Math.round(estimateDuration(spec, { ...d, suite: s })))).toEqual([370, 1018, 2674]); + expect(suite.choices?.map((c) => choiceLabel(c, suite))).toEqual(['Core 15', 'Standard 69', 'Hard Mode 28', 'All 97']); + }); }); describe('progress', () => { @@ -491,4 +522,19 @@ describe('valueWithUnit matches the live screens', () => { expect(valueWithUnit(88.04, '°C')).toBe('88 °C'); expect(valueWithUnit(412.3, 'TFLOPS')).toBe('412 TFLOPS'); }); + it('shows a count (no unit) as a whole number, not "0.00"', () => { + expect(valueWithUnit(0, '')).toBe('0'); // tool-eval's safety violations + expect(valueWithUnit(2, '')).toBe('2'); + }); + it('shows a result that is whole by definition (extras.whole, a tool-eval score) without decimals', () => { + const score = result({ metric: 'score', unit: '%', value: 89, extras: { headline: true, whole: true } }); + expect(resultText(score)).toBe('89%'); + expect(headlineText([score])).toBe('89%'); + expect(signedFormat(score)(88.6)).toBe('89'); // counting up stays whole + expect(headlineText([{ ...score, node_id: 'spark1' }, { ...score, node_id: 'spark2', value: 92 }])).toBe('89 / 92 %'); + // Measured percentages keep 3 significant figures, even when they land on a whole number. + expect(resultText(result({ metric: 'tflops_change_pct', value: -12, unit: '%' }))).toBe('−12.0%'); + expect(resultText(result({ metric: 'completion_rate', value: 86.7, unit: '%' }))).toBe('86.7%'); + expect(valueWithUnit(89, '%')).toBe('89.0%'); + }); }); diff --git a/web/src/views/bench/model.ts b/web/src/views/bench/model.ts index e15e001..631d283 100644 --- a/web/src/views/bench/model.ts +++ b/web/src/views/bench/model.ts @@ -183,11 +183,42 @@ function secondsOf(value: number, p: BenchParam): number { const isDurationParam = (p: BenchParam) => /^(s|sec|seconds|min|minutes|h|hours)$/i.test(p.unit ?? '') || DURATION_NAME.test(p.name); +/** Concurrency levels of an llm-inference run, like the agent: [concurrency], or when sweeping the powers of two + * below it followed by concurrency itself (32 -> 1, 2, 4, 8, 16, 32; 24 -> 1, 2, 4, 8, 16, 24). */ +export function llmLevels(concurrency: number, sweep: boolean): number[] { + const c = Math.max(1, Math.floor(concurrency)); + const below: number[] = []; + if (sweep) for (let l = 1; l < c; l *= 2) below.push(l); + return [...below, c]; +} + +/** The agent's llm-inference estimate (est_llm), term for term: 5 s, plus for every request wave (each level runs + * max(requests, level) requests, `level` at a time) 0.5 s, the prompt at 2000 tokens/s and the output at 20 tokens/s. */ +function llmSeconds(v: Partial): number { + const n = (k: string, d: number) => { + const x = v[k]; + return typeof x === 'number' ? x : d; + }; + const requests = n('requests', 8); + const waves = llmLevels(n('concurrency', 1), v.sweep === true).reduce((sum, l) => sum + Math.ceil(Math.max(requests, l) / l), 0); + return 5 + waves * (0.5 + n('prompt_tokens', 512) / 2000 + n('max_tokens', 128) / 20); +} + +/** tool-eval-bench scenarios per suite: [standard, Hard Mode]. On a Spark a standard scenario takes about 24 s and a + * Hard Mode one about 36 s, after about 10 s of preflight. */ +const TOOL_EVAL_SUITES: Record = { core: [15, 0], standard: [69, 0], hard: [0, 28], all: [69, 28] }; +const toolEvalSeconds = (suite: ParamValue | undefined) => { + const [standard, hard] = TOOL_EVAL_SUITES[String(suite)] ?? [0, 0]; + return 10 + standard * 24 + hard * 36; +}; + /** * Live duration estimate for the chosen parameters, from the catalog's estimate for the defaults: a duration * parameter adds its difference from the default; a count parameter, or a multi-select, scales the rest. + * llm-inference uses the agent's own formula (its job timeout is based on it); tool-eval scales by its suite. */ export function estimateDuration(spec: BenchSpec, values: Partial): number { + if (spec.id === 'llm-inference') return llmSeconds({ ...defaultParams(spec), ...values }); let extra = 0; let factor = 1; for (const p of spec.params) { @@ -200,6 +231,10 @@ export function estimateDuration(spec: BenchSpec, values: Partial): factor *= Math.max(1, v.length) / Math.max(1, d.length); } } + if (spec.id === 'tool-eval') { + const suite = defaultParams(spec).suite; + factor *= toolEvalSeconds(values.suite ?? suite) / toolEvalSeconds(suite); + } return Math.max(1, (spec.est_duration_s + extra) * factor); } @@ -217,6 +252,10 @@ const CHOICE_TEXT: Record = { seq_read: 'Sequential read', rand_read: 'Random read', rand_write: 'Random write', + core: 'Core 15', + standard: 'Standard 69', + hard: 'Hard Mode 28', + all: 'All 97', }; /** 4096 -> "4 KiB", 1048576 -> "1 MiB". */ @@ -311,14 +350,26 @@ export function formatValue(v: number | null | undefined): string { /** Units as people write them: "us" -> "µs". */ export const unitText = (unit: string) => (unit === 'us' ? 'µs' : unit); -/** "412 TFLOPS", "1.92 µs". Clocks and temperatures read as on the live screens: "1.88 GHz", "88 °C". */ +/** A result that is a whole number by definition (`extras.whole`, such as a tool-eval score) reads "89%", not "89.0%". */ +const isWhole = (r: Pick) => r.extras.whole === true; +const formatWhole = (v: number) => (Math.round(v) < 0 ? `−${-Math.round(v)}` : String(Math.round(v))); + +/** "412 TFLOPS", "1.92 µs". Clocks and temperatures read as on the live screens: "1.88 GHz", "88 °C"; a count (no + * unit, a whole number, such as tool-eval's safety violations) as "0". */ export function valueWithUnit(v: number | null | undefined, unit: string): string { if (v == null || !Number.isFinite(v)) return '—'; if (unit === 'MHz' && Math.abs(v) >= 1000) return `${(v / 1000).toFixed(2)} GHz`; if (unit === '°C') return `${Math.round(v)} °C`; + if (!unit && Number.isInteger(v)) return String(v); return `${formatValue(v)}${!unit ? '' : unit === '%' ? '%' : ` ${unitText(unit)}`}`; } +/** valueWithUnit for a result, honouring `extras.whole`. */ +function resultValue(r: BenchResult): string { + if (!isWhole(r) || !Number.isFinite(r.value)) return valueWithUnit(r.value, r.unit); + return `${formatWhole(r.value)}${!r.unit ? '' : r.unit === '%' ? '%' : ` ${unitText(r.unit)}`}`; +} + export function deltaPct(value: number | null | undefined, ref: number | null | undefined): number | null { if (value == null || ref == null || !Number.isFinite(value) || !Number.isFinite(ref) || ref === 0) return null; return ((value - ref) / Math.abs(ref)) * 100; @@ -361,13 +412,17 @@ export const isUnsupported = (r: BenchResult) => r.extras.unsupported === true; /** A code-path smoke run (GPU payload on the CPU, agent bench_payload_device = "cpu"): not a GPU measurement. */ export const isSmoke = (results: readonly BenchResult[]) => results.some((r) => r.extras.smoke === true); -/** A signed change ("tflops_change_pct"): positive values carry a "+" so a gain never reads as a loss. */ -export const signedFormat = (r: Pick) => (v: number) => (/_change_pct$/.test(r.metric) && v > 0 && formatValue(v) !== '0.00' ? `+${formatValue(v)}` : formatValue(v)); +/** A signed change ("tflops_change_pct"): positive values carry a "+" so a gain never reads as a loss. A whole + * result stays whole while it counts up, so the number doesn't change width when it lands. */ +export const signedFormat = (r: Pick) => { + const format = isWhole(r) ? formatWhole : formatValue; + return (v: number) => (/_change_pct$/.test(r.metric) && v > 0 && Number(format(v)) !== 0 ? `+${format(v)}` : format(v)); +}; /** "412 TFLOPS", "+27.3%", or "Unsupported" for a result the node could not measure. */ export function resultText(r: BenchResult): string { if (isUnsupported(r)) return 'Unsupported'; - const text = valueWithUnit(r.value, r.unit); // clocks and temperatures are rewritten ("2.15 GHz", "81 °C") + const text = resultValue(r); // clocks and temperatures are rewritten ("2.15 GHz", "81 °C") return signedFormat(r)(r.value).startsWith('+') ? `+${text}` : text; } @@ -395,8 +450,8 @@ export function headlineText(all: readonly BenchResult[]): string { const same = headlineSet(all); const first = same[0]; if (!first) return '—'; - if (same.length < 2) return valueWithUnit(first.value, first.unit); - return `${same.map((r) => formatValue(r.value)).join(' / ')} ${first.unit}`; + if (same.length < 2) return resultValue(first); + return `${same.map((r) => (isWhole(r) ? formatWhole(r.value) : formatValue(r.value))).join(' / ')} ${first.unit}`; } /** What the headline number is, with its nodes in the same order: "Single core (Cortex-X925) · spark1 / spark2". */ diff --git a/web/src/views/bench/streams/LivePanel.tsx b/web/src/views/bench/streams/LivePanel.tsx new file mode 100644 index 0000000..56bca63 --- /dev/null +++ b/web/src/views/bench/streams/LivePanel.tsx @@ -0,0 +1,247 @@ +// The hero of a running llm-inference benchmark: every concurrent request as a live card, the sweep's levels, the +// aggregate output and where each stream stands. Before the first snapshot the same layout stands in as placeholders, +// so the panel never opens as an empty box. Level changes are announced once; nothing else is. +import { Check, CircleX, LoaderCircle } from 'lucide-react'; +import { useEffect, useMemo, useRef, useState, type CSSProperties, type ReactNode } from 'react'; +import { AnimatedNumber } from '../../../design/components/AnimatedNumber'; +import { announce } from '../../../design/components/LiveAnnouncer'; +import { Panel } from '../../../design/components/Panel'; +import { SegmentedControl } from '../../../design/components/SegmentedControl'; +import { Skeleton } from '../../../design/components/Skeleton'; +import { Sparkline } from '../../../design/components/Sparkline'; +import { Stat } from '../../../design/components/Stat'; +import type { Formatter } from '../../../lib/format'; +import { formatValue } from '../model'; +import { + accumulate, + cardState, + countsText, + densityFor, + formatTokS, + gridColumns, + stateCounts, + STATE_LOOK, + type CardState, + type Density, + type DensityChoice, + type LiveSnapshot, + type Stream, + type Texts, +} from './model'; +import { SkeletonCard, StreamCard, StreamDialog } from './StreamCard'; +import './streams.css'; + +const tokS = (n: number) => formatTokS(n); +const toneClass = (state: CardState) => (STATE_LOOK[state].tone === 'accent' ? 'strm-tone-accent' : `tone-${STATE_LOOK[state].tone}`); +const tokSText = (n: number) => `${formatTokS(n)} tok/s`; +const msParts: Formatter = (n, unit) => (unit === 's' || (!unit && n >= 1000) ? { value: formatValue(n / 1000), unit: 's' } : { value: formatValue(n), unit: 'ms' }); +const ofParts = (total: number): Formatter => (n) => ({ value: String(Math.round(n)), unit: `of ${total}` }); +const plural = (n: number, one: string, many: string) => `${n} ${n === 1 ? one : many}`; + +const SIZES = [ + { value: 'text', label: 'Text', ariaLabel: 'Text: cards show several lines' }, + { value: 'compact', label: 'Compact', ariaLabel: 'Compact: one line per stream' }, +] as const; + +/** `requests`: the run's request count, which every later level of a sweep runs (at least one per stream). */ +export function LivePanel({ snap, requests, actions }: { snap: LiveSnapshot; requests: number; actions?: ReactNode }) { + const [choice, setChoice] = useState('auto'); + const [open, setOpen] = useState(null); + // The text each card has seen, folded in once per snapshot (derived state: no extra render pass per update). + const [seen, setSeen] = useState<{ snap: LiveSnapshot; texts: Texts }>(() => ({ snap, texts: accumulate(null, snap) })); + if (seen.snap !== snap) setSeen({ snap, texts: accumulate(seen.texts, snap) }); + + const n = snap.streams.length; + const density = densityFor(n, choice); + const sweeping = snap.levels.length > 1; + const levelText = `Level ${snap.level_index + 1} of ${snap.levels.length}`; + + // Announce a new level once (not on first sight: the page may open mid-sweep). 'info' is filtered by the announcer. + const announced = useRef(snap.level_index); + useEffect(() => { + if (!sweeping || announced.current === snap.level_index) return; + announced.current = snap.level_index; + announce(`${levelText}: ${plural(snap.level, 'stream', 'streams')}`, 'warning'); + }, [sweeping, snap.level_index, snap.level, levelText]); + + const opened = open === null ? undefined : snap.streams.find((s) => s.slot === open); + return ( + + {actions} + + + } + className="strm-live" + > +
+ {sweeping ? : null} + + + {snap.streams.map((s) => ( + + ))} + +
+ {opened ? ( + { setOpen(null); }} + /> + ) : null} + + ); +} + +export function Grid({ density, count, label, children }: { density: Density; count: number; label?: string; children: ReactNode }) { + return ( +
+ {children} +
+ ); +} + +/** Every level of the sweep on one rail: finished ones with a check and their output (or how many failed, when + * none got through), the current one with its progress, the rest quiet. Before the first snapshot (`snap` missing) + * every level is still to come. */ +function LevelRail({ levels, current, requests, snap }: { levels: readonly number[]; current: number; requests: number; snap?: LiveSnapshot }) { + const finished = new Map((snap?.completed_levels ?? []).map((l) => [l.level, l])); + return ( +
    + {levels.map((level, i) => { + const state = snap && i < current ? 'done' : snap && i === current ? 'current' : 'next'; + const done = state === 'done' ? finished.get(level) : undefined; + const failed = done?.ok === 0 ? (done.failed ?? done.requests ?? 0) : 0; // every request of the level failed + const fill = state === 'done' ? 1 : state === 'current' && snap?.requests ? snap.done / snap.requests : 0; + return ( +
  1. + + + + {failed ? 'Failed: ' : state === 'done' ? 'Finished: ' : state === 'current' ? 'Running: ' : 'To come: '} + {plural(level, 'stream', 'streams')} + + + + {failed + ? `${failed} failed` + : state === 'done' + ? done?.out_tok_s != null ? `${formatTokS(done.out_tok_s)} tok/s` : 'Done' + : state === 'current' && snap ? `${snap.done} of ${snap.requests}` : plural(Math.max(requests, level), 'request', 'requests')} + +
    2. + ); + })} +
+ ); +} + +/** Where each slot stands, one dot per stream in slot order, with the counts spelled out under it. */ +function DotMatrix({ streams }: { streams: readonly Stream[] }) { + const counts = stateCounts(streams); + const text = countsText(counts); + return ( + + 16 ? Math.ceil(streams.length / 2) : Math.max(8, streams.length) } as CSSProperties}> + {streams.map((s) => )} + + + + ); +} + +function StatRow({ snap }: { snap: LiveSnapshot }) { + const counts = stateCounts(snap.streams); + const active = counts.wait + counts.decode; + const total = snap.streams.length; + const activeParts = useMemo(() => ofParts(total), [total]); + const requestParts = useMemo(() => ofParts(snap.requests), [snap.requests]); + const hist = snap.tok_s_hist; + const peak = Math.max(0, snap.tok_s ?? 0, ...hist.map((v) => v ?? 0)); + const sweeping = snap.levels.length > 1; + return ( +
+
+
+ Aggregate output + + + tok/s + + All streams over the last 2 s{peak > 0 ? ` · peak ${formatTokS(peak)}` : ''} +
+
+ {hist.length > 1 ? : null} +
+
+ } /> + +
+ ); +} + +/** Before the first snapshot: the panel as it will look, with the planned levels and one placeholder per stream. */ +export function LiveSkeleton({ endpoint, levels, requests }: { endpoint: string; levels: readonly number[]; requests: number }) { + const count = levels[0] ?? 1; + const density = densityFor(count, 'auto'); + return ( + + + } + className="strm-live" + busy + > +
+ {levels.length > 1 ? : null} + + + {Array.from({ length: count }, (_, i) => )} + +
+ + ); +} diff --git a/web/src/views/bench/streams/StreamCard.tsx b/web/src/views/bench/streams/StreamCard.tsx new file mode 100644 index 0000000..8017619 --- /dev/null +++ b/web/src/views/bench/streams/StreamCard.tsx @@ -0,0 +1,266 @@ +// One concurrent request as a card: slot, state, live tokens/s, the text it is writing (reasoning muted above the +// answer, newest line at the bottom), and tokens against the output limit. A card is a button: it opens the stream +// large in a dialog that stays live. Text types itself (typewriter.ts); everything else re-renders with the snapshot. +import { CircleX } from 'lucide-react'; +import type { CSSProperties } from 'react'; +import { AnimatedNumber } from '../../../design/components/AnimatedNumber'; +import { Dialog } from '../../../design/components/Dialog'; +import { Skeleton } from '../../../design/components/Skeleton'; +import '../../../design/components/StatusPill.css'; +import { formatMs, formatTokS, slotLabel, STATE_LOOK, streamSummary, type CardState, type Density, type SlotText, type Stream } from './model'; +import { useCountUp, useTypewriter } from './typewriter'; + +const rateText = (n: number) => formatTokS(n); +const waitText = (ms: number) => formatMs(ms); + +/** State as icon + word in its tone (the accent for decoding, which is activity rather than status). */ +export function StatePill({ state, iconOnly }: { state: CardState; iconOnly?: boolean }) { + const look = STATE_LOOK[state]; + const Icon = look.icon; + return ( + + + ); +} + +function Typed({ text, added, id, live, className }: { text: string; added: number; id: string; live: boolean; className?: string }) { + const ref = useTypewriter(text, added, id, live); + return ; +} + +function WaitCounter({ ms, id }: { ms: number | null; id: string }) { + const ref = useCountUp(ms, id, waitText); + return ; +} + +interface TextProps { + stream: Stream; + text: SlotText | undefined; + state: CardState; + live: boolean; +} + +/** Reasoning (muted, under a small label) then the answer, with a caret after whichever is being written. */ +function StreamText({ stream: s, text, state, live, oneLine }: TextProps & { oneLine?: boolean }) { + const id = text?.key ?? `${s.slot}:${s.req}`; + const think = text?.think ?? s.think; + const answer = text?.answer ?? s.text; + const thinking = !answer && !!think; + const caret = state === 'decode' && live ?