diff --git a/CHANGELOG.md b/CHANGELOG.md index 43cb27b..73eedda 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,325 @@ All notable changes to Local Model Bench are recorded here. The format follows Dates are the date the version was prepared. +## 1.15.0 — 2026-09-17 + +### Changed + +- **Flash attention is on by default, and a run states it rather than inheriting it.** + It was only ever turned on as a side effect of quantizing the cache, so every other + run took whatever LM Studio happened to have set — which is not a neutral choice. + Measured on Gemma 4 26B A4B at f16 cache, changing nothing else: + + | Engine | Flash | Prompt processing | Fixed cost per request | + | --- | --- | --- | --- | + | ROCm | off | 260 tok/s | 7.1 s | + | ROCm | on | 1281 tok/s | 0.06 s | + | Vulkan | off | 277 tok/s | 9.2 s | + | Vulkan | on | 1822 tok/s | 0.10 s | + + That is about five times the prompt processing and the difference between seconds and + milliseconds before a first token, on both engines. It also explains an earlier + reading of these runs: ROCm looked slower than Vulkan only because every fast ROCm run + happened to have quantized its cache, which forced flash attention on. + + **Use flash attention** on the Run screen is on unless turned off, written to LM + Studio's per-model configuration for each load and restored afterwards, and checked + against what LM Studio reports before anything is measured. Turning it off is a + supported choice — it is the only way to measure the cost — and the run warns what it + will do. A quantized cache with flash attention off is refused up front, because + llama.cpp cannot do it and LM Studio's refusal names neither setting. + + Runs saved before this leave the field unset and were measured under whatever LM + Studio had at the time, so they are not evidence either way. + +## 1.14.0 — 2026-09-17 + +### Added + +- **The server does not have to be this machine.** The endpoint was restricted to + loopback, so LM Studio on another PC, a box on the LAN, or anything else speaking the + same API could not be benchmarked from here. Any http:// or https:// address is now + accepted. What is still refused is anything that is not an origin: credentials in the + URL, which belong in the API token field where they are encrypted at rest and never + reach the window, and a path, query or fragment, which would change what the address + means. + + Pointing somewhere else changes what this app can honestly claim, so it says so rather + than leaving the old promise on screen. The sidebar reads **Remote endpoint** with the + host instead of *Your prompts stay on this PC*, and the settings field warns that + prompts, responses and any token will leave the machine — noting when a plain http + address means they travel unencrypted. The badge follows the saved setting, not what is + being typed, so it describes where runs actually go. + +## 1.13.0 — 2026-09-17 + +### Changed + +- **The three starter JSON tests now accept a fenced answer.** subnet, facts and + json-extract ask for "only JSON" and say nothing about code fences, so a model that + wraps correct JSON in ```json was being scored zero for a markdown habit the prompt + never mentioned. Measured on Gemma 4 26B A4B: every subnet value correct, scored 0; + every extraction value correct, scored 20. Those three tests set `allowCodeFence`, + and already-saved copies are migrated once on startup, so existing installs pick the + change up rather than only new ones. A test you have edited yourself keeps whatever + rules it has. + + The accounting pack is deliberately excluded: its prompt says "no prose or code + fences", which makes the fence part of what it tests, and it still fails one. So does + any test that has not opted in, along with a fence that never closes, several fenced + blocks, and prose around the JSON. + + This changes what those three tests score. Runs saved before it keep the numbers they + were given, so a quality comparison that spans the change is comparing two scoring + rules as well as two models. + +## 1.12.1 — 2026-09-17 + +### Fixed + +- **A correct JSON answer inside a code fence scored zero.** Models asked for "only + JSON" very often return it wrapped in a ```json fence. That is markup around the + answer rather than prose instead of it, and this app already unwrapped exactly the + same wrapper when reading a judge's reply — but not when scoring the model being + tested. Measured on Gemma 4 26B A4B: all five subnet values correct, fenced, every + check failing with "Unexpected token '`'", for a score of 0; and the extraction test + scored 20 with all four values right. The JSON-shaped checks now read the answer with + a single complete surrounding fence removed, and say so in the check detail rather + than unwrapping silently. A fence that never closes, a reply holding several fenced + blocks, and prose around the JSON are all still failures, because those are different + things to get wrong. Every other check still sees exactly what the model wrote. + +## 1.12.0 — 2026-09-17 + +### Added + +- **Off / text-only can now actually load without the projector.** The toggle only + ever controlled whether an image was *sent*; the projector loaded regardless, because + LM Studio attaches one from the model's index entry and from nowhere else. Rechecked + against LM Studio 2.40.0: none of the 24 `llm.load.*` keys in its bundle mentions + vision, and its llama-server carries no `--mmproj` flag at all, since it drives the + engine through its own bindings rather than a command line. So the toggle now drives + the one thing that does work — the text-only copy this app already builds. With + **Off / text-only** selected, any chosen model that would still load a projector is + named, with one button to select the text-only copies that exist and another to make + the ones that do not. Copies are hard links and cost no disk. A model with no + projector is left alone, and a run that cannot be made fully text-only says so + rather than implying otherwise. + +### Fixed + +- **Calibrated prompt processing was blank on every sweep.** The engine stored each + calibration under one key shape and the results screen looked it up under another, + so the column existed and never resolved — on exactly the runs it was built for. Both + sides now use one `calibrationKey()`, which also drops a doubled prefix that spelled + depth 0 as "MTP MTP off". Calibrations saved under the old shape are still read, so + finished runs keep their measurements. + +## 1.11.1 — 2026-09-17 + +### Fixed + +- **The depth axis existed but nothing opened on it.** 1.11.0 added *Read across* and + then defaulted it to concurrent requests, so a prediction-depth sweep still opened + showing every depth stacked at one horizontal position — the control was there, but + the graphs looked exactly as they had before and the feature was invisible unless + you went looking for it. The graphs now open on whichever dimension the run actually + varied: a run that swept depths at a single concurrency level opens on maximum + predictions, and anything that varied concurrency still opens on concurrency. + Choosing an axis by hand continues to override it. + +## 1.11.0 — 2026-09-17 + +### Added + +- **The graphs can read across MTP depth.** They always plotted against concurrent + requests, which is the right axis for a concurrency sweep and the wrong one for a + prediction-depth sweep: a sweep run at a single concurrency level put every depth at + the same horizontal position, so six depths drew six dots stacked on one vertical + line and the only trace of the depth was a legend entry. **Read across** on the + Automatic graphs now switches the horizontal axis to maximum predictions, turning + each model into one curve over its depths. It appears only when a run measured more + than one depth, and depth stops being folded into the series name while it is the + axis, so a model is one line rather than one flat point per depth. The concurrency + axis is unchanged for every other run. + +## 1.10.1 — 2026-09-17 + +### Fixed + +- **Quantizing the context made every load fail.** llama.cpp cannot use a quantized + key/value cache without flash attention, and LM Studio refuses the load outright + rather than falling back: *"V Cache Quantization requires flash attention to be + enabled."* 1.10.0 wrote the cache fields and assumed flash attention was already on, + which is not dependable — it varies by engine and by whatever the model was last + loaded with. A run that asks for a quantized cache now writes the flag that makes it + possible, into the same per-model configuration, for the same single load, and + restores it with the rest. A run that leaves the cache alone does not touch the + setting at all. A load that comes back reporting no flash attention is refused + before anything is measured, rather than reporting a cache that was not in use. + +## 1.10.0 — 2026-09-17 + +### Added + +- **Choose which llama.cpp build a run uses.** LM Studio keeps one selected engine + at a time and has no per-load switch, so which build produced a set of numbers was + not recorded anywhere and could not be recovered afterwards. A run can now name an + engine; it is selected before the first load and the previous choice is put back + when the run ends, whatever happens to the run. On an RX 9070 the difference is not + subtle: measured on Gemma 4 12B Q4_K_XL, Vulkan 2.40.0 reached 1027 tok/s of prompt + processing against ROCm 2.40.0's 148 tok/s, and ROCm carried roughly 16 s of fixed + cost before its first token at every prompt size tested. +- **Quantize the context.** The key/value cache is the part of a run's memory that + grows with concurrency rather than with the weights: an instance holds context × + parallel slots of it. K and V can now be set independently to q8_0, q5_0, q4_0, + iq4_nl or f16. LM Studio exposes no flag for this, so the setting is written to its + own per-model configuration for the length of one load and the file is put back + byte for byte afterwards, the same discipline the MTP head already used. Measured + cost on Gemma 4 12B Q4_K_XL: q8_0 for both cost about 9% of prompt processing + (569 tok/s to 519 tok/s). +- **Results read one question at a time.** The comparison table is offered under + three headings — all measurements, prompt processing, and token generation — each + carrying the columns that bear on that question. Generation gains total throughput + beside the per-request rate, the MTP depth, drafted-token acceptance, and what each + extra concurrent user costs every other one. +- **Set another engine's numbers beside this run's.** Where saved runs measured the + same models on a different engine, the results screen offers Show ROCm, Show Vulkan + or Show all engines, and the table gains an engine column. The graphs gain a + Compare with menu listing other engines and other models, drawn as their own lines. + Nothing is recalculated: every overlaid point is a saved row, newest run per + measurement. The engine is also a filter and a grouping in the history overview. + +### Fixed + +- **Prompt processing was reported at roughly a third of its real speed.** The figure + came from LM Studio's prompt_processing interval, which in practice lands within a + millisecond or two of time-to-first-token, so it carried the whole fixed cost of a + request — HTTP, tokenization, scheduling, the first sampling step — on top of the + prefill. At the prompt sizes the performance tests use, that fixed cost *was* the + measurement. No single request can separate a fixed cost from a per-token one, so + each loaded model is now measured twice, over a short prompt and a long one; the + fixed cost is identical in both and cancels, leaving the real per-token rate and + naming the overhead that was being charged to the GPU. The older per-request figure + is kept beside it rather than silently replaced. +- **A load that ran out of room named none of the settings that caused it.** LM Studio + reports it in its own words, which never mention context, concurrency or the cache. + Failures that look like memory now carry the arithmetic the run already did — the + tokens actually asked for, the weights, LM Studio's own estimate — and the three + settings that change it. + +## 1.9.0 — 2026-09-15 + +### Added + +- **An optional preflight for the MTP head.** LM Studio's metadata confirming that + a model's prediction heads exist and match does not mean its runtime can load + them against that exact file. **Check the MTP head loads first** on the Run + screen loads each selected model once with MTP on, at the run's own parallel + slots and context, before anything is measured. A model whose head will not load + is named and its MTP depths are skipped; its MTP-off baseline still runs. Off by + default: it costs one extra load per model whose head does load, and what it buys + is finding out early rather than a shorter run. Measured on a two-model, two-depth + sweep against a head that will not load: reported at 25 s with the check and 82 s + without, with identical measurements either way. + +### Fixed + +- **A failed `lms` command was unreadable.** The CLI draws an animated progress bar, + and on failure the whole stream — cursor codes, colour codes and one repaint frame + per percent — became the error message. Saved run logs held up to 11,195 characters + for a single failure, with LM Studio's own one-line cause buried inside. The + progress frames and escape codes are now stripped and the duplicated failure block + collapsed, turning that example into 353 readable characters. +- **A run that lost one model said only how many things failed.** The summary now + names each model and depth that went unmeasured, and states that everything else + in the run was measured and saved. +- **Native MTP for models whose prediction heads are a separate file.** LM Studio + ships MTP two ways: heads built into the model's own GGUF, and heads stored with + it as their own file that LM Studio indexes separately. Only the first had a + command-line switch, so Gemma 4 — which ships the second kind — was reported as + having no native MTP at all and could not be benchmarked with it. A model is now + paired with a head in its own folder by structure alone: prediction layers + declared, output width equal to that model's hidden size, and that model's + architecture under LM Studio's "-assistant" suffix. Two heads that both fit are + reported as ambiguous rather than guessed between. +- **Loading a separate head.** LM Studio exposes the setting in neither its CLI nor + its HTTP API, so it is written into LM Studio's own per-model configuration for + the length of one load and that file is restored byte for byte immediately after, + whether the load succeeded or failed; a file this app created is deleted again + along with any folders it had to make. +- **Confirming a separate head.** LM Studio reports nothing about one through its + API, so a load is confirmed against its engine log, which has to name that exact + head file before anything is measured. A sweep's MTP-off baseline is checked the + same way in reverse: a head loaded for the baseline abandons that depth rather + than letting every other depth be compared against a speculative run. +- **Text-only copies of vision models.** A vision projector belongs to the model entry + LM Studio indexed, not to the load: its whole projector handling reads `mmproj_path` + straight off that entry, with no flag, no API field and no load-config key in front of + it. So Vision off could only ever mean "no image was sent", never "the projector was + left out" — and a model like Gemma 4, whose projector sits in its own folder, could not + be measured without it. The Models screen gains a **Vision projectors** panel that makes + a second, projector-free entry for LM Studio to index: a sibling folder of hard links to + the same weights, with the projector left out and any MTP head linked in so the copy + keeps native MTP. Nothing is duplicated — the copy shares the original's bytes and uses + no extra disk space — and removing it deletes links only, refusing outright if the folder + holds any file that exists nowhere else. +- **Accepted-draft rate.** Every wave run with MTP on now records how much of the drafting + the main model actually kept: the share of drafted tokens accepted, and the mean accepted + run per drafting step. LM Studio publishes neither through its API, so both are read from + the per-request lines its engine prints, pooled over the wave by drafted tokens rather than + averaged across requests. They appear in the sweep panel, the Markdown export, and as + `draftAcceptance` and `draftMeanLen` columns in CSV. MTP off reports no rate at all, which + is not the same as a rate of zero. +- **Concurrency is swept alongside depth.** A sweep could always be given several concurrency + levels, but the results pooled them into one row per depth. Each level is now measured and + reported on its own, and a depth is only ever compared with another depth under the same + load — the depth that is fastest at one request at a time need not be the fastest with five + in flight, and pooling hid exactly that. +- **MTP speed sweep button** on the Run screen: one click sets the run to speed + only, turns MTP on, and sweeps depths 0 through 5 — every draft depth from 1 to 5 + against the MTP-off baseline. +- Model cards carry an MTP badge saying which of the two shapes a model has, or why + its support is unknown. +- **Maximum predictions sweep.** With native MTP on, a run can measure several + prediction depths instead of one. Depth is fixed when a model is loaded, so each + depth is a separate load and a separate pass over the whole workload; the run + preview says how much larger that makes the run before it is started. Depth 0 + means MTP off, so the sweep can include the baseline a speed-up has to beat. +- Results gain a **Maximum predictions sweep** panel: a row per depth with speed, + throughput, latency, time to first token and objective score, the spread behind + each average, and a sentence naming the fastest depth and what it gained. A gap + no larger than the spread of the measurements is reported as exactly that rather + than as a winner, including when the baseline is the one marginally ahead. +- Markdown exports gain a **Native MTP sweep** section, and CSV exports an + `mtpDepth` column. +- The run history overview gains an **MTP depth** condition, so pooling across + runs separates depth 2 from depth 4 instead of merging both under "on". + +### Changed + +- Every saved response and wave records the MTP depth it was measured at. Summary + rows, the comparison table, the automatic graphs, benchmark pack scores and the + history overview are all keyed on it, so two depths are never averaged together. + Runs saved before this change carry no depth and are grouped exactly as before. +- A retry re-runs each failed request at the depth it was originally measured at, + and does not load a depth that has nothing to retry. + +### Fixed + +- The version shown in the window was typed into the markup by hand and had stopped + matching the installer: 1.9.0 was packaged and installed while its own sidebar + still read "1.8.0". It now comes from `package.json` at build time, so the number + in the window and the number in the installer's name cannot disagree again. +- A sweep announced its own baseline as "MTP MTP off" while loading and while running + each wave: the progress line prefixed "MTP " to a label that already began with it. +- The Playwright QA scripts under `scripts/` waited for a window heading that was + renamed in 1.8.0, so every one of them timed out on its first assertion. They + now wait for the heading that exists. +- `electron/validation.ts` held two literal control bytes inside a character + class, which made git treat the whole file as binary and its diffs unreadable. + They are now written as escapes. The token check behaves identically. + ## 1.8.0 — 2026-09-14 First version prepared for public release. Everything below is measured against diff --git a/README.md b/README.md index fda1760..a33f60f 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ AI provider. - [Step 3: Read the results](#step-3-read-the-results) - [If something looks wrong](#if-something-looks-wrong) - [Build from source](#build-from-source) — only if you want to compile it yourself +- [Choose the engine, and compare engines](#choose-the-engine-and-compare-engines-1100) — which llama.cpp build, and setting two side by side +- [Quantize the context](#quantize-the-context-1100) — the memory that grows with concurrency Everything after that is reference material. You do not need it to get going. @@ -61,7 +63,7 @@ account, and nothing here needs Python or a terminal. 1. Go to the [Releases page](https://github.com/M007-Net/local-model-bench/releases) and - download the file named `Local-Model-Bench-Setup-1.8.0.exe`. + download the file named `Local-Model-Bench-Setup-1.9.0.exe`. > If that page is empty or you cannot open it, no release has been published > yet. Until one is, the only way to get the app is @@ -73,7 +75,7 @@ account, and nothing here needs Python or a terminal. Right-click the Start button, choose **Terminal**, and paste this in: ```powershell - Get-FileHash -Algorithm SHA256 "$HOME\Downloads\Local-Model-Bench-Setup-1.8.0.exe" + Get-FileHash -Algorithm SHA256 "$HOME\Downloads\Local-Model-Bench-Setup-1.9.0.exe" ``` It prints a long string of letters and numbers. It must match the one listed on @@ -167,8 +169,8 @@ There is no built-in model list or model-family allowlist. The app discovers eve The current backend is **LM Studio native v1**, with its CLI used to load models and verify context and parallel settings. This is a Windows desktop app, not a universal API client: Ollama, cloud providers, embeddings, image generation, and standalone Hugging Face checkpoints are not implemented backends. Vision-capable language models can take the text tests; these benchmarks do not send images. - **Reasoning:** the menu shows the intersection of settings reported by the selected models. Model default sends no explicit reasoning setting and works with models that expose no reasoning control. Unsupported selections are blocked, not silently changed. -- **Native MTP:** optional and never inferred from a model name. On requires confirmed metadata for the exact model file and verified loaded settings. It currently uses LM Studio's local GGUF metadata cache; missing or stale metadata leaves support unknown. MTP requires an LM Studio/CLI runtime exposing the MTP flags and effective settings. Older runtimes may need an update even for a controlled MTP-off run. -- **Vision mode:** `Auto` leaves LM Studio's defaults alone, `Off / text-only` sends no image, and `On` runs a deterministic local image smoke test alongside your text tests. `On` is offered only for models LM Studio positively reports as vision-capable; an unknown capability is never treated as support, and a run stops before measuring anything if the loaded model does not confirm it. LM Studio has **no** load-time option to attach or detach a vision projector — the projector belongs to the model entry itself (an `mmproj` file stored beside the GGUF), so text-only means no image is sent and never claims vision weights were unloaded. A vision-capable model stays fully usable for ordinary text benchmarks, so a separate non-vision copy of the same model is not needed. Benchmark images are generated locally and sent as base64 data URLs. This app refuses any image that is not a local `data:image/` URL, so no image request ever leaves the machine. +- **Native MTP:** optional and never inferred from a model name, and covering both prediction heads built into a model file and heads shipped as a separate file with it. Several prediction depths can be measured in one run; see [Find the best MTP setting](#find-the-best-mtp-setting-190). On requires confirmed metadata for the exact model file and verified loaded settings. It currently uses LM Studio's local GGUF metadata cache; missing or stale metadata leaves support unknown. MTP requires an LM Studio/CLI runtime exposing the MTP flags and effective settings. Older runtimes may need an update even for a controlled MTP-off run. +- **Vision mode:** `Auto` leaves LM Studio's defaults alone, `Off / text-only` sends no image, and `On` runs a deterministic local image smoke test alongside your text tests. `On` is offered only for models LM Studio positively reports as vision-capable; an unknown capability is never treated as support, and a run stops before measuring anything if the loaded model does not confirm it. LM Studio has **no** load-time option to attach or detach a vision projector — the projector belongs to the model entry itself (an `mmproj` file stored beside the GGUF), so text-only means no image is sent and never claims vision weights were unloaded. A vision-capable model stays fully usable for ordinary text benchmarks. When you want to measure the same weights with the projector left out entirely, see [Benchmark a vision model without its projector](#benchmark-a-vision-model-without-its-projector). Benchmark images are generated locally and sent as base64 data URLs. This app refuses any image that is not a local `data:image/` URL, so no image request ever leaves the machine. - **Context and concurrency:** the app checks advertised context limits when available, then verifies actual loaded settings. Hardware limits or unsupported load options produce an explicit error rather than silently changing the experiment. - **Missing metadata:** models remain usable; unknown size, quantization, and capabilities are not invented. Display-only architecture/size hints can be corrected in Results and never determine whether a model may run. @@ -208,6 +210,193 @@ Search by model name, key, quantization, publisher, architecture, or format. Com **Select shown** and **Deselect shown** apply only to visible cards. Hidden selections are preserved and counted above the library. **Clear selection** removes all selections. Cards identify vision support directly. +## Benchmark a vision model without its projector + +Some models keep a vision projector (an `mmproj-*.gguf`) in the same folder as +their weights. Gemma 4 does. LM Studio attaches that projector whenever it loads +the model, and there is no way to ask it not to: its whole projector handling is +two lines that read the path off the model entry LM Studio indexed, with no flag, +no API field and no setting in front of them. **Vision off** in a run therefore +means "no image was sent", never "the projector was left out" — the weights are +still loaded and still taking VRAM. + +The only thing that changes it is giving LM Studio a *second entry* to index, with +the same weights and no projector. That is what the **Vision projectors** panel on +the Models screen does. **Make text-only copy** creates a folder beside the +model's own named `… - Text only`, containing: + +- a hard link to the model's weights — not a copy. It is the same bytes on disk + and uses no extra space; +- a hard link to any MTP head stored with the model, at the same relative place, + so the text-only copy keeps native MTP; +- no projector. + +LM Studio picks the folder up on its own and lists it as a separate model whose +vision capability is `false`. Refresh the library and it appears beside the +original, so you can select either. + +**Remove copy** deletes the folder. Because every file in it is a link, that frees +no space and destroys nothing: the weights stay where they were. Removal refuses +outright if the folder holds any file that exists nowhere else, and it will only +delete folders this app made, identified by a marker file rather than by name. + +Hard links need the model folder to be on a single NTFS volume, which a sibling +folder inside the same models directory always is. On anything else the copy is +refused with a reason rather than silently duplicating many gigabytes. + +## Find the best MTP setting (1.9.0) + +Native MTP lets a model draft several tokens ahead with its own prediction heads, +and the main model then checks that draft. Drafting further ahead can be faster — +or slower, once too many drafted tokens get thrown away. The right number is a +property of the model and your hardware, so the only way to know it is to measure. + +The quickest way in is the **MTP speed sweep** button on the **Run** screen. It +switches the run to speed only, turns MTP on, and sets the depths to +`0, 1, 2, 3, 4, 5` — every draft depth from 1 to 5, measured against MTP off. + +To set it up by hand instead, turn on **Enable native MTP** and tick **Sweep +maximum predictions**. **Depth 0 means MTP off**, so the sweep includes the +baseline that any speed-up has to beat. + +Depth is fixed when a model is loaded, so **each depth is a separate load and a +separate pass over your whole workload**. Six depths is six times the run. The +run preview says so, and shows the total before you start. Start small: one test, +concurrency 1, two waves, three depths. + +Concurrency and depth are swept together. Give the run more than one +**Concurrent requests** level and every level is measured at every depth and +reported on its own: five depths at five concurrency levels is 25 combinations per +model. A depth is only ever compared with another depth under the same load, +because the depth that is fastest at one request at a time need not be the fastest +with five in flight. + +Results gain a **Maximum predictions sweep** panel, one block per model and one +table per concurrency level: + +- a row per depth with generation speed, total throughput, latency, time to first + token and objective score; +- **draft accepted**, the share of drafted tokens the main model kept, and **mean + accepted run**, how many it kept per drafting step. Both are LM Studio's own + figures, printed by its engine as each request finishes and pooled over the wave + by drafted tokens. MTP off reports neither, because nothing was drafted — which + is not the same as drafting that was always rejected; +- the spread of the measurements behind each speed; +- a plain sentence naming the fastest depth and what it gained over the baseline. + +When the gap between two depths is no bigger than the spread of the measurements +themselves, the panel says so instead of declaring a winner. Add waves and run it +again rather than acting on that. The spread assumes requests are independent, +which requests sharing a concurrent wave are not, so treat it as a floor on the +real uncertainty. + +Every response records the depth it was measured at. Graphs draw one line per +model **and** depth, the comparison table and CSV keep `mtpDepth`, `draftAcceptance` and `draftMeanLen` columns, and +the run history overview will not pool two depths into one row. + +Sweeping needs confirmed MTP support for every selected model — the same +requirement as turning MTP on at all, checked before anything is loaded. If LM +Studio does not apply a depth it was asked for, that depth is abandoned without +measuring anything, and the rest of the sweep continues. + +**Check the MTP head loads first** (optional) goes one step further. Confirmed +support means LM Studio's metadata says the heads exist and match; it does not +mean its runtime can actually load them against that exact model file. Tick this +and every selected model is loaded once with MTP on — at the same parallel slots +and context the run itself will use — before anything is measured. A model whose +head will not load is named in the run log and its MTP depths are skipped; the +rest of that model's sweep, including the MTP-off baseline, still runs. + +It is off by default because it costs one extra load per model whose head does +load. **What it buys is when you find out, not a shorter run.** Measured on a +two-model, two-depth sweep where the second model's head will not load: the run +said so at 25 seconds with the check on and at 82 seconds without it, and the +measurements were identical either way. The whole run was 93 seconds with the +check and 82 seconds without — the check made it slightly *longer*, because the +model whose head did load paid for an extra load and there was only one depth to +skip. On a real workload that gap inverts: the depth that cannot load is found +before the baseline is measured rather than after, so you can stop instead of +waiting out a comparison that has nothing to compare against. Sweeps with several +drafting depths also fail once here instead of once per depth. + +### Models whose prediction heads are a separate file + +Some models keep their MTP heads inside the model file; others ship them as their +own file stored with the model, which LM Studio indexes separately. Gemma 4 does +the second, Qwen3.8 the first. Both count as native MTP here, and a model card +says which one it has. + +The difference matters because LM Studio offers no command-line switch for the +separate kind and no field for it in its HTTP API — its own window is the only +place the setting exists. So for those models this app writes the setting into +LM Studio's own per-model configuration immediately before a load and restores +that file, byte for byte, immediately after, whether the load worked or not. A +file it had to create is deleted again. Nothing is left behind for LM Studio to +apply to a load this app did not make. + +Because LM Studio never reports that head back through its API, the load is +confirmed against its engine log instead: the log has to name that exact head +file, or nothing is measured at that depth. The MTP-off baseline is checked the +same way in reverse — if a head was loaded for it, the depth is abandoned rather +than used as a baseline it is not. + +If two heads in a model's folder both fit it, support is reported as unknown and +the card names both. Removing the duplicate in LM Studio is the fix; this app +will not pick one for you. + +## Choose the engine, and compare engines (1.10.0) + +LM Studio ships several llama.cpp builds — a Vulkan one, a ROCm one on AMD, a CUDA +one on NVIDIA — and uses whichever is selected when a model loads. There is no +per-load switch, so until now which build produced a set of numbers was not recorded +anywhere and could not be recovered afterwards. + +**Inference engine** on the Run screen names one. It is selected before the first +load and your previous choice is put back when the run ends, whatever happens to the +run. Leave it on *Whatever LM Studio has selected* and nothing changes. + +The difference can be large and is worth measuring rather than assuming. On an +RX 9070 with Gemma 4 12B Q4_K_XL: + +| | Vulkan 2.40.0 | ROCm 2.40.0 | +| --- | --- | --- | +| Prompt processing | 1027 tok/s | 148 tok/s | +| Fixed cost before the first token | none measurable | about 16 s | +| Generation | 61 tok/s | 51 tok/s | + +Once two engines have measured the same model, **Results** offers *Show ROCm*, +*Show Vulkan* or *Show all engines* above the tables, and the comparison table gains +an engine column. The graphs gain a **Compare with** menu in the corner listing other +engines and other models; each becomes its own line. Nothing is recalculated — every +overlaid point is a saved row from a finished run, newest run per measurement — and +the other conditions of those runs are whatever they were, so read the engine column +together with the rest of the row. The engine is also a filter and a grouping on the +all-runs overview. + +## Quantize the context (1.10.0) + +The key/value cache is the part of a run's memory that grows with concurrency rather +than with the weights: an instance is loaded holding context × parallel slots of it. +On a card where the weights already nearly fill VRAM, that is usually what decides +whether a run measures the GPU or measures a spill into system RAM. + +**Quantize the context (KV cache)** sets the K and V caches independently to q8_0, +q5_0, q4_0, iq4_nl or f16. K tolerates quantization less well than V, so q8_0 for K +with something smaller for V is the usual choice. llama.cpp cannot use a quantized +cache without flash attention and LM Studio refuses the load outright rather than +falling back, so a run that asks for one turns flash attention on for each load it +makes, in the same file and with the same restore. + +LM Studio exposes no command-line flag for this, so the setting is written into its +own per-model configuration for the length of one load and the file is put back byte +for byte immediately afterwards — the same discipline the separate MTP head uses. +Nothing is left behind to change a load you make from LM Studio's own window later. + +It is not free: measured on Gemma 4 12B Q4_K_XL, q8_0 for both cost about 9% of +prompt processing (569 tok/s to 519 tok/s). Quantizing the cache also changes what +the model attends to, so treat quality scores from a quantized-cache run as measured +under that setting rather than as comparable to an f16 run. + ## Understanding the measurements - **Generation / request:** LM Studio's reported generation tokens per second for each request. Reasoning tokens are included where the server reports them. @@ -297,7 +486,7 @@ optional LM Studio API token is encrypted at rest and is never sent to the renderer: the window is told only whether one is configured. Inference scheduling runs in a worker thread. -As of 1.8.0 the suite is 164 tests, covering streaming fragments, timing math, +As of 1.9.0 the suite is 199 tests, covering streaming fragments, timing math, concurrency, scoring, cancellation, loading errors, retries, SQLite recovery, the pooling and facet rules behind the run history overview, the update check's version, asset, host and checksum rules, the question-file reader and the scoring rules an imported pack @@ -313,7 +502,7 @@ Because it is unsigned, check the installer's SHA-256 against the value publishe with the release before running it: ```powershell -Get-FileHash -Algorithm SHA256 '.\Local-Model-Bench-Setup-1.8.0.exe' +Get-FileHash -Algorithm SHA256 '.\Local-Model-Bench-Setup-1.9.0.exe' ``` If the hash does not match what the release page lists, do not run it. @@ -382,6 +571,14 @@ $env:LMB_VISION = "on" npx tsx scripts/live-vision.ts ``` +One UI script needs neither LM Studio nor a model, because it drives the window against saved data: + +```powershell +node scripts/qa-mtp-sweep.mjs +``` + +It checks the sweep controls and the sweep panel, graphs and export. Its results half labels a saved run's real measurements with MTP depths so the depth-aware screens have something to draw; those labels are the script's invention, so nothing it prints is a statement about MTP speed. It writes only to `work/`. + No live script automatically chooses a large model or downloads weights. Timing and resource use depend on your selected model and hardware. Some historical UI QA scripts require saved fixture runs; `npm test` is the reproducible core check. Commit source, bundled benchmark data, assets, vendor files, dependency manifests, and documentation. Do not commit `work/`, `outputs/`, model weights, API tokens, `.env` files, SQLite databases, or exported personal conversations. `.gitignore` excludes these; it does not remove files already tracked by Git. Exported benchmark histories may contain prompts, responses, local paths, and machine metadata and should be reviewed before sharing. diff --git a/electron/cache-quant.ts b/electron/cache-quant.ts new file mode 100644 index 0000000..a326872 --- /dev/null +++ b/electron/cache-quant.ts @@ -0,0 +1,118 @@ +import {existsSync,mkdirSync,readFileSync,readdirSync,rmdirSync,unlinkSync,writeFileSync} from 'node:fs'; +import {homedir} from 'node:os'; +import path from 'node:path'; +import type {Model} from '../src/types'; +import {needsFlashAttention} from '../src/cache-quant'; +import type {CacheQuant} from '../src/cache-quant'; + +// Quantizing the KV cache. The context is what actually fills a 16 GB card: a run loaded with +// `--context-length C --parallel P` holds C*P tokens of keys and values at f16, and that is the +// part that grows with concurrency rather than with the weights. Storing them at q8_0 halves it +// and at q4_0 quarters it, which is often the difference between a model that fits and a model +// that silently spills into system RAM at a third of the speed. +// +// Like the MTP sidecar, LM Studio exposes no command-line flag and no REST field for this, so the +// one route in is the per-model default config under .lmstudio/.internal, which `lms load` merges +// into every load of that exact file. The same discipline applies here and for the same reason: +// the setting is written immediately before a load and the file is put back byte for byte +// immediately after it, whether the load succeeded or failed. A file this app created is deleted +// again along with any folders it had to make, so nothing is left behind to change a load the +// user makes from LM Studio's own window later. +// +// LM Studio's key names and value shape, read from its own stored configs: +// {"key":"llm.load.llama.kCacheQuantizationType","value":{"checked":true,"value":"q8_0"}} +const K_CACHE='llm.load.llama.kCacheQuantizationType'; +const V_CACHE='llm.load.llama.vCacheQuantizationType'; +// llama.cpp cannot use a quantized KV cache without flash attention, and LM Studio refuses the load +// outright rather than falling back: "V Cache Quantization requires flash attention to be enabled." +// Its default is not dependable — it varies by engine and by whatever the model was last loaded +// with — so a run that asks for a quantized cache writes the flag that makes it possible, in the +// same file and for the same single load. Nothing is written when the cache is left alone, so a +// run that did not ask for this never touches the user's own flash-attention setting. +const FLASH='llm.load.llama.flashAttention'; +const CONFIG_ROOT=['.lmstudio','.internal','user-concrete-model-default-config']; + +// Resolved from LM Studio's own index rather than built from the model key, because the key is a +// display identifier and the config path is keyed by the file. Ambiguity is an error rather than +// a guess: writing the wrong file would change a load this app is not making. +export function modelResource(model:Model,home=homedir()):string{ + let index:any; + try{index=JSON.parse(readFileSync(path.join(home,'.lmstudio','.internal','model-index-cache.json'),'utf8'));} + catch{throw Error('LM Studio’s model index could not be read, so cache quantization cannot be applied. Run with cache quantization off.');} + const matches=(index?.models??[]).filter((m:any)=>m.defaultIdentifier===model.key&&m.sizeBytes===model.size_bytes&&m.format===model.format); + if(matches.length!==1)throw Error(`${model.display_name}: LM Studio’s index does not identify exactly one file for this model, so cache quantization cannot be applied safely.`); + return matches[0].indexedModelIdentifier; +} + +// The resource comes from LM Studio's index rather than from a user, but it still becomes a path +// this app writes to, so it is checked like one. +export function cacheConfigPath(resource:string,home=homedir()):string{ + const parts=String(resource).split(/[\\/]/).filter(Boolean); + if(!parts.length||parts.some(p=>p==='.'||p==='..'||/^[A-Za-z]:$/.test(p)))throw Error('LM Studio returned a model resource that is not a relative path, so its cache settings cannot be written safely.'); + return path.join(home,...CONFIG_ROOT,...parts.slice(0,-1),parts[parts.length-1]+'.json'); +} +function shape(bytes:Buffer){ + const value=JSON.parse(bytes.toString('utf8')); + if(!value||typeof value!=='object'||Array.isArray(value))throw Error('LM Studio’s stored settings for this model are not a configuration object. No measurements were taken.'); + const load=value.load&&typeof value.load==='object'?value.load:{}; + return {...value,operation:value.operation??{fields:[]},load:{...load,fields:Array.isArray(load.fields)?load.fields:[]}}; +} +function pruneEmpty(dir:string,stop:string){ + for(let at=dir;at.length>stop.length&&at.startsWith(stop);at=path.dirname(at)){ + try{if(readdirSync(at).length)return;rmdirSync(at);}catch{return;} + } +} +export type Restore=()=>void; + +// `off` leaves both keys written as unchecked rather than absent, so a cache type the user last +// picked in LM Studio's own window cannot leak into a run that asked for an unquantized cache. +// That is the whole point of measuring: the run gets the setting the run asked for. +// +// Flash attention is written on every load for the same reason, and because leaving it to whatever +// LM Studio happened to have set is not a neutral choice. Measured on Gemma 4 26B A4B at f16 cache, +// changing nothing else: with it off, prompt processing fell from 1281 to 260 tok/s on ROCm and from +// 1822 to 277 on Vulkan, and a fixed cost of 7.1 s and 9.2 s respectively appeared before the first +// token of every request. A run that does not say otherwise gets it on. +export function applyCacheQuant(model:Model,k:CacheQuant,v:CacheQuant,flash:boolean,home=homedir()):Restore{ + const file=cacheConfigPath(modelResource(model,home),home),existed=existsSync(file),original=existed?readFileSync(file):null; + const config=original?shape(original):{preset:'',operation:{fields:[]},load:{fields:[]}}; + const keep=(config.load.fields as {key?:unknown}[]).filter(f=>f&&typeof f.key==='string'&&f.key!==K_CACHE&&f.key!==V_CACHE&&f.key!==FLASH); + const field=(key:string,q:CacheQuant)=>({key,value:q==='off'?{checked:false,value:'f16'}:{checked:true,value:q}}); + config.load.fields=[...keep,{key:FLASH,value:flash},field(K_CACHE,k),field(V_CACHE,v)]; + mkdirSync(path.dirname(file),{recursive:true}); + writeFileSync(file,JSON.stringify(config,null,2)); + let done=false; + return ()=>{ + if(done)return;done=true; + if(original)writeFileSync(file,original); + else{try{unlinkSync(file);}catch{}pruneEmpty(path.dirname(file),path.join(home,...CONFIG_ROOT));} + }; +} + +// What LM Studio reports back after the load, so a run can refuse to measure a cache setting that +// was asked for and not applied. LM Studio reports the instance config with the same value shape. +export function verifyCacheQuant(config:Record,k:CacheQuant,v:CacheQuant,flash?:boolean):void{ + // Flash attention is always requested now, so it is always checked — a load that came back without + // the setting the run asked for would be measuring something else entirely, and the difference is + // large enough (five times the prompt processing) to invalidate the numbers rather than dent them. + const flashGot=(config as any).flash_attention; + if(flash!==undefined&&typeof flashGot==='boolean'&&flashGot!==flash) + throw Error(`LM Studio loaded the model with flash attention ${flashGot?'on':'off'}, not ${flash?'on':'off'} as the run asked. No measurements were taken.`); + if(k==='off'&&v==='off')return; // No cache type was requested, so there is nothing more to confirm. + const got=(key:string):string=>{ + const raw=(config as any)[key]; + if(raw===undefined||raw===null)return 'unreported'; + if(typeof raw==='string')return raw; + if(typeof raw==='object'&&'checked' in raw)return raw.checked?String(raw.value):'off'; + return 'unreported'; + }; + // LM Studio reports these under snake_case names in the instance config it returns. + const kGot=got('k_cache_quantization_type'),vGot=got('v_cache_quantization_type'); + // Without flash attention a quantized cache cannot be in use, whatever the cache fields say. + if(needsFlashAttention(k,v)&&(config as any).flash_attention===false) + throw Error('LM Studio loaded the model without flash attention, which a quantized KV cache requires. No measurements were taken.'); + // An engine that does not report the fields at all is not proof of failure, so it is not treated + // as one; anything it does report has to match, or the numbers would describe another setting. + if(kGot!=='unreported'&&kGot!==k)throw Error(`LM Studio loaded the K cache as ${kGot}, not the requested ${k}. No measurements were taken.`); + if(vGot!=='unreported'&&vGot!==v)throw Error(`LM Studio loaded the V cache as ${vGot}, not the requested ${v}. No measurements were taken.`); +} diff --git a/electron/cli-output.ts b/electron/cli-output.ts new file mode 100644 index 0000000..ec7e96e Binary files /dev/null and b/electron/cli-output.ts differ diff --git a/electron/engine-log.ts b/electron/engine-log.ts new file mode 100644 index 0000000..bc77c73 --- /dev/null +++ b/electron/engine-log.ts @@ -0,0 +1,85 @@ +import {closeSync,openSync,readSync,readdirSync,statSync} from 'node:fs'; +import {homedir} from 'node:os'; +import path from 'node:path'; + +// Reading what LM Studio's inference engine says about itself. +// +// Two things this app needs are printed to LM Studio's server log and offered nowhere else — +// not by `lms`, and not by any field of the v1 HTTP API: +// * the draft model a load opened, which is the only proof a separate MTP head was attached +// (see mtp-sidecar.ts); +// * how much of the drafting actually paid off, which llama.cpp prints per finished request +// as `draft acceptance = 0.47748 (53 accepted / 111 generated), mean len = 2.43`. +// So a stretch of work is watched by remembering how long each log file was before it began +// and reading back only what was appended. No timestamps are parsed, and a file rotated in the +// middle of a watch was not there to be measured, so it is read whole. +export type LogWatch={root:string;sizes:Map}; +const LOG_ROOT=['.lmstudio','server-logs']; +const CAP=8*1024*1024; +function logFiles(root:string):string[]{ + const out:string[]=[]; + const walk=(dir:string,depth:number)=>{ + let entries;try{entries=readdirSync(dir,{withFileTypes:true});}catch{return;} + for(const entry of entries){ + const full=path.join(dir,entry.name); + if(entry.isDirectory()){if(depth>0)walk(full,depth-1);} + else if(entry.name.endsWith('.log'))out.push(full); + } + }; + walk(root,2); + return out.sort(); +} +const sizeOf=(file:string)=>{try{return statSync(file).size;}catch{return 0;}}; +function slice(file:string,from:number,to:number):string{ + const start=Math.max(from,to-CAP),length=to-start; + if(length<=0)return ''; + const handle=openSync(file,'r'); + try{const buffer=Buffer.alloc(length);readSync(handle,buffer,0,length,start);return buffer.toString('utf8');}finally{closeSync(handle);} +} +export function watchEngineLog(home=homedir()):LogWatch{ + const root=path.join(home,...LOG_ROOT); + return {root,sizes:new Map(logFiles(root).map(file=>[file,sizeOf(file)]))}; +} +// Everything written since the watch began. Reading it again is cheap and gives the same +// answer plus whatever has arrived since, so a caller may poll while it waits for a line. +export function sinceWatch(watch:LogWatch):string{ + return logFiles(watch.root).map(file=>slice(file,watch.sizes.get(file)??0,sizeOf(file))).join('\n'); +} +// Every draft model llama.cpp opened since the watch began, lower-cased with forward slashes +// so a path can be compared with the head this app asked for. +export function draftModelsLoaded(watch:LogWatch):string[]{ + const found=new Set(); + for(const match of sinceWatch(watch).matchAll(/loading draft model '([^']+)'/g))found.add(match[1].replaceAll('\\','/').toLowerCase()); + return [...found]; +} + +// How much of the drafting was kept. llama.cpp prints one line per finished request, so under +// concurrency there is one per request in the wave and they are pooled here: the acceptance +// rate is total accepted over total drafted, which weights a long request more heavily than a +// short one exactly as it should. `tasks` is how many lines were actually found, so a caller +// can say whether every request in a wave was accounted for rather than assuming it was. +export type DraftAcceptance={tasks:number;accepted:number;generated:number;acceptance:number;meanLen:number|null}; +const ACCEPTANCE=/draft acceptance = [\d.]+ \(\s*(\d+) accepted \/\s*(\d+) generated\), mean len =\s*([\d.]+)/g; +export function draftAcceptance(watch:LogWatch):DraftAcceptance|null{ + let accepted=0,generated=0,tasks=0,lengths=0; + for(const [,a,g,len] of sinceWatch(watch).matchAll(ACCEPTANCE)){ + accepted+=Number(a);generated+=Number(g);lengths+=Number(len);tasks++; + } + if(!tasks||generated<=0)return null; + // mean len is a per-request average over speculative steps, and the step count is not printed, + // so these are averaged across requests rather than pooled. At concurrency 1 that is one + // request and the figure is exactly what llama.cpp reported. + return {tasks,accepted,generated,acceptance:accepted/generated,meanLen:lengths/tasks}; +} +// llama.cpp prints its timings as a request finishes, which can be just after the HTTP +// response has been handed back, so a wave waits briefly for the lines it expects rather than +// recording a rate from however many happened to have landed. Returns whatever it has when the +// wait runs out; `tasks` then shows the shortfall instead of hiding it. +export async function settledDraftAcceptance(watch:LogWatch,expected:number,timeoutMs=4000,tick=100):Promise{ + const until=Date.now()+timeoutMs; + for(;;){ + const found=draftAcceptance(watch); + if(expected<=0||(found&&found.tasks>=expected)||Date.now()>=until)return found; + await new Promise(r=>setTimeout(r,tick)); + } +} diff --git a/electron/engine.ts b/electron/engine.ts index 4cb4bd1..2a462e6 100644 --- a/electron/engine.ts +++ b/electron/engine.ts @@ -4,80 +4,231 @@ import { api, cli, infer, listModels } from './lmstudio'; import { objectiveScore, gradingPackage, parseGrade } from './scoring'; import { waveMetrics } from './metrics'; import { mtpArgs,verifyMtp } from './mtp'; +import { applySidecar } from './mtp-sidecar'; +import { draftModelsLoaded,settledDraftAcceptance,watchEngineLog } from './engine-log'; +import { mtpDepthText,onSteps,preflightStep,sweepSteps,type MtpStep } from '../src/mtp-sweep'; import { visionArgs,verifyVision,visionSummary,type VisionMode } from './vision'; +import { applyCacheQuant,verifyCacheQuant } from './cache-quant'; +import { cacheQuantText,flashOn } from '../src/cache-quant'; +import { runtimeLabel,selectRuntime } from './runtime'; +import { estimateLoad,loadAdvice,looksLikeMemory } from './load-estimate'; +import { calibratePrefill,calibrationKey,calibrationPrompt,calibrationRepeats,prefillText } from '../src/prefill'; import { serverContext } from '../src/defaults'; import { idleSampler, type GpuSampler } from './gpu'; export type EngineEvent = {type:'sample';sample:Sample}|{type:'wave';wave:Run['waves'][number]}|{type:'progress';progress:Progress}|{type:'log';message:string}|{type:'model';key:string;info:unknown}|{type:'grade';sampleId:string;grade:Grade}|{type:'gpu';gpu:RunGpu}|{type:'finish';status:Run['status'];error?:string}; -export type Adapter = {models:typeof listModels;cli:typeof cli;infer:typeof infer;api:typeof api}; -const real:Adapter={models:listModels,cli,infer,api}; +// The last four read and write LM Studio's own files rather than talking to its server, and are +// listed here so a test can exercise those paths without touching a real LM Studio. +export type Adapter = {models:typeof listModels;cli:typeof cli;infer:typeof infer;api:typeof api;sidecar?:typeof applySidecar;watchLog?:typeof watchEngineLog;draftModels?:typeof draftModelsLoaded;draftAcceptance?:typeof settledDraftAcceptance;cacheQuant?:typeof applyCacheQuant}; +const real:Adapter={models:listModels,cli,infer,api,sidecar:applySidecar,watchLog:watchEngineLog,draftModels:draftModelsLoaded,draftAcceptance:settledDraftAcceptance,cacheQuant:applyCacheQuant}; export function variant(prompt:string,testId:string,concurrency:number,wave:number,slot:number){const tag=createHash('sha256').update(`${testId}/${concurrency}/${wave}/${slot}`).digest('hex').slice(0,24);return `[Benchmark record ${tag}; ignore this record identifier in your answer.]\n${prompt}`;} export async function runEngine(run:Run,settings:Settings,signal:AbortSignal,emit:(e:EngineEvent)=>void,adapter:Adapter=real,retries?:Sample[],gradeOnly=false,gpu:GpuSampler=idleSampler('GPU telemetry was not started for this run.')){ - let completed=0,active=0,currentModel='',failures=0;const measured:Sample[]=[];let owned:string|null=null;const runStart=Date.now(); - let total=gradeOnly?0:retries?retries.length:run.config.modelKeys.length*run.tests.length*run.config.waves*run.config.concurrency.reduce((a,b)=>a+b,0); + let completed=0,active=0,currentModel='',failures=0;const measured:Sample[]=[]; + // What could not be measured, named. A bare count said three things failed but never which, so a + // sweep that lost one depth on one model read exactly like one that lost everything. + const unmeasured:string[]=[];let owned:string|null=null;const runStart=Date.now(); + // MTP depth is a load-time setting, so each depth is its own load of the model and its own + // pass over the whole workload. Without a sweep this is one step and nothing below changes. + const steps=sweepSteps(run.config);let currentStep:MtpStep=steps[0]; + // mtpDepthText, never "MTP " + label: the label for depth 0 is already "MTP off", so the + // shorter form read "MTP MTP off" on every baseline load and wave of a sweep. + const stepNote=()=>steps.length>1?` · ${mtpDepthText(currentStep.depth)}`:''; + const perStep=run.tests.length*run.config.waves*run.config.concurrency.reduce((a,b)=>a+b,0); + let total=gradeOnly?0:retries?retries.length:steps.length*run.config.modelKeys.length*perStep; const progress=(phase:string,message:string)=>emit({type:'progress',progress:{runId:run.id,phase,message,completed,total,active,model:currentModel}}); const log=(message:string)=>emit({type:'log',message}); const cleanup=async()=>{if(owned){const id=owned;owned=null;try{await adapter.api(settings,'/api/v1/models/unload',{instance_id:id});log(`Unloaded ${id}`);}catch(e){log(`Cleanup could not unload ${id}: ${(e as Error).message}`);}}}; - const load=async(key:string,parallel:number,models:Model[],role='benchmark')=>{ + const load=async(key:string,parallel:number,models:Model[],role='benchmark',step:MtpStep=steps[0])=>{ if(signal.aborted)throw Error('Cancelled');const model=models.find(m=>m.key===key);if(!model)throw Error(`Downloaded model not found: ${key}`); // Parallel slots share one KV cache, so the instance needs room for every slot at once. const context=run.config.contextLength,total=serverContext(context,parallel); if(model.max_context_length>0&&total>model.max_context_length)throw Error(`${model.display_name} supports ${model.max_context_length} context tokens; ${parallel} parallel slots at ${context} tokens each need ${total}. Lower the context length or the highest concurrency.`); if(run.config.reasoning!=='default'&&!model.capabilities?.reasoning?.allowed_options?.includes(run.config.reasoning))throw Error(`${model.display_name} does not support reasoning setting ${run.config.reasoning}. Select Model default or another supported option.`); owned=`lmb-${run.id.slice(0,8)}-${randomUUID().slice(0,8)}`; - progress('loading',`Loading ${model.display_name} with ${parallel} parallel slots and ${total} context tokens (${context} per slot)…`); - const mode=role==='judge'?'off':run.config.mtp; + progress('loading',`Loading ${model.display_name} with ${parallel} parallel slots and ${total} context tokens (${context} per slot)${role==='judge'?'':stepNote()}…`); + // A judge takes no part in a sweep: it is loaded without MTP whatever the run asked for. + const mode=role==='judge'?'off':step.mode,draftTokens=role==='judge'?2:step.tokens; // The judge only ever reads text, so it never requests vision. const visionMode:VisionMode|undefined=role==='judge'?'off':run.config.vision; const args=['load',key,'--identifier',owned,'--context-length',String(total),'--parallel',String(parallel),'--yes'];if(run.config.gpu!=='auto')args.push('--gpu',run.config.gpu); - args.push(...mtpArgs(model,mode,run.config.mtpDraftTokens??2)); + args.push(...mtpArgs(model,mode,draftTokens)); args.push(...visionArgs(model,visionMode)); // Always empty: LM Studio has no projector flag. Validates the request first. - const started=performance.now();const loadOutput=await adapter.cli(settings,args,signal); + // A model whose MTP heads are a separate file cannot carry them on the command line, so the + // setting is written into LM Studio's own per-model config for the length of this one load + // and taken straight back out again. The engine log is watched across the load either way: + // at every other depth, including the sweep's MTP-off baseline, the point is to prove that + // no head was loaded. + const paired=model.nativeMtp?.kind==='sidecar',watch=paired&&mode!==undefined?(adapter.watchLog??watchEngineLog)():null; + if(paired&&mode==='on')log(`Native MTP head: ${model.nativeMtp!.draftResource} · loaded as a separate file. LM Studio offers no command-line flag for this, so the setting is written to its per-model configuration for this load only and restored immediately afterwards.`); + // The KV cache type goes through the same per-model config file as the MTP head, for the same + // reason: LM Studio has no flag for it. Cache first, head second, so the head's snapshot of the + // file already contains the cache fields; the two are undone in the opposite order, so whatever + // the file held before this load is what it holds after it. + const cacheK=run.config.cacheK??'off',cacheV=run.config.cacheV??'off'; + const flash=flashOn(run.config.flashAttention); + const quantising=cacheK!=='off'||cacheV!=='off'; + if(quantising)log(`KV cache: ${cacheQuantText(cacheK,cacheV)} · written to LM Studio's per-model configuration for this load only and restored immediately afterwards. The cache is the part of this run's memory that grows with concurrency, so quantizing it is what lets a bigger model or a higher concurrency stay on the GPU.`); + log(`Flash attention: ${flash?'on':'off'}${flash?'':' — off was asked for, so expect prompt processing several times slower and seconds of fixed cost before each first token'}. Written to LM Studio's per-model configuration for this load only and restored immediately afterwards.`); + // Written on every load now, not only when the cache is quantized: the run states the condition + // rather than inheriting whatever LM Studio last had, and the difference is about five times the + // prompt processing plus several seconds of fixed cost per request. + const undoCache=(adapter.cacheQuant??applyCacheQuant)(model,cacheK,cacheV,flash); + const restore=paired&&mode==='on'?(adapter.sidecar??applySidecar)(model,draftTokens):null; + const started=performance.now();let loadOutput:string; + try{loadOutput=await adapter.cli(settings,args,signal);} + catch(e){ + // LM Studio says it could not fit the model, in words that name none of the settings that + // decided how much room it needed. The arithmetic this run already did is added here, where + // someone reading the log is actually looking. Nothing is retried and nothing is changed. + // Only when the failure actually looks like one of room. A load that failed for another reason + // is passed through exactly as LM Studio reported it, because memory advice there would send + // someone to change a setting that was never the problem. + if(signal.aborted||!looksLikeMemory((e as Error).message))throw e; + const estimate=await estimateLoad(settings,adapter.cli,key,total,parallel,signal); + throw Error(`${(e as Error).message}\n\n${loadAdvice(model,context,parallel,cacheK,cacheV,estimate)}`); + } + finally{restore?.();undoCache?.();} const fresh=await adapter.models(settings);const owner=fresh.find(m=>m.loaded_instances.some(i=>i.id===owned));const instance=owner?.loaded_instances.find(i=>i.id===owned); if(owner&&owner.key!==key)throw Error('LM Studio resolved the model key to a different file. No measurements were taken.'); if(!instance)throw Error('Loaded instance was not returned by LM Studio. Check that the CLI and API address use the same server.'); if(instance.config.parallel!==parallel||instance.config.context_length!==total)throw Error(`Loaded settings differ: requested parallel=${parallel}, context=${total}; received parallel=${instance.config.parallel??'unknown'}, context=${instance.config.context_length}. No measurements were taken.`); - verifyMtp(instance.config,mode,model.format,run.config.mtpDraftTokens??2); + // A depth LM Studio did not apply throws here, before a single request is measured, so no + // saved response can be attributed to a depth the server was not actually running. + verifyMtp(instance.config,mode,model.format,draftTokens,{...(paired?{kind:'sidecar' as const,draftPath:model.nativeMtp!.draftPath}:{}),draftersLoaded:watch?(adapter.draftModels??draftModelsLoaded)(watch):[]}); + // A cache type LM Studio did not apply would make the run report a memory saving it never got, + // and compare against other runs as though it had. Checked here, before anything is measured. + verifyCacheQuant(instance.config as Record,cacheK,cacheV,flash); // Vision is confirmed from the reloaded model entry's reported capability, the only // evidence LM Studio gives; a failure throws here, before any request is measured. const vision=verifyVision(owner,visionMode); log(visionSummary(vision)); - log(`Native MTP: ${mode??'legacy/default'}${mode==='on'?' · '+(run.config.mtpDraftTokens??2)+' draft tokens':''}`); + log(`Native MTP: ${mode??'legacy/default'}${mode==='on'?` · ${draftTokens} draft tokens · ${paired?'separate MTP head':'heads built into the model file'}`:''}${steps.length>1&&role!=='judge'?` (sweep step ${steps.indexOf(step)+1} of ${steps.length})`:''}`); log(`${model.display_name}: ${parallel} parallel slots share ${total} context tokens, ${context} per concurrent request.`); - const info={model,instance,vision,loadMs:performance.now()-started,loadOutput,reasoning:run.config.reasoning==='default'?(model.capabilities?.reasoning?.default??'not exposed'):run.config.reasoning};emit({type:'model',key:role==='judge'?'judge:'+key:key,info});return {model,id:owned}; + const info={model,instance,vision,loadMs:performance.now()-started,loadOutput,mtp:{depth:role==='judge'?null:step.depth,mode:mode??'legacy/default',draftTokens:mode==='on'?draftTokens:null,kind:mode==='on'?(model.nativeMtp?.kind??null):null,head:paired&&mode==='on'?model.nativeMtp!.draftResource:null},reasoning:run.config.reasoning==='default'?(model.capabilities?.reasoning?.default??'not exposed'):run.config.reasoning}; + // A preflight is a question, not a measurement, so it leaves no record of a model behind. + if(role!=='preflight')emit({type:'model',key:role==='judge'?'judge:'+key:key,info}); + // A sweep loads the same model once per depth, so each depth also keeps its own record of + // what LM Studio confirmed and how long the load took; the plain key stays as it was. + if(steps.length>1&&role!=='judge'&&role!=='preflight')emit({type:'model',key:`${key} · MTP ${step.label}`,info}); + return {model,id:owned}; + }; + // Two requests, one short and one long, timed the same way every measured request is. Their + // difference is the real per-token prompt-processing rate; see src/prefill.ts for why a single + // request cannot give one. This runs once per loaded instance, costs two requests, and is + // recorded beside the model rather than mixed in with the measured samples. + const calibrate=async(model:Model,instance:string,key:string,step:MtpStep)=>{ + try{ + progress('warmup',`Calibrating prompt processing for ${model.display_name} (excluded from results)…`); + const point=async(repeats:number)=>{ + // A unique prefix per request: LM Studio reuses a cached prompt prefix, and a second timing + // served from that cache would measure the cache rather than the prefill. + const prompt=`[Calibration ${randomUUID()}]\n${calibrationPrompt(repeats)}`; + const r=await adapter.infer({...settings,timeoutSec:run.config.timeoutSec},instance,prompt,4,0,run.config.reasoning,signal); + if(r.status!=='completed')throw Error(r.error||r.status); + const ms=r.metrics.prefillMs??r.metrics.ttftMs; + if(r.metrics.inputTokens===null||ms===null)throw Error('LM Studio returned no prompt timing.'); + return {tokens:r.metrics.inputTokens,ms}; + }; + const small=await point(calibrationRepeats.small),big=await point(calibrationRepeats.big); + const calibration=calibratePrefill(small,big); + log(`Prompt processing for ${model.display_name}: ${prefillText(calibration)}. ${calibration.note}`); + emit({type:'model',key:calibrationKey(key,steps.length>1?step.depth:null),info:calibration}); + }catch(e){ + // A calibration that could not be taken costs the run nothing else: the per-request figures + // are still recorded, they are simply left uncorrected. + log(`Prompt-processing calibration skipped for ${model.display_name}: ${(e as Error).message}`); + } }; const request=async(model:Model,instance:string,test:TestCase,c:number,wave:number,slot:number,waveId:string,warmup=false,previous?:Sample)=>{ - const prompt=previous?.prompt??(test.benchmark?test.prompt:variant(test.prompt,test.id,c,wave,slot));active++;progress(warmup?'warmup':'benchmarking',warmup?'Warming up (excluded from results)…':`${test.name} · ${c} concurrent · wave ${wave+1}`); + const prompt=previous?.prompt??(test.benchmark?test.prompt:variant(test.prompt,test.id,c,wave,slot));active++;progress(warmup?'warmup':'benchmarking',warmup?'Warming up (excluded from results)…':`${test.name} · ${c} concurrent · wave ${wave+1}${stepNote()}`); const max=warmup?32:test.maxTokens>0?Math.min(test.maxTokens,run.config.maxTokens):run.config.maxTokens; let result:Awaited>;const startedAt=Date.now(); try{result=await adapter.infer({...settings,timeoutSec:run.config.timeoutSec},instance,prompt,max,run.config.temperature,run.config.reasoning,signal,test.image);}finally{active--;} const finishedAt=Date.now(); - const sample:Sample={...result,id:randomUUID(),runId:run.id,modelKey:model.key,modelName:model.display_name,testId:test.id,testName:test.name,concurrency:c,waveId,wave,slot,warmup,prompt,objective:result.status==='completed'&&!warmup?objectiveScore(result.output,test):{score:null,checks:[]},grades:[],created:new Date().toISOString(),retryOf:previous?.id,gpu:gpu.window(startedAt,finishedAt)}; + // The depth is recorded on the measurement itself, not read back from the run's settings: + // a sweep gives one run several depths, and every response has to say which one produced it. + const sample:Sample={...result,...(currentStep.depth===null?{}:{mtpTokens:currentStep.depth}),id:randomUUID(),runId:run.id,modelKey:model.key,modelName:model.display_name,testId:test.id,testName:test.name,concurrency:c,waveId,wave,slot,warmup,prompt,objective:result.status==='completed'&&!warmup?objectiveScore(result.output,test):{score:null,checks:[]},grades:[],created:new Date().toISOString(),retryOf:previous?.id,gpu:gpu.window(startedAt,finishedAt)}; if(!warmup){completed++;measured.push(sample);if(sample.status!=='completed')failures++;} emit({type:'sample',sample});progress('benchmarking',`${test.name} · ${completed} of ${total} finished`);return sample; }; const wave=async(model:Model,instance:string,test:TestCase,c:number,index:number,previous?:Sample[])=>{ const id=randomUUID(),start=performance.now(),startedAt=Date.now(); + // How much of the drafting was kept is printed per finished request by llama.cpp and offered + // by no API, so a wave with MTP on reads it out of the log it wrote while the wave ran. One + // line per request, pooled over the wave: under concurrency that is exactly the granularity + // the number belongs at, because the slots share a batch and draft against each other. + const speculating=currentStep.mode==='on'; + const acceptance=speculating?(adapter.watchLog??watchEngineLog)():null; const results=await Promise.all(Array.from({length:c},(_,slot)=>request(model,instance,test,c,index,slot,id,false,previous?.[slot]))); - emit({type:'wave',wave:waveMetrics(id,run.id,model.key,test.id,c,performance.now()-start,results,gpu.window(startedAt,Date.now()))}); + const elapsed=performance.now()-start,window=gpu.window(startedAt,Date.now()); + const draft=acceptance?await (adapter.draftAcceptance??settledDraftAcceptance)(acceptance,results.filter(s=>s.status==='completed').length):null; + emit({type:'wave',wave:waveMetrics(id,run.id,model.key,test.id,c,elapsed,results,window,draft)}); }; + // Which llama.cpp build runs this. LM Studio has one selected engine at a time and no per-load + // flag, so a run that names one selects it here and puts the previous choice back in the finally + // below — whatever else happens to the run. Naming none keeps LM Studio exactly as the user left + // it, which is what every run made before this field existed did. + let restoreRuntime:(()=>Promise)|null=null; try{ + if(run.config.runtime&&!gradeOnly){ + restoreRuntime=await selectRuntime(settings,adapter.cli,run.config.runtime,signal); + log(`Runtime: ${runtimeLabel(run.config.runtime.split('@')[0])} · ${run.config.runtime}. LM Studio's engine selection is global, so it is switched for this run and restored afterwards.`); + } + // Which engine was selected is already recorded in run.environment.runtime, captured when the + // run was created. Nothing is asked here, so a run that names no runtime issues no extra + // command and behaves exactly as it did before this field existed. const models=await adapter.models(settings); const other=models.flatMap(m=>m.loaded_instances.map(i=>`${m.display_name} (${i.id})`));if(other.length)log('Other loaded instances may affect memory and speed: '+other.join(', ')); if(!gradeOnly){ + // Whether a model can attach its MTP head at all is answered by one load, before anything is + // measured. Without this, a sweep finds out that a depth is impossible only after it has spent + // the time measuring the MTP-off baseline it meant to compare that depth against, and that + // baseline is then left with nothing to compare to. The probe loads at the same parallel slots + // and context the run itself will use, so a head that only fails to fit under the run's real + // pressure fails here too rather than passing a gentler test than the run. + const blocked=new Map(); + const probe=run.config.mtpPreflight&&!retries?preflightStep(steps):null; + if(probe){ + const depths=onSteps(steps); + for(const key of run.config.modelKeys){ + if(signal.aborted)break;currentModel=key;currentStep=probe; + progress('preflight',`Checking ${key} can load its MTP head…`); + try{ + await load(key,Math.max(...run.config.concurrency),models,'preflight',probe); + log(`MTP preflight: ${key} loaded its MTP head.`); + }catch(e){ + // One model that cannot draft at all is one failure, not one per depth it would have run. + blocked.set(key,(e as Error).message);failures++; + unmeasured.push(`${key} · MTP ${depths.map(s=>s.label).join(', ')} (the MTP head would not load)`); + total-=depths.length*perStep; + log(`MTP preflight: ${key} cannot load its MTP head, so its MTP depths were skipped before anything was measured: ${(e as Error).message}`); + }finally{await cleanup();} + } + currentStep=steps[0]; + } for(const key of run.config.modelKeys){ if(signal.aborted)break;currentModel=key; - try{ - const {model,id}=await load(key,Math.max(...run.config.concurrency),models); - const warmupTest={...run.tests[0],prompt:'Reply with the word ready.',rules:[],image:undefined};const warm=await request(model,id,warmupTest,1,-1,0,randomUUID(),true); - if(warm.status!=='completed')throw Error(`Warm-up failed: ${warm.error}`); - if(retries){ - const groups=new Map();for(const s of retries.filter(s=>s.modelKey===key)){const list=groups.get(s.waveId)??[];list.push(s);groups.set(s.waveId,list);} - for(const group of groups.values()){if(signal.aborted)break;const test=run.tests.find(t=>t.id===group[0].testId)!;await wave(model,id,test,group.length,group[0].wave,group);} - }else{ - for(const test of run.tests){for(const c of run.config.concurrency){for(let i=0;is.modelKey===key&&(s.mtpTokens??null)===step.depth):null; + if(mine&&!mine.length)continue; + try{ + const {model,id}=await load(key,Math.max(...run.config.concurrency),models,'benchmark',step); + const warmupTest={...run.tests[0],prompt:'Reply with the word ready.',rules:[],image:undefined};const warm=await request(model,id,warmupTest,1,-1,0,randomUUID(),true); + if(warm.status!=='completed')throw Error(`Warm-up failed: ${warm.error}`); + await calibrate(model,id,key,step); + if(mine){ + const groups=new Map();for(const s of mine){const list=groups.get(s.waveId)??[];list.push(s);groups.set(s.waveId,list);} + for(const group of groups.values()){if(signal.aborted)break;const test=run.tests.find(t=>t.id===group[0].testId)!;await wave(model,id,test,group.length,group[0].wave,group);} + }else{ + for(const test of run.tests){for(const c of run.config.concurrency){for(let i=0;i(run.modelInfo[modelKey] as {vision?:VisionState}|undefined)?.vision; export const visionRunNote=(run:Run)=>{const v=(run.environment as {vision?:{requested:string;note:string}}).vision;const states=run.config.modelKeys.map(k=>visionOf(run,k)).filter(Boolean) as VisionState[];return {requested:run.config.vision??v?.requested??'legacy/default',effective:[...new Set(states.map(s=>s.effective))].join(' | ')||'Not confirmed: no model was loaded for this run.',limitation:[...new Set(states.map(s=>s.limitation).filter(Boolean))].join(' ')||v?.note||''};}; +// Accepted-draft figures for the waves behind one summary row, pooled the way the sweep panel +// pools them: total kept over total drafted. Null where nothing was drafted at all, which is +// not the same as drafting that was always rejected. +const draftShare=(waves:Wave[])=>{const found=waves.map(w=>w.draft).filter(Boolean) as NonNullable[];const generated=found.reduce((n,d)=>n+d.generated,0);return generated>0?found.reduce((n,d)=>n+d.accepted,0)/generated:null;}; +const draftRun=(waves:Wave[])=>{const found=(waves.map(w=>w.draft).filter(Boolean) as NonNullable[]).filter(d=>d.meanLen!==null);const tasks=found.reduce((n,d)=>n+d.tasks,0);return tasks>0?found.reduce((n,d)=>n+d.meanLen!*d.tasks,0)/tasks:null;}; export function summaries(run:Run){ - const groups=new Map();for(const s of run.samples.filter(s=>!s.warmup)){const key=JSON.stringify([s.modelKey,s.testId,s.concurrency]);const list=groups.get(key)??[];list.push(s);groups.set(key,list);} - return [...groups.values()].map(ss=>{const s=ss[0],ok=ss.filter(x=>x.status==='completed'),waves=run.waves.filter(w=>w.modelKey===s.modelKey&&w.testId===s.testId&&w.concurrency===s.concurrency);const gpu=combineStats(ss.map(x=>x.gpu));return {vision:run.config.vision??'legacy/default',visionEffective:visionOf(run,s.modelKey)?.effective??'Not confirmed',visionImagesSent:visionOf(run,s.modelKey)?.imagesSent??false,visionProjectorUnloaded:visionOf(run,s.modelKey)?.projectorUnloaded??false,visionLimitation:visionOf(run,s.modelKey)?.limitation??'',mtp:run.config.mtp??'legacy/default',mtpDraftTokens:run.config.mtp==='on'?(run.config.mtpDraftTokens??2):null,model:s.modelName,modelKey:s.modelKey,test:s.testName,testId:s.testId,concurrency:s.concurrency,requests:ss.length,failures:ss.length-ok.length,failureRate:(ss.length-ok.length)/ss.length,generationTps:average(ok.map(x=>x.metrics.generationTps)),estimatedPrefillTps:average(ok.map(x=>x.metrics.prefillTps)),throughput:average(waves.map(w=>w.throughput)),medianMs:percentile(ok.map(x=>x.metrics.durationMs),0.5),p95Ms:percentile(ok.map(x=>x.metrics.durationMs),0.95),ttftMs:average(ok.map(x=>x.metrics.ttftMs)),objective:average(ok.map(x=>x.objective.score)),localJudge:average(ok.map(x=>x.grades.filter(g=>g.source==='local').at(-1)?.score??null)),externalJudge:average(ok.map(x=>x.grades.filter(g=>g.source==='external').at(-1)?.score??null)),gpuHotSpotAvg:gpu?.tempHotSpot?.avg??null,gpuHotSpotMax:gpu?.tempHotSpot?.max??null,gpuCoreTempAvg:gpu?.tempCore?.avg??null,gpuCoreTempMax:gpu?.tempCore?.max??null,gpuMemoryTempMax:gpu?.tempMemory?.max??null,gpuPowerAvg:gpu?.power?.avg??null,gpuLoadAvg:gpu?.load?.avg??null,gpuReadings:gpu?.samples??0};}); + // An MTP sweep measures one model at several prediction depths in a single run, so the depth + // is part of what identifies a measurement: pooling depths together would average a setting + // against itself and hide the very difference the sweep was run to find. + const groups=new Map();for(const s of run.samples.filter(s=>!s.warmup)){const key=JSON.stringify([s.modelKey,s.testId,s.concurrency,s.mtpTokens??null]);const list=groups.get(key)??[];list.push(s);groups.set(key,list);} + return [...groups.values()].map(ss=>{const s=ss[0],ok=ss.filter(x=>x.status==='completed'),waves=run.waves.filter(w=>w.modelKey===s.modelKey&&w.testId===s.testId&&w.concurrency===s.concurrency&&(w.mtpTokens??null)===(s.mtpTokens??null));const gpu=combineStats(ss.map(x=>x.gpu));return {vision:run.config.vision??'legacy/default',visionEffective:visionOf(run,s.modelKey)?.effective??'Not confirmed',visionImagesSent:visionOf(run,s.modelKey)?.imagesSent??false,visionProjectorUnloaded:visionOf(run,s.modelKey)?.projectorUnloaded??false,visionLimitation:visionOf(run,s.modelKey)?.limitation??'',mtp:s.mtpTokens===undefined?(run.config.mtp??'legacy/default'):s.mtpTokens===0?'off':'on',mtpDraftTokens:s.mtpTokens===undefined?(run.config.mtp==='on'?(run.config.mtpDraftTokens??2):null):(s.mtpTokens||null),mtpDepth:s.mtpTokens??null,draftAcceptance:draftShare(waves),draftMeanLen:draftRun(waves),model:s.modelName,modelKey:s.modelKey,test:s.testName,testId:s.testId,concurrency:s.concurrency,requests:ss.length,failures:ss.length-ok.length,failureRate:(ss.length-ok.length)/ss.length,generationTps:average(ok.map(x=>x.metrics.generationTps)),estimatedPrefillTps:average(ok.map(x=>x.metrics.prefillTps)),throughput:average(waves.map(w=>w.throughput)),medianMs:percentile(ok.map(x=>x.metrics.durationMs),0.5),p95Ms:percentile(ok.map(x=>x.metrics.durationMs),0.95),ttftMs:average(ok.map(x=>x.metrics.ttftMs)),objective:average(ok.map(x=>x.objective.score)),localJudge:average(ok.map(x=>x.grades.filter(g=>g.source==='local').at(-1)?.score??null)),externalJudge:average(ok.map(x=>x.grades.filter(g=>g.source==='external').at(-1)?.score??null)),gpuHotSpotAvg:gpu?.tempHotSpot?.avg??null,gpuHotSpotMax:gpu?.tempHotSpot?.max??null,gpuCoreTempAvg:gpu?.tempCore?.avg??null,gpuCoreTempMax:gpu?.tempCore?.max??null,gpuMemoryTempMax:gpu?.tempMemory?.max??null,gpuPowerAvg:gpu?.power?.avg??null,gpuLoadAvg:gpu?.load?.avg??null,gpuReadings:gpu?.samples??0};}); +} +// Written only when a run actually measured more than one depth. A sweep whose later steps +// never loaded leaves one depth of results, which is a normal run and is reported as one. +export function mtpSweepReport(run:Run){ + if(measuredDepths(run).length<2)return ''; + const f=(n:number|null,d=2)=>n===null?'Unavailable':n.toFixed(d); + const accepted=(r:{draft:{acceptance:number|null;accepted:number;generated:number;meanLen:number|null;tasks:number}|null})=> + r.draft?.acceptance==null?'not applicable':`${(r.draft.acceptance*100).toFixed(1)}% (${r.draft.accepted} of ${r.draft.generated} drafted, mean accepted run ${f(r.draft.meanLen)}, from ${r.draft.tasks} reported request(s))`; + const body=sweepVerdicts(run).map(v=>`### ${v.modelName} — ${v.concurrency} concurrent request${v.concurrency===1?'':'s'}\n\n${v.summary}\n\n`+v.rows.map(r=> + `- ${r.label}: ${f(r.generationTps)} tok/s${r.generationSd===null?'':` (spread ${f(r.generationSd)})`}; total ${f(r.throughput)} tok/s; drafted tokens accepted ${accepted(r)}; median ${f(r.medianMs===null?null:r.medianMs/1000)} s; time to first token ${f(r.ttftMs===null?null:r.ttftMs/1000)} s; objective ${f(r.objective,1)}; ${r.completed} of ${r.requests} completed.`).join('\n')).join('\n\n'); + return `\n## Native MTP sweep\n\nEach depth is a separate load of the model. Depth 0 is MTP off; every other depth is how many tokens the model's own prediction heads draft ahead before the main model verifies them. Depths are compared only within one concurrency level, because the depth that is fastest on its own need not be the fastest under load. The accepted-draft figures are LM Studio's own, printed by its engine as each request finishes. Requests inside one concurrent wave are not independent of each other, so the spread quoted here is a floor on the real uncertainty rather than a significance test.\n\n${body}\n`; } const range=(s:GpuStat|null|undefined,unit:string,digits=1)=>!s?'Unavailable':`${s.min.toFixed(digits)}–${s.max.toFixed(digits)} ${unit} (average ${s.avg.toFixed(digits)})`; export function gpuReport(gpu:RunGpu|undefined){ @@ -25,5 +45,5 @@ export function exportText(run:Run,format:string){ if(format==='csv'){const headers=rows.length?Object.keys(rows[0]):['model','test','concurrency','requests'];return '\uFEFF'+[headers.map(cell).join(','),...rows.map(row=>headers.map(k=>cell((row as any)[k])).join(','))].join('\r\n');} if(format!=='md')throw Error('Unsupported export format'); const f=(n:number|null)=>n===null?'Unavailable':n.toFixed(2); - return benchmarkReport(run)+`\n\n# ${run.config.name||'Local Model Bench'}\n\nCreated: ${run.created}\nStatus: ${run.status}\nVision requested: ${visionRunNote(run).requested}; confirmed effective state: ${visionRunNote(run).effective}${visionRunNote(run).limitation?'\n\nVision limitation: '+visionRunNote(run).limitation:''}\n\nNative MTP: ${run.config.mtp??'legacy/default'}; draft tokens: ${run.config.mtp==='on'?(run.config.mtpDraftTokens??2):'not applicable'}\n\nPrefill rates are estimates from client-timed processing events. Cache reuse is not measurable. Warm-ups are excluded. Generation includes reasoning tokens where reported by LM Studio.\n\n## Measurements\n\n`+rows.map(r=>`### ${r.model} / ${r.test} / concurrency ${r.concurrency}\n\nGeneration: ${f(r.generationTps)} tok/s; estimated prefill: ${f(r.estimatedPrefillTps)} tok/s; aggregate throughput: ${f(r.throughput)} tok/s. Median latency: ${f(r.medianMs)} ms; p95: ${f(r.p95Ms)} ms. Failures: ${r.failures}/${r.requests}.\n\nObjective: ${f(r.objective)}; local judge: ${f(r.localJudge)}; external judge: ${f(r.externalJudge)}.\n\nGPU hot spot: ${f(r.gpuHotSpotAvg)} °C average, ${f(r.gpuHotSpotMax)} °C peak; core ${f(r.gpuCoreTempAvg)} °C average; board power ${f(r.gpuPowerAvg)} W; from ${r.gpuReadings} reading(s) inside these requests.\n`).join('\n')+'\n## GPU thermals\n\n'+gpuReport(run.gpu)+'\n\n## Responses\n\n'+run.samples.filter(s=>!s.warmup).map(s=>`### ${s.modelName} — ${s.testName} (${s.id})\n\nStatus: ${s.status}${s.error?' — '+s.error:''}\n\n**Prompt**\n\n${s.prompt}\n\n**Response**\n\n${s.output}\n\n**Objective checks**\n\n${s.objective.checks.map(c=>`- ${c.passed?'PASS':'FAIL'}: ${c.label}: ${c.detail}`).join('\n')}\n\n${s.grades.map(g=>`**${g.source} grade (${g.judge}): ${g.score}**\n\n${g.summary}\n`).join('\n')}`).join('\n')+'\n## Run log\n\n'+run.logs.join('\n'); + return benchmarkReport(run)+`\n\n# ${run.config.name||'Local Model Bench'}\n\nCreated: ${run.created}\nStatus: ${run.status}\nVision requested: ${visionRunNote(run).requested}; confirmed effective state: ${visionRunNote(run).effective}${visionRunNote(run).limitation?'\n\nVision limitation: '+visionRunNote(run).limitation:''}\n\nNative MTP: ${run.config.mtp??'legacy/default'}; draft tokens: ${run.config.mtp==='on'?(run.config.mtpDraftTokens??2):'not applicable'}${measuredDepths(run).length>1?`; swept depths: ${measuredDepths(run).map(d=>mtpDepthLabel(d)).join(', ')}`:''}\n\nPrefill rates are estimates from client-timed processing events. Cache reuse is not measurable. Warm-ups are excluded. Generation includes reasoning tokens where reported by LM Studio.\n\n## Measurements\n\n`+rows.map(r=>`### ${r.model} / ${r.test} / concurrency ${r.concurrency}${r.mtpDepth===null?'':' / '+mtpDepthText(r.mtpDepth)}\n\nGeneration: ${f(r.generationTps)} tok/s; estimated prefill: ${f(r.estimatedPrefillTps)} tok/s; aggregate throughput: ${f(r.throughput)} tok/s. Median latency: ${f(r.medianMs)} ms; p95: ${f(r.p95Ms)} ms. Failures: ${r.failures}/${r.requests}.\n\nObjective: ${f(r.objective)}; local judge: ${f(r.localJudge)}; external judge: ${f(r.externalJudge)}.\n\nGPU hot spot: ${f(r.gpuHotSpotAvg)} °C average, ${f(r.gpuHotSpotMax)} °C peak; core ${f(r.gpuCoreTempAvg)} °C average; board power ${f(r.gpuPowerAvg)} W; from ${r.gpuReadings} reading(s) inside these requests.\n`).join('\n')+mtpSweepReport(run)+'\n## GPU thermals\n\n'+gpuReport(run.gpu)+'\n\n## Responses\n\n'+run.samples.filter(s=>!s.warmup).map(s=>`### ${s.modelName} — ${s.testName} (${s.id})\n\nStatus: ${s.status}${s.error?' — '+s.error:''}\n\n**Prompt**\n\n${s.prompt}\n\n**Response**\n\n${s.output}\n\n**Objective checks**\n\n${s.objective.checks.map(c=>`- ${c.passed?'PASS':'FAIL'}: ${c.label}: ${c.detail}`).join('\n')}\n\n${s.grades.map(g=>`**${g.source} grade (${g.judge}): ${g.score}**\n\n${g.summary}\n`).join('\n')}`).join('\n')+'\n## Run log\n\n'+run.logs.join('\n'); } diff --git a/electron/history.ts b/electron/history.ts index 97545f5..0a04cf0 100644 --- a/electron/history.ts +++ b/electron/history.ts @@ -2,17 +2,23 @@ import {summaries} from './export'; import {profilesFor,type ModelProfile} from '../src/model-profile'; import {familyOf,promptSizeOf,quantTier,sizeBucket,sizeLabel,type HistoryRow} from '../src/history'; import type {Model,Run,Sample} from '../src/types'; +import {backendOf} from './runtime'; +import {calibrationFor} from '../src/prefill'; // summaries() already groups a run into (model, test, concurrency) measurements and computes every rate, // failure count and GPU statistic. This adds only what pooling across runs needs: which run a measurement // came from, what the model was, and the counts and durations behind each average. -const groupKey=(s:Pick)=>JSON.stringify([s.modelKey,s.testId,s.concurrency]); +// The MTP depth is part of the key for the same reason it is part of a summary row: one run can +// hold several depths, and pooling them would average a setting against itself. +const groupKey=(s:Pick,depth:number|null)=>JSON.stringify([s.modelKey,s.testId,s.concurrency,depth]); export function historyRows(runs:Run[],saved:Record):HistoryRow[]{ return runs.flatMap(run=>{ const profiles=profilesFor(run,[],saved); + // One run is one engine, so this is resolved once per run rather than per measurement. + const backend=backendOf(run); const measured=new Map(); - for(const s of run.samples){if(s.warmup)continue;const list=measured.get(groupKey(s))??[];list.push(s);measured.set(groupKey(s),list);} + for(const s of run.samples){if(s.warmup)continue;const key=groupKey(s,s.mtpTokens??null);const list=measured.get(key)??[];list.push(s);measured.set(key,list);} return summaries(run).map(r=>{ - const samples=measured.get(groupKey(r))??[],ok=samples.filter(s=>s.status==='completed'); + const samples=measured.get(groupKey(r,r.mtpDepth))??[],ok=samples.filter(s=>s.status==='completed'); const model=(run.modelInfo[r.modelKey] as {model?:Model}|undefined)?.model; const profile=profiles[r.modelKey]??{totalB:null,activeB:null,kind:'unknown' as const,source:'No saved model metadata'}; const test=run.tests.find(t=>t.id===r.testId); @@ -24,6 +30,9 @@ export function historyRows(runs:Run[],saved:Record):Histor quantization:model?.quantization?.name??'',quantTier:quantTier(model?.quantization?.name), publisher:typeof model?.publisher==='string'?model.publisher:'',architecture:String(model?.architecture??''), promptSize:promptSizeOf(r.testId),testKind:test?.kind??'quality',benchmarkPack:test?.benchmark?.packId??null, + backend:backend.label,backendRef:backend.ref, + prefillCalibratedTps:calibrationFor(run,r.modelKey,r.mtpDepth)?.marginalTps??null, + prefillOverheadMs:calibrationFor(run,r.modelKey,r.mtpDepth)?.overheadMs??null, reasoning:run.config.reasoning,temperature:run.config.temperature,contextLength:run.config.contextLength, maxTokens:run.config.maxTokens,waves:run.config.waves, completed:ok.length,durationsMs:ok.map(s=>s.metrics.durationMs).filter(n=>Number.isFinite(n)), diff --git a/electron/json-answer.ts b/electron/json-answer.ts index 6b7e8e3..7a503c6 100644 --- a/electron/json-answer.ts +++ b/electron/json-answer.ts @@ -1,3 +1,20 @@ +// A model asked for "only JSON" very often returns it inside a ```json fence. That is markup around +// the answer, not prose instead of it, and this app already unwraps exactly the same wrapper when it +// reads a judge's reply (see parseGrade). Not doing it here scored a perfect answer zero: Gemma 4 +// 26B A4B returned all five subnet values correctly, fenced, and every check failed with +// "Unexpected token '`'". Only a single fence that opens the text and closes it is removed; prose +// around the JSON is still a failure, because that is a different thing to get wrong. +export function unfence(text:string):{json:string;fenced:boolean}{ + const trimmed=text.trim(); + if(!trimmed.startsWith('```'))return {json:trimmed,fenced:false}; + const opener=/^```[A-Za-z0-9_-]*[^\S\r\n]*\r?\n?/; + const closer=/\r?\n?```$/; + // A fence that never closes is not a wrapper, so it is left exactly as it was. + if(!closer.test(trimmed))return {json:trimmed,fenced:false}; + const inner=trimmed.replace(opener,'').replace(closer,''); + // A fence that never closes, or that wraps several blocks, is left exactly as it was. + return inner.includes('```')?{json:trimmed,fenced:false}:{json:inner.trim(),fenced:true}; +} // JSON.parse accepts duplicate keys. Reject those so conflicting answers cannot // silently collapse to whichever value appeared last. export function parseAnswer(text:string):unknown { diff --git a/electron/lmstudio.ts b/electron/lmstudio.ts index 29c72a6..c9ca5f8 100644 Binary files a/electron/lmstudio.ts and b/electron/lmstudio.ts differ diff --git a/electron/load-estimate.ts b/electron/load-estimate.ts new file mode 100644 index 0000000..264e713 --- /dev/null +++ b/electron/load-estimate.ts @@ -0,0 +1,70 @@ +import type {Model,Settings} from '../src/types'; +import type {cli} from './lmstudio'; +import {cacheScalePair} from '../src/cache-quant'; +import type {CacheQuant} from '../src/cache-quant'; +import {readableCliError} from './cli-output'; + +// Why a load failed, in terms of the settings that caused it. +// +// When LM Studio cannot fit a model it says so in its own words, which name no setting this app +// controls. A run that asked for four concurrent requests at 8192 context was really asking for a +// 32768-token cache, and the message that comes back does not mention concurrency, context, or the +// cache at all — so the obvious reading is "this model is too big", when the model was fine and +// the highest concurrency level was what did not fit. +// +// None of this changes whether a load succeeds. It changes what the run log says when one does not. + +export type LoadEstimate={gpuGiB:number|null;totalGiB:number|null;confidence:string;verdict:string}; + +// `lms load --estimate-only` prints a short block rather than JSON. Anything it does not print is +// left null rather than defaulted, because a wrong number here would be worse than no number. +export function parseEstimate(text:string):LoadEstimate{ + const clean=readableCliError(text); + const gib=(label:string)=>{ + const m=clean.match(new RegExp(label+'\\s*:?\\s*([\\d.,]+)\\s*(GiB|MiB|GB|MB)','i')); + if(!m)return null; + const n=Number(m[1].replace(/,/g,'')); + if(!Number.isFinite(n))return null; + return /MiB|MB/i.test(m[2])?n/1024:n; + }; + return { + gpuGiB:gib('Estimated GPU Memory'), + totalGiB:gib('Estimated Total Memory'), + confidence:(clean.match(/Confidence:\s*([A-Za-z]+)/i)?.[1]??'unreported').toUpperCase(), + verdict:clean.match(/Estimate:\s*([^.]+\.)/i)?.[1]?.trim()??'', + }; +} + +// Whether a failed load looks like it ran out of room, as opposed to the many other reasons a load +// fails — a missing prediction head, a bad flag, a model LM Studio cannot parse. Memory advice on +// one of those would be confidently wrong, and would send someone to lower a context length that +// was never the problem. When in doubt this says no and the error is passed through untouched. +export function looksLikeMemory(message:string):boolean{ + const text=String(message??''); + if(/draft model|draft mtp|projector|mmproj|unknown option|unrecognized/i.test(text))return false; + return /out of memory|insufficient|not enough|no ?t? ?fit|does not fit|cannot fit|too large|allocat\w*fail|failed to allocate|vram|guardrail|resource/i.test(text); +} + +export async function estimateLoad(settings:Settings,run:typeof cli,key:string,context:number,parallel:number,signal?:AbortSignal):Promise{ + try{return parseEstimate(await run(settings,['load',key,'--context-length',String(context),'--parallel',String(parallel),'--estimate-only','--yes'],signal));} + catch{return null;} // A diagnostic that cannot be taken is not itself a failure. +} + +const gib=(bytes:number)=>bytes/1024**3; + +// The arithmetic the run already did, said out loud, plus the three levers that change it. The +// cache figure is the one most worth naming: it is the only term that grows with concurrency, and +// it is the one nobody expects, because the run asked for 8192 and got four times that. +export function loadAdvice(model:Model,context:number,parallel:number,cacheK:CacheQuant='off',cacheV:CacheQuant='off',estimate?:LoadEstimate|null):string{ + const total=context*parallel; + const parts:string[]=[]; + parts.push(`This load asked for ${total.toLocaleString()} context tokens: ${context.toLocaleString()} per request × ${parallel} parallel slot${parallel===1?'':'s'}, because LM Studio serves every slot from one shared cache and the run's highest concurrency level is ${parallel}.`); + if(model.size_bytes>0)parts.push(`The weights are ${gib(model.size_bytes).toFixed(2)} GiB; the key/value cache is on top of that and is the part that grows with concurrency.`); + if(estimate?.gpuGiB!=null)parts.push(`LM Studio estimated ${estimate.gpuGiB.toFixed(2)} GiB of GPU memory for it${estimate.confidence!=='unreported'?` (its own confidence: ${estimate.confidence.toLowerCase()})`:''}.`); + const quantised=cacheK!=='off'||cacheV!=='off'; + parts.push('Three things change it: lower the context length, drop the highest concurrency level' + +(quantised + ?`, or quantize the cache further — it is currently at about ${Math.round(cacheScalePair(cacheK,cacheV)*100)}% of an f16 cache.` + :', or quantize the context (KV cache) — q8_0 roughly halves it and q4_0 roughly quarters it, which is often the difference between a model that fits and one that spills into system RAM at a fraction of the speed.')); + return parts.join(' '); +} diff --git a/electron/main.ts b/electron/main.ts index fce4149..b82cd35 100644 --- a/electron/main.ts +++ b/electron/main.ts @@ -4,13 +4,15 @@ import { mkdirSync, writeFileSync, renameSync } from 'node:fs'; import { randomUUID } from 'node:crypto'; import path from 'node:path'; import os from 'node:os'; -import { mtpArgs } from './mtp'; +import { assertMtpPlan } from './mtp'; import { visionArgs, projectorToggle, noProjectorToggleNote } from './vision'; import { visionTests } from './vision-image'; import { Store } from './store'; import { defaultSettings, starterTests, performanceTest } from '../src/defaults'; +import { awaitIndexed, createTextOnly, planTextOnly, removeTextOnly, textOnlyTwins } from './text-only'; import type { Run, RunConfig, Settings, PublicSettings, SettingsUpdate, TestCase, Sample, Progress } from '../src/types'; import { cli, listModels, resolveLms } from './lmstudio'; +import { listRuntimes } from './runtime'; import { validateConfig, validateSettings, validateTest } from './validation'; import { gradingPackage, parseGrade } from './scoring'; import { exportText } from './export'; @@ -102,7 +104,7 @@ function launch(run:Run,retries?:Sample[],gradeOnly=false){assertIdle();activeRu } async function newRun(config:RunConfig,retries?:Sample[],source?:Run){ assertIdle();validateConfig(config,allPacks());const models=await listModels(settings());for(const key of config.modelKeys)if(!models.some(m=>m.key===key))throw Error('Selected model no longer available: '+key); - for(const model of models.filter(m=>config.modelKeys.includes(m.key))){mtpArgs(model,config.mtp,config.mtpDraftTokens??2);visionArgs(model,config.vision);} + for(const model of models.filter(m=>config.modelKeys.includes(m.key))){assertMtpPlan(model,config);visionArgs(model,config.vision);} const tests=source?.tests??[...(config.mode!=='quality'?config.performanceLengths.map(performanceTest):[]),...(config.mode!=='performance'?(config.benchmark?selectBenchmark(config.benchmark,allPacks()):store.tests().filter(t=>config.testIds.includes(t.id))):[]),...(config.vision==='on'?visionTests():[])]; if(!tests.length)throw Error('Select at least one available test.');if(config.mode!=='performance'&&!source&&!config.benchmark&&config.testIds.some(id=>!tests.some(t=>t.id===id)))throw Error('A selected test was deleted. Refresh your selection.'); const now=new Date().toISOString();let runtime='Unavailable';try{runtime=await cli(settings(),['runtime','ls']);}catch{} @@ -116,10 +118,23 @@ if(locked)app.whenReady().then(()=>{ try{store=Store.open(path.join(dataPath,'bench.sqlite'));store.recover();}catch(e){return fatal('The saved benchmark database could not be opened.',e);} if(!store.get('seeded')){starterTests.forEach(t=>store.saveTest(t));store.set('seeded',true);} if(!store.get('accounting-v1-seeded')){const ids=new Set(store.tests().map(t=>t.id));starterTests.filter(t=>t.category==='Accounting'&&!ids.has(t.id)).forEach(t=>store.saveTest(t));store.set('accounting-v1-seeded',true);} + // Tests are seeded once and then belong to the user, so a change to a starter test's scoring has + // to be migrated rather than assumed. This one carries allowCodeFence onto the stored copies of + // the three JSON tests whose prompts never asked for an unfenced answer; anything the user has + // since edited keeps its own rules and prompt, and the accounting pack is untouched. + if(!store.get('fence-tolerance-v1')){ + const tolerant=new Set(starterTests.filter(t=>t.allowCodeFence).map(t=>t.id)); + for(const t of store.tests())if(tolerant.has(t.id)&&!t.allowCodeFence)store.saveTest({...t,allowCodeFence:true}); + store.set('fence-tolerance-v1',true); + } handle('snapshot',()=>({settings:publicSettings(),tests:store.tests(),runs:store.list(),progress,dataPath,packs:allPacks().map(describePack)})); handle('modelProfiles',()=>store.profiles()); handle('saveModelProfile',(key,profile)=>store.saveProfile(key,profile)); handle('models',()=>listModels(settings())); + // The installed llama.cpp builds, so a run can name the one it wants. On an AMD card the ROCm + // and Vulkan builds can differ by a large factor on prompt processing, and which one produced a + // finished run's numbers is not recoverable afterwards unless the run recorded it. + handle('runtimes',()=>listRuntimes(settings(),cli)); handle('startServer',async()=>{assertIdle();const s=settings();const u=new URL(s.baseUrl);resolveLms(s);return cli(s,['server','start','--port',u.port||'1234']);}); handle('saveSettings',(s:SettingsUpdate)=>{assertIdle();validateSettings(s);const current=storedSettings(); // An omitted token means "leave the stored one alone". The window never @@ -222,6 +237,16 @@ if(locked)app.whenReady().then(()=>{ setTimeout(()=>app.quit(),400); return updateState.file; }); + // A vision projector cannot be detached at load time (see text-only.ts), so the only honest + // answer is a second, projector-free entry for LM Studio to index. Both actions are refused + // while a run is in flight, because either changes what LM Studio lists. + handle('textOnlyPlan',async(key:string)=>{const model=(await listModels(settings())).find(m=>m.key===key);if(!model)throw Error('Downloaded model not found: '+key);return planTextOnly(model);}); + handle('makeTextOnly',async(key:string)=>{assertIdle();const model=(await listModels(settings())).find(m=>m.key===key);if(!model)throw Error('Downloaded model not found: '+key); + const folder=createTextOnly(planTextOnly(model)); + if(!await awaitIndexed(folder))throw Error(`The copy was made at ${folder}, but LM Studio has not listed it yet. Refresh the model library in a moment; if it never appears, check that folder is inside LM Studio's models directory.`); + return folder;}); + handle('textOnlyTwins',()=>textOnlyTwins()); + handle('removeTextOnly',async(folder:string)=>{assertIdle();removeTextOnly(folder);await awaitIndexed(folder,undefined,20000,150,false);}); handle('openData',()=>shell.openPath(dataPath)); win=new BrowserWindow({width:1440,height:960,minWidth:1050,minHeight:720,title:'Local Model Bench',icon:path.join(__dirname,'../assets/icon.png'),backgroundColor:'#101416',autoHideMenuBar:true,webPreferences:{preload:path.join(__dirname,'preload.cjs'),nodeIntegration:false,contextIsolation:true,sandbox:true}}); win.webContents.setWindowOpenHandler(()=>({action:'deny'}));win.webContents.on('will-navigate',e=>e.preventDefault()); diff --git a/electron/metrics.ts b/electron/metrics.ts index 861d1a0..2f6cd40 100644 --- a/electron/metrics.ts +++ b/electron/metrics.ts @@ -4,9 +4,12 @@ export function metrics(stats:Record,durationMs:number,start:num const input=finite(stats.input_tokens),elapsed=start!==null&&end!==null&&end>start?end-start:null; return {inputTokens:input,outputTokens:finite(stats.total_output_tokens),reasoningTokens:finite(stats.reasoning_output_tokens),generationTps:finite(stats.tokens_per_second),prefillTps:input!==null&&input>0&&elapsed!==null&&elapsed>=5?input/(elapsed/1000):null,prefillMs:elapsed,ttftMs:finite(stats.time_to_first_token_seconds)===null?null:Number(stats.time_to_first_token_seconds)*1000,durationMs,firstContentMs,prefillMethod:elapsed!==null&&elapsed>=5?'Client-timed prompt-processing events (estimate)':'Unavailable: missing or too-short prompt-processing interval',cacheNote:'Input tokens include formatting; cache reuse and stream buffering are not observable. This is not engine-level uncached prefill speed.'}; } -export function waveMetrics(id:string,runId:string,modelKey:string,testId:string,concurrency:number,durationMs:number,samples:Sample[],gpu:GpuStats|null=null):Wave{ +export function waveMetrics(id:string,runId:string,modelKey:string,testId:string,concurrency:number,durationMs:number,samples:Sample[],gpu:GpuStats|null=null,draft:Wave['draft']=null):Wave{ const good=samples.filter(s=>s.status==='completed');const tokens=good.every(s=>s.metrics.outputTokens!==null)?good.reduce((n,s)=>n+s.metrics.outputTokens!,0):null; - return {id,runId,modelKey,testId,concurrency,durationMs,outputTokens:tokens,throughput:tokens!==null&&durationMs>0?tokens/(durationMs/1000):null,completed:good.length,failed:samples.length-good.length,gpu}; + // The MTP depth comes from the requests the wave is made of rather than being passed in, so a + // wave can never disagree with the responses it summarises about which depth produced them. + const depth=samples[0]?.mtpTokens; + return {id,runId,modelKey,testId,concurrency,durationMs,outputTokens:tokens,throughput:tokens!==null&&durationMs>0?tokens/(durationMs/1000):null,completed:good.length,failed:samples.length-good.length,gpu,...(draft?{draft}:{}),...(typeof depth==='number'?{mtpTokens:depth}:{})}; } export function percentile(values:number[],p:number){if(!values.length)return null;const sorted=[...values].sort((a,b)=>a-b);return sorted[Math.max(0,Math.ceil(p*sorted.length)-1)];} export function average(values:(number|null)[]){const xs=values.filter((v):v is number=>v!==null&&Number.isFinite(v));return xs.length?xs.reduce((a,b)=>a+b,0)/xs.length:null;} diff --git a/electron/mtp-sidecar.ts b/electron/mtp-sidecar.ts new file mode 100644 index 0000000..c2f22e2 --- /dev/null +++ b/electron/mtp-sidecar.ts @@ -0,0 +1,60 @@ +import {existsSync,mkdirSync,readFileSync,readdirSync,rmdirSync,unlinkSync,writeFileSync} from 'node:fs'; +import {homedir} from 'node:os'; +import path from 'node:path'; +import type {Model} from '../src/types'; + +// Loading a separate MTP head. Proving one was loaded is engine-log.ts's job. +// +// LM Studio has no command-line flag and no REST field for a sidecar MTP head (see mtp.ts). +// The one route in is the per-model default config LM Studio keeps for itself under +// .lmstudio/.internal, which `lms load` merges into every load of that exact file. So the +// setting is written immediately before a load and the file is put back byte for byte +// immediately after it, whether the load succeeded or failed; a file this app created is +// deleted again, along with any folders it had to make. Nothing is left behind for LM Studio +// to apply to a load this app did not make. +// +// LM Studio's key names, read from its own load-config schema (llmLlamaMoeLoadConfigSchematics). +const SIDECAR='llm.load.llama.speculativeDecoding.draftMtpSidecar'; +const DRAFT_MODEL='llm.load.llama.speculativeDecoding.draftModel'; +const MAX_TOKENS='llm.load.llama.speculativeDecoding.draftMaxTokens'; +// LM Studio refuses a load with more than one speculative mode enabled, so the others are +// written off rather than left at whatever was last chosen in its own window. +const EXCLUSIVE=['llm.load.llama.speculativeDecoding.draftMtp','llm.load.llama.speculativeDecoding.draftSimple','llm.load.llama.speculativeDecoding.draftDflashSidecar','llm.load.llama.speculativeDecoding.draftDsparkSidecar']; +const CONFIG_ROOT=['.lmstudio','.internal','user-concrete-model-default-config']; + +// The resource comes from LM Studio's index rather than from a user, but it still becomes a +// path this app writes to, so it is checked like one. +export function sidecarConfigPath(resource:string,home=homedir()):string{ + const parts=String(resource).split(/[\\/]/).filter(Boolean); + if(!parts.length||parts.some(p=>p==='.'||p==='..'||/^[A-Za-z]:$/.test(p)))throw Error('LM Studio returned a model resource that is not a relative path, so its MTP settings cannot be written safely.'); + return path.join(home,...CONFIG_ROOT,...parts.slice(0,-1),parts[parts.length-1]+'.json'); +} +function shape(bytes:Buffer){ + const value=JSON.parse(bytes.toString('utf8')); + if(!value||typeof value!=='object'||Array.isArray(value))throw Error('LM Studio’s stored settings for this model are not a configuration object. No measurements were taken.'); + const load=value.load&&typeof value.load==='object'?value.load:{}; + return {...value,operation:value.operation??{fields:[]},load:{...load,fields:Array.isArray(load.fields)?load.fields:[]}}; +} +function pruneEmpty(dir:string,stop:string){ + for(let at=dir;at.length>stop.length&&at.startsWith(stop);at=path.dirname(at)){ + try{if(readdirSync(at).length)return;rmdirSync(at);}catch{return;} + } +} +export type Restore=()=>void; +export function applySidecar(model:Model,tokens:number,home=homedir()):Restore{ + const mtp=model.nativeMtp; + if(mtp?.kind!=='sidecar'||!mtp.resource||!mtp.draftResource)throw Error(`${model.display_name}: no paired MTP head to load.`); + const file=sidecarConfigPath(mtp.resource,home),existed=existsSync(file),original=existed?readFileSync(file):null; + const config=original?shape(original):{preset:'',operation:{fields:[]},load:{fields:[]}}; + const keep=(config.load.fields as {key?:unknown}[]).filter(f=>f&&typeof f.key==='string'&&f.key!==SIDECAR&&f.key!==DRAFT_MODEL&&f.key!==MAX_TOKENS&&!EXCLUSIVE.includes(f.key)); + config.load.fields=[...keep,...EXCLUSIVE.map(key=>({key,value:false})),{key:SIDECAR,value:true},{key:DRAFT_MODEL,value:mtp.draftResource},{key:MAX_TOKENS,value:tokens}]; + mkdirSync(path.dirname(file),{recursive:true}); + writeFileSync(file,JSON.stringify(config,null,2)); + let done=false; + return ()=>{ + if(done)return;done=true; + if(original)writeFileSync(file,original); + else{try{unlinkSync(file);}catch{}pruneEmpty(path.dirname(file),path.join(home,...CONFIG_ROOT));} + }; +} + diff --git a/electron/mtp.ts b/electron/mtp.ts index 5e8aeea..d58fc02 100644 --- a/electron/mtp.ts +++ b/electron/mtp.ts @@ -2,39 +2,112 @@ import {readFileSync,statSync} from 'node:fs'; import {homedir} from 'node:os'; import path from 'node:path'; import type {Model} from '../src/types'; +import {sweepSteps,type MtpConfig} from '../src/mtp-sweep'; -// The v1 model list currently omits supportsMtp. Use LM Studio's own local -// GGUF metadata, matched to an exact model resource and unchanged file stats. -// No architecture/name heuristics and no arbitrary draft-model pairings. +// LM Studio ships native MTP in two shapes, and only one of them is reachable from the +// command line. Verified on 2026-09-15 against lms CLI commit 07b7252 and LM Studio's own +// load path: +// * bundled — the prediction heads sit inside the model's own GGUF. LM Studio's metadata +// sets supportsMtp true and `--speculative-draft-mtp` loads them. Qwen3.8-27B is one. +// * sidecar — the heads ship as a separate GGUF stored with the model, which LM Studio +// indexes in its own "drafter" domain. supportsMtp is false on both files; +// `--speculative-draft-mtp` refuses with "requires a GGUF model with a bundled supported +// MTP head"; and `--speculative-draft-model` cannot name the head because that flag +// resolves only in the llm domain. LM Studio loads it through a load-config field, +// draftMtpSidecar, that neither the CLI nor POST /api/v1/models/load exposes at all. +// Gemma 4 ships its heads this way. See mtp-sidecar.ts for how the load is made. +// The v1 model list reports neither shape, so both are read from LM Studio's own local GGUF +// metadata, matched to an exact model resource and unchanged file stats. No architecture or +// name heuristics, and no arbitrary draft-model pairings. + +type Meta=Record; +const slash=(p:string)=>p.replaceAll('\\','/'); +// A cached record only describes the file it was read from while that file still has the +// size and modification time the cache recorded. Anything else is stale, not evidence. +function verified(cache:any,resource:string):{file:string;metadata:Meta}|null{ + const rows=cache.json.map.filter(([file]:[string])=>slash(file).endsWith('/'+resource)); + if(rows.length!==1)return null; + const [file,record]=rows[0]; + let stats;try{stats=statSync(file);}catch{return null;} + if(stats.size!==record.fileSizeBytes||Math.abs(stats.mtimeMs-record.mtimeMs)>1)return null; + return {file:slash(file),metadata:record.metadata&&typeof record.metadata==='object'?record.metadata:{}}; +} +// A head is paired with a model by structure, never by file name: it has to be stored inside +// that model's own folder, declare prediction layers, project into exactly that model's +// hidden size, and carry that model's architecture under LM Studio's "-assistant" suffix. +function sidecarHeads(index:any,cache:any,mainFile:string,main:Meta){ + const dir=mainFile.slice(0,mainFile.lastIndexOf('/')+1),heads:{resource:string;file:string}[]=[]; + if(typeof main.arch!=='string'||!Number.isInteger(main.embeddingLength))return heads; + for(const entry of index.models){ + if(entry?.domain!=='drafter'||entry.format!=='gguf'||typeof entry.indexedModelIdentifier!=='string')continue; + const file=slash(entry.entryPoint?.absPath??''); + if(!file.startsWith(dir))continue; + const head=verified(cache,entry.indexedModelIdentifier); + if(!head||head.file!==file)continue; + if(!Number.isInteger(head.metadata.nextnPredictLayers)||head.metadata.nextnPredictLayers<1)continue; + if(head.metadata.embeddingLengthOut!==main.embeddingLength||head.metadata.arch!==main.arch+'-assistant')continue; + heads.push({resource:entry.indexedModelIdentifier,file}); + } + return heads; +} export function attachMtp(models:Model[],home=homedir()):Model[]{ let index:any,cache:any; try {const dir=path.join(home,'.lmstudio','.internal');index=JSON.parse(readFileSync(path.join(dir,'model-index-cache.json'),'utf8'));cache=JSON.parse(readFileSync(path.join(dir,'gguf-metadata-cache.json'),'utf8'));}catch{return models;} return models.map(model=>{ - const unknown={...model,nativeMtp:{supported:null,reason:'Native MTP metadata unavailable or stale'}}; + const unknown={...model,nativeMtp:{supported:null,reason:'Native MTP metadata unavailable or stale'}} as Model; try{ const matches=index.models.filter((m:any)=>m.defaultIdentifier===model.key&&m.sizeBytes===model.size_bytes&&m.format===model.format); if(matches.length!==1)return unknown; - const m=matches[0]; - const resource=m.indexedModelIdentifier; - const rows=cache.json.map.filter(([file]:[string])=>file.replaceAll('\\','/').endsWith('/'+resource)); - if(rows.length!==1)return unknown; - const [file,record]=rows[0],stats=statSync(file); - if(stats.size!==record.fileSizeBytes||Math.abs(stats.mtimeMs-record.mtimeMs)>1)return unknown; - const supported=record.metadata.supportsMtp; - if(typeof supported!=='boolean'||m.format!=='gguf')return unknown; - return {...model,nativeMtp:{supported,resource,reason:supported?'LM Studio GGUF metadata confirms built-in MTP':'This GGUF does not contain supported built-in MTP'}}; + const entry=matches[0],resource=entry.indexedModelIdentifier; + const main=verified(cache,resource); + if(!main||entry.format!=='gguf')return unknown; + if(main.metadata.supportsMtp===true)return {...model,nativeMtp:{supported:true,kind:'bundled',resource,reason:'LM Studio GGUF metadata confirms MTP heads built into this file'}}; + if(typeof main.metadata.supportsMtp!=='boolean')return unknown; + const heads=sidecarHeads(index,cache,main.file,main.metadata); + if(heads.length===1)return {...model,nativeMtp:{supported:true,kind:'sidecar',resource,draftResource:heads[0].resource,draftPath:heads[0].file, + reason:`This GGUF has no built-in MTP heads, but LM Studio indexes a matching MTP head stored with it (${heads[0].resource}) that is loaded alongside the model.`}}; + // Two heads that both fit cannot be told apart by anything this app can see, and guessing + // one would be exactly the arbitrary pairing the rest of this file refuses to make. + if(heads.length>1)return {...model,nativeMtp:{supported:null,resource, + reason:`More than one MTP head fits this model (${heads.map(h=>h.resource).join(', ')}). Remove the duplicate in LM Studio so one head can be paired without guessing.`}}; + return {...model,nativeMtp:{supported:false,resource,reason:'This GGUF contains no built-in MTP heads, and no matching MTP head is stored with it.'}}; }catch{return unknown;} }); } export function mtpArgs(model:Model,mode:'on'|'off'|undefined,tokens=2):string[]{ if(mode===undefined)return []; // Historical run configurations retain their original behavior. - if(mode==='on'&&model.nativeMtp?.supported!==true)throw Error(`${model.display_name}: native MTP support is not confirmed for this exact file. Choose MTP off or a supported model.`); + if(mode==='on'&&model.nativeMtp?.supported!==true)throw Error(`${model.display_name}: native MTP support is not confirmed for this exact file. ${model.nativeMtp?.reason??''} Choose MTP off or a supported model.`.replace(/\s+/g,' ')); if(mode==='off'&&model.format!=='gguf')return []; - return mode==='on'?['--speculative-draft-mtp','--speculative-draft-max-tokens',String(tokens)]:['--no-speculative-draft-mtp']; + // A sidecar head has no command-line flag at all, so its load carries no extra arguments: + // the draft-token count travels with the head in the config written by mtp-sidecar.ts. + if(mode==='on')return model.nativeMtp?.kind==='sidecar'?[]:['--speculative-draft-mtp','--speculative-draft-max-tokens',String(tokens)]; + return ['--no-speculative-draft-mtp']; } -export function verifyMtp(config:Record,mode:'on'|'off'|undefined,format:unknown,tokens=2):void{ +// What LM Studio said about the load, gathered from both places it says anything: the loaded +// instance's own reported config, and the engine log line naming a draft model it opened. +export type MtpEvidence={kind?:'bundled'|'sidecar';draftPath?:string;draftersLoaded:string[]}; +export function verifyMtp(config:Record,mode:'on'|'off'|undefined,format:unknown,tokens=2,evidence:MtpEvidence={draftersLoaded:[]}):void{ if(mode===undefined||mode==='off'&&format!=='gguf')return; - if(config.speculative_draft_mtp!==(mode==='on'))throw Error(`LM Studio did not confirm MTP ${mode}. No measurements were taken. Update LM Studio or check the model's native MTP support.`); if(config.speculative_draft_simple===true)throw Error('A separate draft model is active. Disable Draft Simple in LM Studio before benchmarking native MTP. No measurements were taken.'); + if(mode==='on'&&evidence.kind==='sidecar'){ + // A sidecar leaves no trace in the instance config LM Studio reports: its REST API has no + // field for one. The single piece of evidence it does give is the engine's own load line + // naming the head file, so that line is required, and required to name this exact head. + const wanted=slash(evidence.draftPath??'').toLowerCase(); + if(!wanted||!evidence.draftersLoaded.includes(wanted))throw Error(`LM Studio did not report loading the MTP head for this model (${evidence.draftPath??'unknown head'}). No measurements were taken.${evidence.draftersLoaded.length?` It loaded ${evidence.draftersLoaded.join(', ')} instead.`:''}`); + if(config.speculative_draft_mtp===true)throw Error('LM Studio reported built-in MTP as well as a separate MTP head. No measurements were taken.'); + if(config.speculative_draft_max_tokens!==tokens)throw Error('LM Studio did not apply the requested MTP draft-token count. No measurements were taken.'); + return; + } + if(config.speculative_draft_mtp!==(mode==='on'))throw Error(`LM Studio did not confirm MTP ${mode}. No measurements were taken. Update LM Studio or check the model's native MTP support.`); if(mode==='on'&&config.speculative_draft_max_tokens!==tokens)throw Error('LM Studio did not apply the requested MTP draft-token count. No measurements were taken.'); + // MTP off has to mean no drafting at all, including a head this app never asked for. A + // setting left behind in LM Studio would otherwise turn a sweep's baseline into a + // speculative run and quietly flatter every depth measured against it. + if(mode==='off'&&evidence.draftersLoaded.length)throw Error(`LM Studio loaded a draft model (${evidence.draftersLoaded.join(', ')}) for a load with MTP off. No measurements were taken. Clear this model's speculative decoding settings in LM Studio.`); +} +// Every depth a sweep will load has to be possible before the first one is loaded. A run that +// would only fail on its fourth depth, an hour into measuring, has wasted the hour. +export function assertMtpPlan(model:Model,config:MtpConfig):void{ + for(const step of sweepSteps(config))mtpArgs(model,step.mode,step.tokens); } diff --git a/electron/preload.ts b/electron/preload.ts index 5986d43..8f5da03 100644 --- a/electron/preload.ts +++ b/electron/preload.ts @@ -1,5 +1,5 @@ import {contextBridge,ipcRenderer} from 'electron'; import type { BenchAPI } from '../src/types'; const call=(name:string,...args:unknown[])=>ipcRenderer.invoke('bench:'+name,...args); -const api:BenchAPI={modelProfiles:()=>call('modelProfiles'),saveModelProfile:(key,p)=>call('saveModelProfile',key,p),snapshot:()=>call('snapshot'),models:()=>call('models'),startServer:()=>call('startServer'),saveSettings:s=>call('saveSettings',s),saveTest:t=>call('saveTest',t),deleteTest:id=>call('deleteTest',id),choosePackFile:()=>call('choosePackFile'),importPack:d=>call('importPack',d),deletePack:id=>call('deletePack',id),startRun:c=>call('startRun',c),cancel:()=>call('cancel'),getRun:id=>call('getRun',id),retry:id=>call('retry',id),grade:(id,judge)=>call('grade',id,judge),copyPackage:(r,s)=>call('copyPackage',r,s),importGrade:(r,s,raw,j)=>call('importGrade',r,s,raw,j),exportRun:(id,f)=>call('exportRun',id,f),history:()=>call('history'),updateState:()=>call('updateState'),checkUpdate:()=>call('checkUpdate'),downloadUpdate:()=>call('downloadUpdate'),installUpdate:()=>call('installUpdate'),onUpdate:fn=>{const handler=(_:unknown,s:any)=>fn(s);ipcRenderer.on('bench:update',handler);return ()=>ipcRenderer.removeListener('bench:update',handler);},openData:()=>call('openData'),onProgress:fn=>{const handler=(_:unknown,p:any)=>fn(p);ipcRenderer.on('bench:progress',handler);return ()=>ipcRenderer.removeListener('bench:progress',handler);}}; +const api:BenchAPI={modelProfiles:()=>call('modelProfiles'),saveModelProfile:(key,p)=>call('saveModelProfile',key,p),snapshot:()=>call('snapshot'),models:()=>call('models'),runtimes:()=>call('runtimes'),startServer:()=>call('startServer'),saveSettings:s=>call('saveSettings',s),saveTest:t=>call('saveTest',t),deleteTest:id=>call('deleteTest',id),choosePackFile:()=>call('choosePackFile'),importPack:d=>call('importPack',d),deletePack:id=>call('deletePack',id),startRun:c=>call('startRun',c),cancel:()=>call('cancel'),getRun:id=>call('getRun',id),retry:id=>call('retry',id),grade:(id,judge)=>call('grade',id,judge),copyPackage:(r,s)=>call('copyPackage',r,s),importGrade:(r,s,raw,j)=>call('importGrade',r,s,raw,j),exportRun:(id,f)=>call('exportRun',id,f),history:()=>call('history'),updateState:()=>call('updateState'),checkUpdate:()=>call('checkUpdate'),downloadUpdate:()=>call('downloadUpdate'),installUpdate:()=>call('installUpdate'),onUpdate:fn=>{const handler=(_:unknown,s:any)=>fn(s);ipcRenderer.on('bench:update',handler);return ()=>ipcRenderer.removeListener('bench:update',handler);},textOnlyPlan:key=>call('textOnlyPlan',key),makeTextOnly:key=>call('makeTextOnly',key),textOnlyTwins:()=>call('textOnlyTwins'),removeTextOnly:folder=>call('removeTextOnly',folder),openData:()=>call('openData'),onProgress:fn=>{const handler=(_:unknown,p:any)=>fn(p);ipcRenderer.on('bench:progress',handler);return ()=>ipcRenderer.removeListener('bench:progress',handler);}}; contextBridge.exposeInMainWorld('bench',api); diff --git a/electron/runtime.ts b/electron/runtime.ts new file mode 100644 index 0000000..d423d1d --- /dev/null +++ b/electron/runtime.ts @@ -0,0 +1,102 @@ +import type {Settings} from '../src/types'; +import type {cli} from './lmstudio'; + +// Choosing which llama.cpp build runs the benchmark. +// +// LM Studio ships several engines for the same model format and uses whichever one is selected at +// the time a model is loaded. On an AMD card the choice between the Vulkan build and the ROCm +// build can change prompt processing substantially, and nothing in a saved run used to record +// which one produced the numbers — so two runs of the same model on the same machine could differ +// by a large factor for a reason the history had no field for. +// +// The selection is global to LM Studio rather than per-load: there is no `lms load --engine`. So a +// run that asks for a specific engine selects it before loading and puts the previous selection +// back when the run ends, the same discipline the MTP sidecar uses for the same reason. + +// `llama.cpp-win-x86_64-amd-rocm-avx2@2.40.0` — the part before the @ is the engine, after it the +// version. `lms runtime select` wants the whole thing. +export type Runtime={engine:string;version:string;ref:string;selected:boolean;format:string}; + +// Reused from cli-output.ts's reasoning: `lms` colours its tables, and a colour code between the +// engine name and the tick would otherwise read as part of one or the other. +const ANSI=/\x1b\[[0-9;?]*[A-Za-z]|\x1b[()][A-Za-z0-9]|[\x00-\x08\x0b-\x1f\x7f]/g; + +export function parseRuntimes(text:string):Runtime[]{ + const out:Runtime[]=[]; + for(const raw of String(text??'').replace(ANSI,'').split(/\r?\n/)){ + const line=raw.trim(); + if(!line||/^LLM ENGINE\b/i.test(line))continue; + const at=line.match(/^(\S+?)@(\S+)\s*(.*)$/); + if(!at)continue; + const [,engine,version,rest]=at; + if(!engine.includes('llama.cpp')&&!engine.includes('mlx'))continue; + out.push({engine,version,ref:`${engine}@${version}`,selected:/[✓✔*]/.test(rest),format:(rest.replace(/[✓✔*]/g,'').trim().split(/\s+/)[0])||''}); + } + return out; +} + +// What to call an engine in the window. The full identifier is exact but unreadable in a dropdown, +// and the accelerator is the only part of it anyone is choosing between. +export function runtimeLabel(engine:string):string{ + const name=engine.toLowerCase(); + if(name.includes('rocm'))return 'ROCm'; + if(name.includes('vulkan'))return 'Vulkan'; + if(name.includes('cuda'))return 'CUDA'; + if(name.includes('metal')||name.includes('mlx'))return 'Metal'; + return 'CPU'; +} + +// Two builds of the same accelerator differ only by version, so the label alone cannot identify +// one. The version is shown whenever more than one build of the same accelerator is installed. +export function runtimeChoices(runtimes:Runtime[]):{ref:string;label:string}[]{ + const counts=new Map(); + for(const r of runtimes)counts.set(runtimeLabel(r.engine),(counts.get(runtimeLabel(r.engine))??0)+1); + return runtimes.map(r=>({ref:r.ref,label:(counts.get(runtimeLabel(r.engine))??0)>1?`${runtimeLabel(r.engine)} ${r.version}`:runtimeLabel(r.engine)})); +} + +// Which engine produced a saved run's numbers. +// +// Two places know. A run that named an engine carries it in its config, which is exact. Every other +// run carries the `lms runtime ls` table captured when it was created, in which exactly one row is +// ticked — that is what LM Studio would have used. Runs made before either existed report Unknown +// rather than being quietly attributed to whatever is selected today, because that would make a +// backend comparison out of measurements nobody took under a known backend. +export const unknownBackend='Unknown engine'; +export type RunBackend={label:string;ref:string|null;version:string|null}; +export function backendOf(run:{config?:{runtime?:string};environment?:Record}):RunBackend{ + const chosen=run.config?.runtime; + if(typeof chosen==='string'&&chosen.includes('@')){ + const [engine,version]=chosen.split('@'); + return {label:runtimeLabel(engine),ref:chosen,version:version||null}; + } + const listing=run.environment?.runtime; + if(typeof listing==='string'&&listing.trim()){ + const selected=parseRuntimes(listing).find(r=>r.selected); + if(selected)return {label:runtimeLabel(selected.engine),ref:selected.ref,version:selected.version}; + } + return {label:unknownBackend,ref:null,version:null}; +} + +export type Cli=typeof cli; +export async function listRuntimes(settings:Settings,run:Cli,signal?:AbortSignal):Promise{ + return parseRuntimes(await run(settings,['runtime','ls'],signal)); +} + +// Returns a restore function rather than nothing, so the caller cannot forget that selecting an +// engine changed something outside this app that has to be put back. +export async function selectRuntime(settings:Settings,run:Cli,ref:string,signal?:AbortSignal):Promise<()=>Promise>{ + const before=await listRuntimes(settings,run,signal); + const current=before.find(r=>r.selected); + const wanted=before.find(r=>r.ref===ref); + if(!wanted)throw Error(`LM Studio has no installed runtime ${ref}. Install it in LM Studio, or choose one of: ${before.map(r=>r.ref).join(', ')||'none reported'}.`); + if(current?.ref===ref)return async()=>{}; // Already selected; nothing to put back. + await run(settings,['runtime','select',ref],signal); + // Confirmed rather than assumed: `lms runtime select` exits zero for a name it did not apply. + const after=await listRuntimes(settings,run,signal); + if(!after.find(r=>r.ref===ref)?.selected)throw Error(`LM Studio did not switch to ${ref}. No measurements were taken.`); + let done=false; + return async()=>{ + if(done||!current)return;done=true; + try{await run(settings,['runtime','select',current.ref]);}catch{/* Restoring is best effort; the run's own result matters more. */} + }; +} diff --git a/electron/scoring.ts b/electron/scoring.ts index 365d2d4..08bcad2 100644 --- a/electron/scoring.ts +++ b/electron/scoring.ts @@ -1,12 +1,16 @@ import type { TestCase, Objective, Grade, Sample } from '../src/types'; -import { parseAnswer, equalAnswer } from './json-answer'; +import { parseAnswer, equalAnswer, unfence } from './json-answer'; import {finalNumber,scoreInstructions,GraderError} from './benchmark-scoring'; export function objectiveScore(output:string,test:TestCase):Objective { const checks=test.rules.map(r=>{ let passed=false,detail='',unscorable=false; try { const s=output.trim(); + // JSON-shaped checks read the answer with a single surrounding code fence removed; every other + // check still sees exactly what the model wrote. + const {json:j,fenced}=test.allowCodeFence?unfence(s):{json:s,fenced:false}; + const note=fenced?' (unwrapped from a code fence)':''; switch(r.type){ case 'final-number': {const n=finalNumber(s);passed=n!==null&&n===Number(r.expected);detail=passed?'Correct final numeric answer':`Expected ${r.expected}; received ${n??'no valid final number'}`;break;} case 'ifeval': {const result=scoreInstructions(s,JSON.parse(r.expected??'[]'));passed=result.passed;detail=result.detail;break;} @@ -14,9 +18,9 @@ export function objectiveScore(output:string,test:TestCase):Objective { case 'contains': passed=s.toLocaleLowerCase().includes((r.expected??'').toLocaleLowerCase());detail=passed?'Required text present':'Required text missing';break; case 'heading': passed=s.split(/\r?\n/).some(line=>line.trim().replace(/^#{1,6}\s+/, '').replace(/^\*\*(.*?)\*\*$/, '$1').replace(/:$/, '').trim().toLowerCase()===(r.expected??'').trim().toLowerCase());detail=passed?'Standalone heading found':'Expected a standalone plain or Markdown heading';break; case 'number': { const n=Number(s); passed=s!==''&&Number.isFinite(n)&&Math.abs(n-Number(r.expected))<=(r.tolerance??0);detail=passed?'Within tolerance':'Expected a single numeric answer within tolerance';break; } - case 'json-equal': {const mismatch=equalAnswer(parseAnswer(s),parseAnswer(r.expected??''));passed=mismatch===null;detail=mismatch??'All required values match exactly';break;} - case 'json': JSON.parse(s);passed=true;detail='Valid JSON without surrounding prose';break; - case 'field': {let actual:any=JSON.parse(s);for(const p of (r.path??'').split('.')){if(p==='__proto__'||p==='constructor'||p==='prototype')throw Error('Invalid path');actual=actual?.[p];}let expected:any=r.expected;try{expected=JSON.parse(r.expected??'');}catch{}passed=JSON.stringify(actual)===JSON.stringify(expected);detail=`Actual: ${JSON.stringify(actual)??'missing'}; expected: ${JSON.stringify(expected)}`;break;} + case 'json-equal': {const mismatch=equalAnswer(parseAnswer(j),parseAnswer(r.expected??''));passed=mismatch===null;detail=mismatch??'All required values match exactly'+note;break;} + case 'json': JSON.parse(j);passed=true;detail='Valid JSON without surrounding prose'+note;break; + case 'field': {let actual:any=JSON.parse(j);for(const p of (r.path??'').split('.')){if(p==='__proto__'||p==='constructor'||p==='prototype')throw Error('Invalid path');actual=actual?.[p];}let expected:any=r.expected;try{expected=JSON.parse(r.expected??'');}catch{}passed=JSON.stringify(actual)===JSON.stringify(expected);detail=`Actual: ${JSON.stringify(actual)??'missing'}; expected: ${JSON.stringify(expected)}`;break;} case 'words': {const n=s?s.split(/\s+/u).length:0;passed=n>=(r.min??0)&&n<=(r.max??Infinity);detail=`${n} words; expected ${r.min??0}–${r.max??'unlimited'}`;break;} case 'lines': {const lines=s.split(/\r?\n/).filter(x=>x.trim());passed=lines.length>=(r.min??0)&&lines.length<=(r.max??Infinity)&&(!r.expected||lines.every(x=>x.startsWith(r.expected!)));detail=`${lines.length} nonempty lines${r.expected?'; required prefix: '+r.expected:''}`;break;} } diff --git a/electron/text-only.ts b/electron/text-only.ts new file mode 100644 index 0000000..d4b2512 --- /dev/null +++ b/electron/text-only.ts @@ -0,0 +1,164 @@ +import {linkSync,mkdirSync,readFileSync,readdirSync,rmSync,statSync,writeFileSync} from 'node:fs'; +import {homedir} from 'node:os'; +import path from 'node:path'; +import type {Model} from '../src/types'; + +// Benchmarking a vision-capable model without its vision projector. +// +// LM Studio attaches a projector from the model's index entry and from nothing else. Its own +// load path reduces to two lines: +// const visionPath = indexedModel.visionAdapter?.absPath; +// if (visionPath !== undefined) loadParams.mmproj_path = visionPath; +// There is no command-line flag, no field of POST /api/v1/models/load, and no key in LM +// Studio's load-config schema that can turn that off — checked against lms CLI commit 07b7252 +// and LM Studio's own bundle on 2026-09-15. A model is vision-capable exactly when an +// mmproj-*.gguf sits in its folder, so the only way to measure the same weights without the +// projector is to give LM Studio a second folder holding those weights and no projector. LM +// Studio indexes that as its own model key, reports capabilities.vision false for it, and +// loads it with no mmproj_path at all. LM Studio's own library already works this way: a +// "… - Vision" folder beside a plain one is the same GGUF indexed twice. +// +// Nothing is copied. Every file in the twin is a hard link, so it occupies no additional disk +// space and shares the original's bytes; removing it deletes a directory entry and never the +// weights. An MTP head stored with the model is linked in at the same relative place, so a +// text-only twin keeps native MTP. Hard links need a single NTFS volume, which a sibling +// folder inside the same models directory always is; anything else is refused rather than +// quietly duplicating many gigabytes. + +export const TWIN_SUFFIX=' - Text only'; +export const MARKER='.local-model-bench-text-only.json'; +const slash=(p:string)=>p.replaceAll('\\','/'); +const isProjector=(name:string)=>/^mmproj.*\.gguf$/i.test(name); + +function readIndex(home:string){ + return JSON.parse(readFileSync(path.join(home,'.lmstudio','.internal','model-index-cache.json'),'utf8')); +} +// The same identity rule mtp.ts uses: an exact key, an unchanged size, and one matching entry. +// Two entries that both answer to a key are not a model this app will act on. +export function indexEntry(model:Model,home=homedir()){ + const matches=readIndex(home).models.filter((m:any)=>m.defaultIdentifier===model.key&&m.sizeBytes===model.size_bytes&&m.format===model.format); + if(matches.length!==1)throw Error(`LM Studio does not describe exactly one stored model for ${model.display_name}. Refresh the library in LM Studio and try again.`); + return matches[0]; +} + +export type TwinFile={from:string;to:string}; +export type TwinPlan={key:string;displayName:string;sourceDir:string;folder:string;files:TwinFile[];projectors:string[];headIncluded:boolean}; + +// What making a twin would do, worked out before anything is written so the window can say it +// and so an impossible one fails with a reason rather than half a folder. +export function planTextOnly(model:Model,home=homedir()):TwinPlan{ + if(model.capabilities?.vision!==true)throw Error(`${model.display_name} has no vision projector attached, so it already loads text-only.`); + const entry=indexEntry(model,home); + const own=slash(entry.entryPoint?.absPath??''); + if(!own)throw Error('LM Studio did not say which file this model loads from.'); + const sourceDir=own.slice(0,own.lastIndexOf('/')); + if(sourceDir.endsWith(TWIN_SUFFIX))throw Error(`${model.display_name} is already a text-only copy.`); + const folder=sourceDir+TWIN_SUFFIX; + // The model's own weights, and any MTP head stored with it at the same relative place. The + // projector is the one thing deliberately left behind. + const files:TwinFile[]=[{from:own,to:path.join(folder,path.basename(own))}]; + const head=model.nativeMtp?.kind==='sidecar'?slash(model.nativeMtp.draftPath??''):''; + if(head&&head.startsWith(sourceDir+'/'))files.push({from:head,to:path.join(folder,...head.slice(sourceDir.length+1).split('/'))}); + const projectors=(()=>{try{return readdirSync(sourceDir).filter(isProjector).sort();}catch{return [];}})(); + if(!projectors.length)throw Error(`No projector file was found beside ${model.display_name}, so a text-only copy would be identical to it.`); + return {key:model.key,displayName:model.display_name,sourceDir,folder,files,projectors,headIncluded:files.length>1}; +} + +export function createTextOnly(plan:TwinPlan):string{ + if(statSync(plan.sourceDir).isDirectory()!==true)throw Error('The model folder has moved. Refresh the library in LM Studio and try again.'); + let made=false; + try{ + mkdirSync(plan.folder,{recursive:false});made=true; + for(const file of plan.files){ + mkdirSync(path.dirname(file.to),{recursive:true}); + // linkSync, never copyFileSync: a twin has to share the original's bytes, and failing + // loudly on a volume that cannot hard link is better than silently writing another 17 GB. + try{linkSync(file.from,file.to);} + catch(e){throw Error(`A hard link could not be made for ${path.basename(file.from)} (${(e as Error).message}). This needs the model folder to be on an NTFS volume. Nothing was changed.`);} + } + writeFileSync(path.join(plan.folder,MARKER),JSON.stringify({createdBy:'local-model-bench',source:plan.sourceDir,sourceKey:plan.key,created:new Date().toISOString(), + note:'Every .gguf here is a hard link to the folder named in "source" and uses no extra disk space. Deleting this folder removes the links only; the model itself is untouched.', + files:plan.files.map(f=>slash(path.relative(plan.folder,f.to)))},null,2)); + }catch(e){ + // A twin that was only half made would be indexed by LM Studio as a real model, so a + // failure takes the whole folder back out rather than leaving one. + if(made)try{rmSync(plan.folder,{recursive:true,force:true});}catch{} + throw e; + } + return plan.folder; +} + +// LM Studio notices a new folder by itself and re-indexes it in about a quarter of a second, +// measured on 2026-09-15. That is quick, but not instant, and a copy that is not indexed yet +// is not selectable yet: returning before then would hand back a model the library does not +// list. So the copy is not reported as made until LM Studio's own index holds it. +// Removal has the same lag in reverse: a copy that has just been deleted stays in the library +// until LM Studio notices, so both directions wait for the index to agree with the disk. +export async function awaitIndexed(folder:string,home=homedir(),timeoutMs=20000,tick=150,present=true):Promise{ + const prefix=slash(folder)+'/',until=Date.now()+timeoutMs; + for(;;){ + try{if(readIndex(home).models.some((m:any)=>slash(String(m.entryPoint?.absPath??'')).startsWith(prefix))===present)return true;}catch{} + if(Date.now()>=until)return false; + await new Promise(r=>setTimeout(r,tick)); + } +} + +// `key` is the twin's OWN model key, which is what a run has to select to load the weights without +// the projector. LM Studio assigns it when it indexes the folder, so it is read back from the index +// rather than derived from the folder name; null while LM Studio has not indexed the twin yet. +export type Twin={folder:string;source:string;sourceKey:string;created:string;key:string|null}; +function readMarker(folder:string):Twin|null{ + try{ + const marker=JSON.parse(readFileSync(path.join(folder,MARKER),'utf8')); + if(marker?.createdBy!=='local-model-bench'||typeof marker.source!=='string')return null; + return {folder:slash(folder),source:slash(marker.source),sourceKey:String(marker.sourceKey??''),created:String(marker.created??''),key:null}; + }catch{return null;} +} +// Twins this app made, found by their marker rather than by their name, so a folder a user +// happened to name "… - Text only" themselves is never treated as this app's to delete. +// Which model key LM Studio gave a twin folder. Matched on the indexed entry point living inside +// that folder, because the key itself is LM Studio's to choose and has no reliable spelling. +function keyForFolder(folder:string,home:string):string|null{ + try{ + const inside=slash(folder).replace(/\/+$/,'')+'/'; + const hit=readIndex(home).models.filter((m:any)=>slash(String(m?.entryPoint?.absPath??'')).startsWith(inside)); + return hit.length===1&&typeof hit[0].defaultIdentifier==='string'?hit[0].defaultIdentifier:null; + }catch{return null;} +} +export function textOnlyTwins(home=homedir()):Twin[]{ + const root=path.join(home,'.lmstudio','models'),found:Twin[]=[]; + const walk=(dir:string,depth:number)=>{ + let entries;try{entries=readdirSync(dir,{withFileTypes:true});}catch{return;} + for(const entry of entries){ + if(!entry.isDirectory())continue; + const full=path.join(dir,entry.name); + const twin=readMarker(full); + if(twin)found.push(twin); + else if(depth>0)walk(full,depth-1); + } + }; + walk(root,3); + // Resolved once here rather than per caller: the index is one file read for the whole list. + return found.map(t=>({...t,key:keyForFolder(t.folder,home)})); +} +// Removal refuses anything that is not one of this app's twins, and anything holding a file +// that exists nowhere else. Between them, a delete here can only ever drop directory entries +// whose bytes are still reachable from the model they were linked from. +export function removeTextOnly(folder:string,home=homedir()):void{ + const twin=readMarker(folder); + if(!twin)throw Error('That folder was not created by Local Model Bench, so it will not be deleted from here. Remove it in LM Studio or in Explorer if you meant to.'); + const root=slash(path.join(home,'.lmstudio','models')); + if(!slash(path.resolve(folder)).startsWith(root+'/'))throw Error('That folder is outside the LM Studio models directory.'); + const unique:string[]=[]; + const walk=(dir:string)=>{ + for(const entry of readdirSync(dir,{withFileTypes:true})){ + const full=path.join(dir,entry.name); + if(entry.isDirectory()){walk(full);continue;} + if(entry.name===MARKER)continue; + if(statSync(full).nlink<2)unique.push(slash(path.relative(folder,full))); + } + }; + walk(folder); + if(unique.length)throw Error(`This folder holds ${unique.length} file(s) that exist nowhere else (${unique.slice(0,3).join(', ')}). It was not deleted. Move anything you want to keep out of it first.`); + rmSync(folder,{recursive:true,force:true}); +} diff --git a/electron/validation.ts b/electron/validation.ts index 36bba2a..60bffa9 100644 Binary files a/electron/validation.ts and b/electron/validation.ts differ diff --git a/electron/vision.ts b/electron/vision.ts index a03e6f5..863a616 100644 --- a/electron/vision.ts +++ b/electron/vision.ts @@ -11,12 +11,22 @@ import type {Model} from '../src/types'; // flash_attention and the speculative_draft_* family used for native MTP. // * A loaded instance's returned config carries no vision or projector field at all, // so unlike MTP there is nothing to read back that could confirm a projector state. -// Vision is fixed when LM Studio indexes a model: a base GGUF is vision-capable exactly -// when an mmproj-*.gguf sits beside it in the same folder, and LM Studio surfaces that +// Re-checked on 2026-09-15, this time against LM Studio's own load-config schema rather than +// only its CLI and HTTP API, because that schema turned out to carry MTP settings the other +// two do not expose (see mtp-sidecar.ts). It carries nothing for vision either: the whole of +// LM Studio's projector handling is +// const visionPath = indexedModel.visionAdapter?.absPath; +// if (visionPath !== undefined) loadParams.mmproj_path = visionPath; +// read straight off the index entry, with no condition and no setting in front of it. +// Vision is therefore fixed when LM Studio indexes a model: a base GGUF is vision-capable +// exactly when an mmproj-*.gguf sits beside it in the same folder, and LM Studio surfaces that // pairing as its own model key whose capabilities.vision is true. The same GGUF bytes // indexed without a neighbouring mmproj report capabilities.vision false. A projector // therefore cannot be attached or detached at load time, and text-only can only ever // mean "send no image" — never "the vision weights were unloaded". +// What that last sentence leaves open is measuring the same weights with no projector at all, +// which needs a second entry for LM Studio to index rather than a different load. text-only.ts +// makes one out of hard links. export const projectorToggle=false; export const noProjectorToggleNote='LM Studio has no load-time option to attach or detach a vision projector. The projector belongs to the model entry itself (an mmproj file stored beside the GGUF), so text-only means no image is sent with any request; it does not unload vision weights.'; diff --git a/package-lock.json b/package-lock.json index aa2bde3..d4e45ac 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "local-model-bench", - "version": "1.6.1", + "version": "1.15.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "local-model-bench", - "version": "1.6.1", + "version": "1.15.0", "dependencies": { "lucide-react": "^0.468.0", "react": "^19.1.0", diff --git a/package.json b/package.json index fea0d55..c814d07 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "local-model-bench", - "version": "1.8.0", + "version": "1.15.0", "description": "Private desktop benchmarks and transparent grading for LM Studio models", "author": "Local Model Bench contributors", "license": "MIT", diff --git a/scripts/qa-accounting-mtp.mjs b/scripts/qa-accounting-mtp.mjs index c77ee0c..0da7ffe 100644 --- a/scripts/qa-accounting-mtp.mjs +++ b/scripts/qa-accounting-mtp.mjs @@ -7,7 +7,7 @@ const app=await electron.launch({executablePath:path.resolve('outputs/win-unpack const page=await app.firstWindow();page.setDefaultTimeout(20000); const errors=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); const snapshot=await page.evaluate(()=>window.bench.snapshot()); assert.equal(snapshot.tests.filter(t=>t.category==='Accounting').length,12); const models=await page.evaluate(()=>window.bench.models()); diff --git a/scripts/qa-accounting-upgrade.mjs b/scripts/qa-accounting-upgrade.mjs index 00324d7..80359a1 100644 --- a/scripts/qa-accounting-upgrade.mjs +++ b/scripts/qa-accounting-upgrade.mjs @@ -11,10 +11,10 @@ const custom={id:'accounting-cash-capital',name:'Preserve my custom accounting c db.prepare('INSERT INTO tests VALUES (?,?)').run(custom.id,JSON.stringify(custom));db.close(); const launch=()=>electron.launch({executablePath:path.resolve('outputs/win-unpacked/Local Model Bench.exe'),args:[],env:{...process.env,LMB_DATA_DIR:dir},timeout:60000}); let app=await launch();try{ - let page=await app.firstWindow();await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + let page=await app.firstWindow();await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); let s=await page.evaluate(()=>window.bench.snapshot());assert.equal(s.tests.filter(t=>t.category==='Accounting').length,12);assert.deepEqual(s.tests.find(t=>t.id===custom.id),custom); await page.evaluate(()=>window.bench.deleteTest('accounting-supplies')); - await app.close();app=await launch();page=await app.firstWindow();await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await app.close();app=await launch();page=await app.firstWindow();await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); s=await page.evaluate(()=>window.bench.snapshot());assert.equal(s.tests.filter(t=>t.category==='Accounting').length,11);assert.ok(!s.tests.some(t=>t.id==='accounting-supplies'));assert.deepEqual(s.tests.find(t=>t.id===custom.id),custom); assert.equal(await app.evaluate(({app})=>app.getVersion()),'1.5.0'); fs.writeFileSync('work/accounting-mtp-ui/upgrade-result.json',JSON.stringify({passed:true,packagedVersion:'1.5.0',customTestPreserved:true,deletedTestRemainsDeleted:true},null,2)); diff --git a/scripts/qa-charts.mjs b/scripts/qa-charts.mjs index ef53180..448167a 100644 --- a/scripts/qa-charts.mjs +++ b/scripts/qa-charts.mjs @@ -9,7 +9,7 @@ const source=JSON.parse(fs.readFileSync('outputs/Live-validation-results.json',' const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:dir},timeout:60000}); const page=await app.firstWindow();page.setDefaultTimeout(15000);const errors=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); // Inject saved data into an isolated QA database, then reload to exercise ordinary retrieval. const db=new DatabaseSync(path.join(dir,'bench.sqlite'));const {samples,waves,...doc}=source; db.prepare('INSERT OR REPLACE INTO runs VALUES (?,?)').run(source.id,JSON.stringify(doc)); diff --git a/scripts/qa-comparison.mjs b/scripts/qa-comparison.mjs index c8715e3..2618093 100644 --- a/scripts/qa-comparison.mjs +++ b/scripts/qa-comparison.mjs @@ -8,7 +8,7 @@ const source=JSON.parse(fs.readFileSync('outputs/Live-validation-results.json',' const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:dir},timeout:60000}); const page=await app.firstWindow();page.setDefaultTimeout(15000);const errors=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); const db=new DatabaseSync(path.join(dir,'bench.sqlite'));const {samples,waves,...doc}=source; db.prepare('INSERT OR REPLACE INTO runs VALUES (?,?)').run(source.id,JSON.stringify(doc)); for(const s of samples)db.prepare('INSERT OR REPLACE INTO samples VALUES (?,?,?)').run(s.id,s.runId,JSON.stringify(s)); diff --git a/scripts/qa-gpu.ts b/scripts/qa-gpu.ts index f607317..bb39277 100644 --- a/scripts/qa-gpu.ts +++ b/scripts/qa-gpu.ts @@ -32,7 +32,7 @@ store.saveRun(run);for(const s of samples)store.saveSample(s);store.close(); const app=await electron.launch({args:[root],env:{...process.env,LMB_DATA_DIR:dataDir},timeout:60000}); const page=await app.firstWindow();const errors:string[]=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); await page.locator('nav').getByRole('button',{name:'Results',exact:true}).click(); await page.getByLabel('Saved run').selectOption(runId); await page.getByRole('heading',{name:'GPU thermals'}).waitFor(); diff --git a/scripts/qa-library.mjs b/scripts/qa-library.mjs index 11c225a..da37e08 100644 --- a/scripts/qa-library.mjs +++ b/scripts/qa-library.mjs @@ -6,7 +6,7 @@ const root=process.cwd();fs.mkdirSync('work/library-qa',{recursive:true}); const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:path.join(root,'work/library-qa/data')},timeout:60000}); const page=await app.firstWindow();page.setDefaultTimeout(20000);const errors=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); let models;try{models=await page.evaluate(()=>window.bench.models());}catch{await page.evaluate(()=>window.bench.startServer());models=await page.evaluate(()=>window.bench.models());await page.getByRole('button',{name:'Refresh models',exact:true}).click();} assert.ok(models.length>0); await page.getByRole('button',{name:'Reset view',exact:true}).click(); diff --git a/scripts/qa-mtp-sweep.mjs b/scripts/qa-mtp-sweep.mjs new file mode 100644 index 0000000..7d44e02 --- /dev/null +++ b/scripts/qa-mtp-sweep.mjs @@ -0,0 +1,138 @@ +import {_electron as electron} from 'playwright'; +import {DatabaseSync} from 'node:sqlite'; +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; + +// Drives the real window to check the MTP sweep: the controls that plan one, and the panel, +// graphs and export that report one. No LM Studio is needed and nothing is inferred here. +// +// The results half reads the saved validation run and relabels its measurements with MTP +// depths - concurrency 1 becomes "MTP off" and concurrency 2 becomes "2 tokens" - so that the +// depth-aware rendering has something real to render. The NUMBERS are real measurements; the +// DEPTHS are labels this script invented. Whichever depth the panel calls fastest here is an +// artefact of that relabelling and says nothing whatsoever about multi-token prediction. That +// is why nothing from this script is written into the repository. +const root=process.cwd(),dir=path.join(root,'work/mtp-sweep-qa/data'); +fs.mkdirSync(dir,{recursive:true}); +const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:dir},timeout:60000}); +const page=await app.firstWindow();page.setDefaultTimeout(15000); +const errors=[];page.on('pageerror',e=>errors.push(e.message)); +const preview=()=>page.locator('.run-summary'); +try{ + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); + + // --- Planning a sweep ----------------------------------------------------------------- + await page.locator('nav').getByRole('button',{name:'Run',exact:true}).click(); + await page.getByRole('heading',{name:'Configure a benchmark',exact:true}).waitFor(); + const sweepToggle=page.getByLabel('Sweep MTP prediction depths'); + assert.equal(await sweepToggle.count(),0,'the sweep is offered only once MTP itself is on'); + await page.getByLabel('Enable native MTP').check(); + await sweepToggle.waitFor(); + assert.equal(await page.getByLabel('MTP draft tokens').count(),1,'a single depth is still the default'); + + await sweepToggle.check(); + const depths=page.getByLabel('MTP depths to sweep'); + assert.equal(await depths.inputValue(),'0, 1, 2, 3, 4, 5'); + assert.equal(await page.getByLabel('MTP draft tokens').count(),0,'one fixed depth is meaningless during a sweep'); + await page.getByText('6× the size of the same run without a sweep',{exact:false}).waitFor(); + await preview().getByText('MTP off, 1 token, 2 tokens, 3 tokens, 4 tokens, 5 tokens',{exact:true}).waitFor(); + + // The preflight is opt-in, and says so in the preview either way. It is a property of MTP + // rather than of sweeping, so it stays available once the sweep itself is turned off. + const preflight=page.getByLabel('Check the MTP head before measuring'); + assert.equal(await preflight.isChecked(),false,'a run does not pay for the check unless it asks'); + await preview().getByText('Off',{exact:true}).first().waitFor(); + await preflight.check(); + await preview().getByText('On · checked first',{exact:true}).waitFor(); + await preflight.uncheck(); + await preview().getByText('On · checked first',{exact:true}).waitFor({state:'detached'}); + await preflight.check(); + // What is typed is kept as typed, then ordered and de-duplicated when the field is left. + await depths.fill('4, 0, 0, 2');await depths.blur(); + assert.equal(await depths.inputValue(),'0, 2, 4'); + await preview().getByText('MTP off, 2 tokens, 4 tokens',{exact:true}).waitFor(); + // The warning is live feedback while typing. Leaving the field is what drops what it could not + // read, so the warning has served its purpose by then and the field is simply correct. + await depths.fill('0, 99, two'); + await page.getByText('Ignored: 99, two.',{exact:false}).waitFor(); + assert.equal(await depths.getAttribute('aria-invalid'),'true'); + await depths.blur(); + assert.equal(await depths.inputValue(),'0'); + assert.equal(await page.getByText('Ignored:',{exact:false}).count(),0); + + await depths.fill('');await depths.blur(); + await page.getByText('Enter at least one depth, or turn the sweep off.',{exact:true}).waitFor(); + assert.equal(await page.getByRole('button',{name:'Start benchmark'}).isDisabled(),true); + + // Turning MTP off has to take the sweep with it, or the run would be rejected on start. + await depths.fill('0, 2');await depths.blur(); + await page.getByLabel('Enable native MTP').uncheck(); + assert.equal(await page.getByLabel('Sweep MTP prediction depths').count(),0); + await page.getByLabel('Enable native MTP').check(); + assert.equal(await page.getByLabel('Sweep MTP prediction depths').isChecked(),false,'the sweep does not come back on by itself'); + + // --- Reporting a sweep ---------------------------------------------------------------- + const source=JSON.parse(fs.readFileSync('outputs/Live-validation-results.json','utf8')); + const {samples,waves,...doc}=source; + const DEPTHS=[0,2]; + const run={...doc,id:doc.id+'-sweep-fixture',config:{...doc.config,name:'Depth-labelled fixture (not an MTP measurement)',mtp:'on',mtpSweep:DEPTHS,concurrency:[1,2]}}; + const db=new DatabaseSync(path.join(dir,'bench.sqlite')); + db.prepare('INSERT OR REPLACE INTO runs VALUES (?,?)').run(run.id,JSON.stringify(run)); + // The real run measured two concurrency levels at one setting. Every measurement is copied + // once per depth so that depth and concurrency are crossed the way a real sweep crosses them + // — which is the whole point of the panel under test. The numbers in each copy are identical, + // so no depth can beat another here; only the layout is being checked. + for(const depth of DEPTHS){ + const tag=`-f${depth}`; + for(const s of samples)db.prepare('INSERT OR REPLACE INTO samples VALUES (?,?,?)').run(s.id+tag,run.id,JSON.stringify({...s,id:s.id+tag,runId:run.id,waveId:s.waveId+tag,...(s.warmup?{}:{mtpTokens:depth})})); + // Invented acceptance figures, present only at a depth that drafts at all, so the panel has + // both a filled cell and an empty one to render. + const draft=depth===0?undefined:{tasks:1,accepted:40,generated:64,acceptance:40/64,meanLen:2.5}; + for(const w of waves)db.prepare('INSERT OR REPLACE INTO waves VALUES (?,?,?)').run(w.id+tag,run.id,JSON.stringify({...w,id:w.id+tag,runId:run.id,mtpTokens:depth,...(draft?{draft}:{})})); + } + db.close(); + + await page.reload(); + await page.locator('nav').getByRole('button',{name:'Results',exact:true}).click(); + await page.getByRole('combobox',{name:'Saved run'}).selectOption(run.id); + const panel=page.locator('.panel').filter({has:page.getByRole('heading',{name:'Maximum predictions sweep',exact:true})}); + await panel.waitFor(); + // One block per model, a table per concurrency level inside it, and a row per depth in each: + // two models at two concurrency levels at two depths is eight rows. + assert.equal(await panel.locator('h3').count(),2); + assert.equal(await panel.locator('h4').count(),4,'each concurrency level is reported on its own'); + for(const level of ['1 concurrent request','2 concurrent requests'])await panel.getByRole('heading',{name:level,exact:true}).first().waitFor(); + assert.equal(await panel.locator('tbody tr').count(),8); + for(const label of ['MTP off','2 tokens'])assert.equal(await panel.locator('tbody tr').filter({hasText:label}).count(),4); + assert.equal(await panel.getByText('Not enough completed measurements',{exact:false}).count(),0,'both depths were found at every concurrency level'); + // Acceptance is reported where drafting happened and left blank where none did, rather than + // shown as a rate of zero. + assert.equal(await panel.locator('tbody tr').filter({hasText:'62.5%'}).count(),4,'the accepted-draft share is rendered per depth and concurrency'); + assert.equal(await panel.locator('tbody tr').filter({hasText:'MTP off'}).filter({hasText:'62.5%'}).count(),0,'MTP off drafts nothing, so it reports no acceptance'); + + // Each depth is its own line in the graphs rather than a second point on one model's curve. + const legend=await page.locator('.chart-legend').first().innerText(); + for(const key of source.config.modelKeys)for(const label of ['MTP off','MTP 2 tokens']) + assert.ok(legend.includes(`${key} · ${label}`),`legend is missing ${key} at ${label}`); + + const dest=path.join(root,'work/mtp-sweep-qa/report.md'); + await app.evaluate(({dialog},dest)=>{dialog.showSaveDialog=async()=>({canceled:false,filePath:dest});},dest); + await page.getByRole('button',{name:'MD',exact:true}).click(); + await page.getByText('Export saved: '+dest,{exact:true}).waitFor(); + const report=fs.readFileSync(dest,'utf8'); + assert.match(report,/## Native MTP sweep/); + assert.match(report,/swept depths: MTP off, 2 tokens/); + assert.match(report,/\/ MTP 2 tokens/); + assert.match(report,/— 2 concurrent requests/); + assert.match(report,/drafted tokens accepted 62\.5% \(40 of 64 drafted/); + assert.match(report,/drafted tokens accepted not applicable/); + + assert.deepEqual(errors,[]); + await panel.scrollIntoViewIfNeeded();await page.waitForTimeout(300); + await page.screenshot({path:'work/mtp-sweep-qa/sweep.png',fullPage:true}); + fs.writeFileSync('work/mtp-sweep-qa/result.json',JSON.stringify({passed:true,controls:'planned, normalized, and cleared',depthsRendered:['MTP off','2 tokens'], + caution:'Depth labels were invented by this script over real measurements; no MTP speed claim can be read from it.',errors,installed:!!process.env.LMB_QA_EXE},null,2)); + console.log('Sweep controls, depth-split panel, per-depth graph series, and the export section verified.'); + console.log('Reminder: the depths in the fixture are labels, not measurements. Nothing here says anything about MTP speed.'); +}finally{await app.close();} diff --git a/scripts/qa-results.mjs b/scripts/qa-results.mjs index 6b7a9f7..4a621ef 100644 --- a/scripts/qa-results.mjs +++ b/scripts/qa-results.mjs @@ -6,7 +6,7 @@ const root=process.cwd();fs.mkdirSync('work/qa-results',{recursive:true}); const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:path.join(root,process.env.LMB_QA_EXE?'work/qa-installed/data':'work/qa-results/data')},timeout:60000}); const page=await app.firstWindow();page.setDefaultTimeout(20000);const errors=[];page.on('pageerror',e=>errors.push(e.message)); try{ - await page.getByRole('heading',{name:'Your models. Real numbers.'}).waitFor(); + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); await page.getByText('LM Studio connected',{exact:true}).waitFor({timeout:20000}); const modelKey=process.env.LMB_MODEL_KEY;if(!modelKey)throw Error('Set LMB_MODEL_KEY to an exact downloaded model key.'); const model=page.locator('.model-card').filter({has:page.locator('.model-key',{hasText:modelKey})});await model.click(); diff --git a/scripts/qa-text-only.mjs b/scripts/qa-text-only.mjs new file mode 100644 index 0000000..b6165ff --- /dev/null +++ b/scripts/qa-text-only.mjs @@ -0,0 +1,82 @@ +import {_electron as electron} from 'playwright'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import assert from 'node:assert/strict'; + +// Drives the real window to make and remove a text-only copy of a real vision model, and +// checks what LM Studio does with it. This one touches the real LM Studio models directory, +// because that is the whole mechanism under test; it writes only hard links, removes them +// again, and asserts afterwards that the original weights are byte-for-byte where they were. +// +// LMB_TEXTONLY_MODEL picks the model; it must be one LM Studio reports as vision-capable. +const root=process.cwd(),dir=path.join(root,'work/text-only-qa/data'); +fs.rmSync(path.join(root,'work/text-only-qa'),{recursive:true,force:true}); +fs.mkdirSync(dir,{recursive:true}); +const WANT=process.env.LMB_TEXTONLY_MODEL||'gemma-4-12b-it@iq3_xxs'; +const app=await electron.launch({...(process.env.LMB_QA_EXE?{executablePath:process.env.LMB_QA_EXE}:{}),args:process.env.LMB_QA_EXE?[]:[root],env:{...process.env,LMB_DATA_DIR:dir},timeout:60000}); +const page=await app.firstWindow();page.setDefaultTimeout(30000); +const errors=[];page.on('pageerror',e=>errors.push(e.message)); +const models=()=>page.evaluate(()=>window.bench.models()); +const twins=()=>page.evaluate(()=>window.bench.textOnlyTwins()); +try{ + await page.getByRole('heading',{name:'Choose models to benchmark'}).waitFor(); + const before=await models(); + const target=before.find(m=>m.key===WANT); + assert.ok(target,`${WANT} is not in the library`); + assert.equal(target.capabilities?.vision,true,`${WANT} is not vision-capable, so there is nothing to leave out`); + assert.equal((await twins()).length,0,'the library starts with no copies this app made'); + + const plan=await page.evaluate(key=>window.bench.textOnlyPlan(key),WANT); + assert.ok(plan.folder.endsWith(' - Text only')); + assert.ok(plan.projectors.length>0,'the plan names the projector it is leaving behind'); + assert.ok(plan.files.every(f=>!/mmproj/i.test(f.to)),'and links none of it'); + const sizesBefore=plan.files.map(f=>fs.statSync(f.from).size); + + const panel=page.locator('.panel').filter({has:page.getByRole('heading',{name:'Vision projectors',exact:true})}); + await panel.waitFor(); + const row=panel.locator('.selection-list > div').filter({hasText:target.display_name}).filter({hasText:target.quantization?.name??''}).first(); + await row.getByRole('button',{name:'Make text-only copy'}).click(); + await page.getByText('Text-only copy made:',{exact:false}).waitFor(); + + // LM Studio indexes the new folder on its own; it is a separate model key reporting no vision. + const after=await models(); + const fresh=after.filter(m=>!before.some(b=>b.key===m.key)); + assert.equal(fresh.length,1,'exactly one new model key appeared: '+fresh.map(m=>m.key).join(', ')); + assert.equal(fresh[0].capabilities?.vision,false,'the copy is what the original could not be: not vision-capable'); + assert.equal(fresh[0].nativeMtp?.supported,target.nativeMtp?.supported,'and it keeps whatever native MTP the original had'); + // Hard links, not copies: same inode, and the folder adds no bytes to the volume. + const linked=plan.files.map(f=>fs.statSync(f.to)); + plan.files.forEach((f,i)=>{ + assert.equal(linked[i].ino,fs.statSync(f.from).ino,'the copy shares the original bytes'); + assert.equal(linked[i].size,sizesBefore[i]); + assert.ok(linked[i].nlink>=2); + }); + assert.equal(fs.existsSync(path.join(plan.folder,path.basename(plan.projectors[0]))),false,'no projector was linked in'); + assert.equal((await twins()).length,1); + + await panel.locator('.selection-list > div').filter({hasText:target.display_name}).filter({hasText:'Text-only copy made'}).first() + .getByRole('button',{name:'Remove copy'}).click(); + await page.getByText('Its weights were not touched.',{exact:false}).waitFor(); + assert.equal(fs.existsSync(plan.folder),false,'the folder is gone'); + assert.equal((await twins()).length,0); + // The point of the whole design: removal dropped links and nothing else. + plan.files.forEach((f,i)=>{ + assert.ok(fs.existsSync(f.from),'the original weights are still there'); + assert.equal(fs.statSync(f.from).size,sizesBefore[i],'and unchanged'); + }); + for(const p of plan.projectors)assert.ok(fs.existsSync(path.join(plan.sourceDir,p)),'and so is the projector'); + const restored=await models(); + assert.deepEqual(restored.map(m=>m.key).sort(),before.map(m=>m.key).sort(),'the library is exactly as it started'); + + assert.deepEqual(errors,[]); + await panel.scrollIntoViewIfNeeded();await page.waitForTimeout(300); + await page.screenshot({path:'work/text-only-qa/panel.png',fullPage:true}); + fs.writeFileSync('work/text-only-qa/result.json',JSON.stringify({passed:true,model:WANT,folder:plan.folder, + linkedFiles:plan.files.map(f=>path.basename(f.to)),projectorLeftBehind:plan.projectors,keptNativeMtp:fresh[0].nativeMtp?.supported===true, + bytesAdded:0,errors,platform:os.platform(),installed:!!process.env.LMB_QA_EXE},null,2)); + console.log(`Text-only copy of ${WANT}: created as hard links, indexed by LM Studio with vision=false, MTP kept, removed again with the original intact.`); +}catch(error){ + console.error(error); + process.exitCode=1; +}finally{await app.close();} diff --git a/src/BenchmarkPanels.tsx b/src/BenchmarkPanels.tsx index 155080e..5e25618 100644 --- a/src/BenchmarkPanels.tsx +++ b/src/BenchmarkPanels.tsx @@ -1,6 +1,7 @@ import React,{useEffect,useState} from 'react'; import {Plus,Trash2,Upload,X} from 'lucide-react'; import type {Run,RunSummary} from './types'; +import {mtpDepthLabel} from './mtp-sweep'; import {benchmarkRows,interpretScore,sameBenchmarkQuestions,scoreBands,summaryFor,type BenchmarkSelection,type PackSummary} from './benchmarks'; import {defaultInstruction,scoringModes,type PackDraft,type PackPreview,type ScoringMode} from './benchmark-import'; import {useModalDialog} from './dialog'; @@ -49,7 +50,7 @@ function PackImport({onClose,onDone}:{onClose:()=>void;onDone:(summary:PackSumma
; } -const settings=(r:Run)=>`MTP ${r.config.mtp??'default'} · reasoning ${r.config.reasoning} · temperature ${r.config.temperature} · context ${r.config.contextLength} · output ${r.config.maxTokens} · waves ${r.config.waves}`; +const settings=(r:Run)=>`MTP ${r.config.mtp??'default'}${r.config.mtpSweep?.length?' · swept '+r.config.mtpSweep.map(d=>mtpDepthLabel(d)).join(', '):r.config.mtp==='on'?' · '+(r.config.mtpDraftTokens??2)+' draft tokens':''} · reasoning ${r.config.reasoning} · temperature ${r.config.temperature} · context ${r.config.contextLength} · output ${r.config.maxTokens} · waves ${r.config.waves}`; export function BenchmarkResults({run,savedRuns,packs}:{run:Run;savedRuns:RunSummary[];packs:PackSummary[]}){ const [score,setScore]=useState(null),[compareId,setCompareId]=useState(''),[other,setOther]=useState(null),[error,setError]=useState(''); useEffect(()=>{setScore(null);setCompareId('');setOther(null);},[run.id]); @@ -57,6 +58,6 @@ export function BenchmarkResults({run,savedRuns,packs}:{run:Run;savedRuns:RunSum const id=run.tests.find(t=>t.benchmark)?.benchmark?.packId;if(!id)return null; const pack=summaryFor(id,packs); const matching=other&&sameBenchmarkQuestions(run,other); - const renderRows=(r:Run)=>
{benchmarkRows(r).map(row=>)}
; + const renderRows=(r:Run)=>
{benchmarkRows(r).map(row=>)}
; return

{pack.name} scores

{settings(run)}

Score = passed responses ÷ attempted responses. A response must pass every check. Request failures count as misses; warm-ups are excluded. Repeated waves and concurrent copies are repeated attempts, not new questions.

{renderRows(run)}{error&&

{error}

}{other&&<>

{other.config.name}

{settings(other)}

{matching?

Same questions and scoring protocol. Check the settings above before attributing a difference to the model.

:

Different questions or scoring protocol. These scores are not a controlled comparison.

}{matching&&renderRows(other)}}
; } diff --git a/src/ChartsPanel.tsx b/src/ChartsPanel.tsx index b3cee00..052aa52 100644 --- a/src/ChartsPanel.tsx +++ b/src/ChartsPanel.tsx @@ -1,22 +1,69 @@ import {useMemo,useState} from 'react'; import type {Run} from './types'; import {summaries} from '../electron/export'; -import {chartSpecs,renderChart,legendHtml,scoreLabels,type ChartRow,type ScoreSource} from './charts'; +import {chartSpecs,renderChart,legendHtml,scoreLabels,defaultXAxis,xAxisLabels,type ChartRow,type ScoreSource,type XAxis} from './charts'; +import {measuredDepths,mtpDepthText} from './mtp-sweep'; +import type {HistoryRow} from './history'; +import {backendOf} from '../electron/runtime'; +import {overlayOptions,overlayRows,stillOffered} from './chart-compare'; +// A sweep names its series after the model and the depth ("qwen … · MTP 2 tokens"). Clicking a +// point still has to inspect the model, so this is what the depth is stripped back off with. +const depthSuffix=' · MTP '; // The legend, graph grid, and hover tooltip are shared by the single-run graphs and the history overview, // which feeds in pooled rows keyed by group label instead of by model key. -export function ChartGrid({rows,score,onModel,hint='Click a point to inspect its model and responses.'}:{rows:ChartRow[];score:ScoreSource;onModel?:(key:string)=>void;hint?:string}){ +export function ChartGrid({rows,score,onModel,xAxis='concurrency',hint='Click a point to inspect its model and responses.'}:{rows:ChartRow[];score:ScoreSource;onModel?:(key:string)=>void;xAxis?:XAxis;hint?:string}){ const [tip,setTip]=useState(''); const show=(target:EventTarget)=>{const point=(target as Element).closest?.('[data-tooltip]');setTip(point?.getAttribute('data-tooltip')??'');}; return <>
-
show(e.target)} onMouseLeave={()=>setTip('')} onFocus={e=>show(e.target)} onBlur={()=>setTip('')} onClick={e=>{const key=(e.target as Element).closest('[data-model]')?.getAttribute('data-model');if(key)onModel?.(key);}} onKeyDown={e=>{if(e.key==='Enter'||e.key===' '){const key=(e.target as Element).getAttribute('data-model');if(key){e.preventDefault();onModel?.(key);}}}}>{chartSpecs(score).map(spec=>

{spec.title}

{spec.description}

)}
+
show(e.target)} onMouseLeave={()=>setTip('')} onFocus={e=>show(e.target)} onBlur={()=>setTip('')} onClick={e=>{const key=(e.target as Element).closest('[data-model]')?.getAttribute('data-model');if(key)onModel?.(key);}} onKeyDown={e=>{if(e.key==='Enter'||e.key===' '){const key=(e.target as Element).getAttribute('data-model');if(key){e.preventDefault();onModel?.(key);}}}}>{chartSpecs(score).map(spec=>

{spec.title}

{spec.description}

)}
{tip&&
{tip}{hint}
} ; } -export function ChartsPanel({run,onModel}:{run:Run;onModel?:(key:string)=>void}){ +export function ChartsPanel({run,onModel,history=[]}:{run:Run;onModel?:(key:string)=>void;history?:HistoryRow[]}){ const [testId,setTestId]=useState(''),[score,setScore]=useState('objective'); + // Which saved series are drawn beside this run. Ids rather than rows, so a selection survives the + // history being reloaded underneath it. + const [compare,setCompare]=useState([]); const chosen=run.tests.some(t=>t.id===testId)?testId:run.tests[0]?.id; - const rows=useMemo(()=>summaries(run).filter(r=>r.testId===chosen),[run,chosen]); - return

Automatic graphs

Updated as requests and grades arrive. Hover over a point for its values.

- + // Depths measured by this run. With more than one, the graphs can read across depth instead of + // across concurrency — which is the only way to see a sweep run at a single concurrency level. + const depths=useMemo(()=>measuredDepths(run),[run]); + // Whether concurrency is a dimension of this run at all. A sweep run at a single concurrency level + // has nothing to read across on that axis, so defaulting to it shows the run's whole subject as a + // stack of points on one vertical line — the graphs open on whichever axis the run actually varied. + const concurrencyVaries=useMemo(()=>new Set(summaries(run).map(r=>r.concurrency)).size>1,[run]); + // null until chosen, so the default can follow the run rather than being frozen at first render. + const [xAxis,setXAxis]=useState(null); + // A run with one depth has no depth axis to offer, and one that loses its sweep while selected + // must not keep drawing an axis that no longer varies. + const axis:XAxis=depths.length<=1?'concurrency':xAxis??defaultXAxis(depths.length,concurrencyVaries); + const own=useMemo(()=>{ + const all=summaries(run).filter(r=>r.testId===chosen); + // Depth goes into the series name only when it is not the axis. Doing both would split each + // model into one flat single-point line per depth rather than a curve across them. + return depths.length>1&&axis!=='mtpDepth'?all.map(r=>({...r,modelKey:`${r.modelKey} · ${mtpDepthText(r.mtpDepth)}`})):all; + },[run,chosen,depths,axis]); + const backend=useMemo(()=>backendOf(run).label,[run]); + const models=useMemo(()=>[...new Set(summaries(run).map(r=>r.modelKey))],[run]); + const options=useMemo(()=>overlayOptions(history,run.id,chosen??'',backend,models),[history,run.id,chosen,backend,models]); + // Changing the test can leave a selection pointing at data that test never produced, which would + // draw nothing and read as a bug rather than as a choice. + const live=useMemo(()=>stillOffered(compare,options),[compare,options]); + const rows=useMemo(()=>[...own,...overlayRows(history,live,run.id,chosen??'',models)],[own,history,live,run.id,chosen,models]); + const backends=options.filter(o=>o.kind==='backend'),others=options.filter(o=>o.kind==='model'); + const toggle=(id:string)=>setCompare(c=>c.includes(id)?c.filter(x=>x!==id):[...c,id]); + return

Automatic graphs

Updated as requests and grades arrive. Hover over a point for its values.

+ {depths.length>1&&} + {/* Saved measurements of the same test, drawn on the same axes. The control is absent entirely + when nothing comparable is saved, rather than an empty menu that invites a click. */} + {options.length>0&&
Compare with{live.length?` · ${live.length}`:''} +
+ {backends.length>0&&<>Other engines{backends.map(o=>)}} + {others.length>0&&<>Other models{others.map(o=>)}} + {live.length>0&&} +

Saved rows from finished runs, drawn as their own lines. Nothing is recalculated, and conditions can differ between runs.

+
} +
+ onModel?.(key.split(depthSuffix)[0])}/>
; } diff --git a/src/HistoryPanel.tsx b/src/HistoryPanel.tsx index 022aa93..e840699 100644 --- a/src/HistoryPanel.tsx +++ b/src/HistoryPanel.tsx @@ -22,7 +22,7 @@ const columns:{key:SortKey;label:string}[]=[ {key:'generationTps',label:'Gen tok/s'},{key:'estimatedPrefillTps',label:'Prefill est.'},{key:'throughput',label:'Total tok/s'}, {key:'medianMs',label:'Median / p95'},{key:'failureRate',label:'Failures'}, {key:'objective',label:'Objective'},{key:'localJudge',label:'Local judge'},{key:'gpuHotSpotMax',label:'GPU hot spot'}]; -const conditionText=(g:HistoryGroup)=>[`concurrency ${g.conditions.concurrency.join(', ')||'—'}`,g.conditions.promptSizes.join(', '),`MTP ${g.conditions.mtp.join(' / ')||'—'}`,`reasoning ${g.conditions.reasoning.join(' / ')||'—'}`].filter(Boolean).join(' · '); +const conditionText=(g:HistoryGroup)=>[`concurrency ${g.conditions.concurrency.join(', ')||'—'}`,g.conditions.promptSizes.join(', '),`MTP ${g.conditions.mtpDepth.join(' / ')||'—'}`,`reasoning ${g.conditions.reasoning.join(' / ')||'—'}`].filter(Boolean).join(' · '); export function HistoryPanel({view,onChange,onOpenRun,signature}:{view:HistoryView;onChange:(v:HistoryView)=>void;onOpenRun:(id:string)=>void;signature:string}){ const [rows,setRows]=useState(null),[error,setError]=useState(''),[loading,setLoading]=useState(false); const [score,setScore]=useState('objective'),[open,setOpen]=useState(''); diff --git a/src/LibraryControls.tsx b/src/LibraryControls.tsx index 2289a0b..a056d2b 100644 --- a/src/LibraryControls.tsx +++ b/src/LibraryControls.tsx @@ -15,7 +15,7 @@ export function LibraryControls({models,view,onChange}:{models:Model[];view:Libr
- + diff --git a/src/benchmarks.ts b/src/benchmarks.ts index b9a427e..b7643f4 100644 --- a/src/benchmarks.ts +++ b/src/benchmarks.ts @@ -1,5 +1,6 @@ import data from './benchmark-data/packs.json'; import type {Run, RunConfig, TestCase} from './types'; +import {mtpDepthText,sweepSteps} from './mtp-sweep'; export type BenchmarkSelection={packId:string;count:number;seed:number}; export type BenchmarkMeta={packId:string;itemId:string;datasetHash:string;protocol:string}; @@ -70,14 +71,18 @@ export function interpretScore(score:number|null){return score===null||!Number.i export function benchmarkRows(run:Run){ const tests=run.tests.filter(t=>t.benchmark),ids=new Set(tests.map(t=>t.id)); if(!tests.length)return []; - return run.config.modelKeys.flatMap(key=>run.config.concurrency.map(concurrency=>{ - const samples=run.samples.filter(s=>!s.warmup&&s.modelKey===key&&s.concurrency===concurrency&&ids.has(s.testId)); + // An MTP sweep runs the whole pack once per prediction depth, so a depth is a separate score + // with its own expected question count. Runs that did not sweep have a single depth of null, + // which matches every response and leaves their rows exactly as they were. + const depths=sweepSteps(run.config).map(s=>s.depth); + return run.config.modelKeys.flatMap(key=>depths.flatMap(depth=>run.config.concurrency.map(concurrency=>{ + const samples=run.samples.filter(s=>!s.warmup&&s.modelKey===key&&s.concurrency===concurrency&&ids.has(s.testId)&&(depth===null||s.mtpTokens===depth)); const passed=samples.filter(s=>s.status==='completed'&&s.objective.score===100).length; const failed=samples.filter(s=>s.status!=='completed').length; const expected=tests.length*run.config.waves*concurrency; const score=samples.length?passed/samples.length*100:null; - return {key,concurrency,passed,attempted:samples.length,expected,failed,score,provisional:samples.length!==expected||run.status!=='completed',truncated:samples.filter(s=>s.possibleTruncation).length,uniqueQuestions:new Set(samples.map(s=>s.testId)).size}; - })); + return {key,concurrency,depth,mtp:mtpDepthText(depth),passed,attempted:samples.length,expected,failed,score,provisional:samples.length!==expected||run.status!=='completed',truncated:samples.filter(s=>s.possibleTruncation).length,uniqueQuestions:new Set(samples.map(s=>s.testId)).size}; + }))); } export function sameBenchmarkQuestions(a:Run,b:Run){ const signature=(r:Run)=>JSON.stringify(r.tests.filter(t=>t.benchmark).map(t=>[t.id,t.benchmark?.datasetHash,t.benchmark?.protocol,t.prompt,t.rules]).sort((x,y)=>String(x[0]).localeCompare(String(y[0])))); diff --git a/src/cache-quant.ts b/src/cache-quant.ts new file mode 100644 index 0000000..fb69bae --- /dev/null +++ b/src/cache-quant.ts @@ -0,0 +1,44 @@ +// KV cache quantization, shared by the window and the engine. +// +// The context is the part of a run's memory that scales with concurrency: LM Studio loads one +// instance holding contextLength × parallel tokens of keys and values, so raising the highest +// concurrency raises the cache, not the weights. On a card where the weights already nearly fill +// VRAM that is what decides whether the run measures the GPU or measures an offload to system RAM. + +// The types llama.cpp accepts for a KV cache, in the order LM Studio lists them. f16 is what an +// unquantized cache already is, so it is offered as the explicit "full precision" choice and +// `off` means "do not write the setting at all". +export type CacheQuant='off'|'f16'|'q8_0'|'q5_0'|'q4_0'|'iq4_nl'; +export const cacheQuants:CacheQuant[]=['off','f16','q8_0','q5_0','q4_0','iq4_nl']; + +// Bits per element actually stored, used for the size estimate the run preview shows. q8_0 and +// q4_0 carry a scale per block of 32, which is why they are not exactly 8 and 4. +const BITS:Record,number>={f16:16,q8_0:8.5,q5_0:5.5,q4_0:4.5,iq4_nl:4.5}; + +export const cacheQuantLabel=(q:CacheQuant):string=> + q==='off'?'Off · LM Studio default':q==='f16'?'f16 · full precision': + q==='q8_0'?'q8_0 · half size':q==='q5_0'?'q5_0':q==='q4_0'?'q4_0 · quarter size':'iq4_nl · quarter size'; + +// Relative to an f16 cache, which is the baseline every existing run in the history was measured +// with. `off` is reported as 1 because LM Studio's own default for these models is f16. +export const cacheScale=(q:CacheQuant):number=>q==='off'?1:BITS[q]/16; + +// A run sets K and V independently because they do not degrade alike: a q4_0 V cache is routinely +// fine where a q4_0 K cache measurably hurts, so the common recommendation is q8_0 K with a +// smaller V. The combined figure is the mean of the two halves. +export const cacheScalePair=(k:CacheQuant,v:CacheQuant):number=>(cacheScale(k)+cacheScale(v))/2; + +// Whether flash attention is required. llama.cpp will not use a quantized KV cache without it, and +// LM Studio refuses the load outright rather than falling back. Its own default is not dependable — +// it varies by engine and by whatever the model was last loaded with — which is why a run states the +// setting instead of inheriting it, and why asking for a quantized cache with flash off is refused +// rather than sent to LM Studio to fail. +export const needsFlashAttention=(k:CacheQuant,v:CacheQuant):boolean=> + (k!=='off'&&k!=='f16')||(v!=='off'&&v!=='f16'); + +export const cacheQuantText=(k:CacheQuant,v:CacheQuant):string=> + k==='off'&&v==='off'?'Off':k===v?`${k} K and V`:`${k} K · ${v} V`; + +// On unless a run says otherwise. Kept here rather than read inline so the engine, the validator +// and the run preview cannot drift about what the default is. +export const flashOn=(v:'on'|'off'|undefined):boolean=>v!=='off'; diff --git a/src/chart-compare.ts b/src/chart-compare.ts new file mode 100644 index 0000000..b80a0e4 --- /dev/null +++ b/src/chart-compare.ts @@ -0,0 +1,111 @@ +import type {ChartRow} from './charts'; +import type {HistoryRow} from './history'; +import {shortDate,unknownFacet} from './history'; +import {unknownBackend} from '../electron/runtime'; + +// Putting saved measurements beside the run you are looking at. +// +// The graphs draw one line per modelKey, which is how the MTP sweep already shows several depths of +// one model at once: it relabels the key and the series separate themselves. The same trick carries +// a different engine, or a model measured in some other run, onto the same axes — so "is ROCm +// faster at prompt processing" becomes a line to look at rather than two screens to remember. +// +// Nothing here recomputes a measurement. Every overlaid point is a saved row from a finished run, +// relabelled so the legend says where it came from. + +export type OverlayKind='backend'|'model'; +export type OverlayOption={id:string;kind:OverlayKind;label:string;detail:string;points:number}; + +export const overlayId=(kind:OverlayKind,value:string)=>`${kind}:${value}`; +const parseId=(id:string):{kind:OverlayKind;value:string}|null=>{ + const at=id.indexOf(':'); + if(at<0)return null; + const kind=id.slice(0,at); + return kind==='backend'||kind==='model'?{kind,value:id.slice(at+1)}:null; +}; + +// One saved row per (model, test, concurrency, depth). Several runs can hold the same combination, +// and plotting all of them would draw one series through points taken weeks apart; the most recent +// run wins, which is also the one whose conditions are most likely to still apply. +function newestPerPoint(rows:HistoryRow[]):HistoryRow[]{ + const best=new Map(); + for(const r of rows){ + const key=JSON.stringify([r.modelKey,r.testId,r.concurrency,r.mtpDepth]); + const prior=best.get(key); + if(!prior||Date.parse(r.runCreated)>Date.parse(prior.runCreated))best.set(key,r); + } + return [...best.values()]; +} + +// What this run could usefully be compared against: the same test, measured somewhere else. A +// backend only appears when it has measurements of a model this run also measured, because an +// engine comparison against a different model would not be an engine comparison. +export function overlayOptions(history:HistoryRow[],runId:string,testId:string,currentBackend:string,currentModels:string[]):OverlayOption[]{ + const elsewhere=history.filter(r=>r.runId!==runId&&r.testId===testId); + const models=new Set(currentModels); + const out:OverlayOption[]=[]; + const backends=new Map(); + for(const r of elsewhere){ + if(r.backend===currentBackend||r.backend===unknownBackend||!models.has(r.modelKey))continue; + backends.set(r.backend,[...(backends.get(r.backend)??[]),r]); + } + for(const [label,rows] of [...backends].sort((a,b)=>a[0].localeCompare(b[0]))){ + const points=newestPerPoint(rows); + out.push({id:overlayId('backend',label),kind:'backend',label:`Compare with ${label}`, + detail:`${[...new Set(points.map(p=>p.modelKey))].length} of this run's model(s), measured on ${label}`,points:points.length}); + } + const others=new Map(); + for(const r of elsewhere){ + if(models.has(r.modelKey))continue; + others.set(r.modelKey,[...(others.get(r.modelKey)??[]),r]); + } + for(const [key,rows] of [...others].sort((a,b)=>a[0].localeCompare(b[0]))){ + const points=newestPerPoint(rows); + const last=points.map(p=>p.runCreated).sort().at(-1); + out.push({id:overlayId('model',key),kind:'model',label:key, + detail:`${points[0]?.backend&&points[0].backend!==unknownBackend?points[0].backend+' · ':''}last measured ${shortDate(last)}`,points:points.length}); + } + return out; +} + +// The chosen overlays as chart rows, with the key relabelled so the legend says where each line came +// from. A backend overlay keeps the model name and adds the engine; a model overlay names the model +// and the engine it was measured on, because the same model on two engines is two different lines. +export function overlayRows(history:HistoryRow[],ids:string[],runId:string,testId:string,currentModels:string[]):ChartRow[]{ + const models=new Set(currentModels); + const out:ChartRow[]=[]; + for(const id of ids){ + const parsed=parseId(id); + if(!parsed)continue; + const matching=history.filter(r=>r.runId!==runId&&r.testId===testId&&( + parsed.kind==='backend'?r.backend===parsed.value&&models.has(r.modelKey):r.modelKey===parsed.value)); + for(const r of newestPerPoint(matching)) + out.push({...r,modelKey:`${r.modelKey} · ${r.backend===unknownBackend?shortDate(r.runCreated):r.backend}`}); + } + return out; +} + +// The engines this run's measurements could be set beside: ones that measured a model this run also +// measured, in some other run. A run whose engine could not be determined is not offered, because +// "Unknown engine" is not a thing to compare against. +export function comparableBackends(history:HistoryRow[],runId:string,currentBackend:string,models:string[]):string[]{ + const set=new Set(models); + const found=new Set(); + for(const r of history) + if(r.runId!==runId&&r.backend!==currentBackend&&r.backend!==unknownBackend&&set.has(r.modelKey))found.add(r.backend); + return [...found].sort(); +} + +// Saved rows for the chosen engines, for models this run measured, newest run per measurement. Used +// by the comparison table, where the engine becomes a column rather than part of the series name. +export function backendRows(history:HistoryRow[],runId:string,backends:string[],models:string[]):HistoryRow[]{ + const want=new Set(backends),set=new Set(models); + return newestPerPoint(history.filter(r=>r.runId!==runId&&want.has(r.backend)&&set.has(r.modelKey))); +} + +// Whether an overlay is still on offer after the test changed. A selection kept for a test that has +// no such saved data would silently draw nothing, so the caller drops it instead. +export const stillOffered=(ids:string[],options:OverlayOption[]):string[]=>{ + const live=new Set(options.map(o=>o.id)); + return ids.filter(id=>live.has(id)); +}; diff --git a/src/charts.ts b/src/charts.ts index a70b4ce..ce3ddba 100644 --- a/src/charts.ts +++ b/src/charts.ts @@ -8,6 +8,18 @@ const fmt=(n:number)=>n.toLocaleString('en-US',{maximumFractionDigits:n<1?2:1}); export function colorFor(key:string){let hash=0;for(const c of key)hash=(Math.imul(hash,31)+c.charCodeAt(0))|0;return `hsl(${Math.abs(hash)%360}, 65%, 72%)`;} type Metric='generationTps'|'estimatedPrefillTps'|'throughput'|'ttftMs'|'medianMs'|'p95Ms'|'failureRate'|ScoreSource; export type ChartSpec={key:Metric|'scatter';title:string;unit:string;description:string;scale?:number;fixedMax?:number}; +// What the horizontal axis counts. Concurrency is the historical answer and stays the default, but +// an MTP sweep at a single concurrency level has nothing to spread along it: every depth lands on +// the same x and the series stack on one vertical line. Depth is the axis that run varies. +export type XAxis='concurrency'|'mtpDepth'; +export const xAxisLabels:Record={concurrency:'Concurrent requests',mtpDepth:'Maximum predictions (0 = MTP off)'}; +// A row from a run that measured no depth sits at 0, which is where "MTP off" belongs anyway. +export const xValueOf=(r:ChartRow,axis:XAxis):number=>axis==='mtpDepth'?(r.mtpDepth??0):r.concurrency; +// Which axis a run should open on, before anyone chooses. A run that swept depths at a single +// concurrency level varied exactly one thing, and it is not concurrency: opening on that axis puts +// every depth at the same x and hides the run's whole subject behind a stack of points. +export const defaultXAxis=(depthsMeasured:number,concurrencyVaries:boolean):XAxis=> + depthsMeasured>1&&!concurrencyVaries?'mtpDepth':'concurrency'; export function chartSpecs(score:ScoreSource):ChartSpec[]{return [ {key:'generationTps',title:'Generation speed',unit:'Tokens / second',description:'Average speed per successful request. Higher is faster.'}, {key:'estimatedPrefillTps',title:'Estimated prefill speed',unit:'Tokens / second',description:'Client-timed prompt processing; caching and buffering can affect this estimate.'}, @@ -21,13 +33,13 @@ export function chartSpecs(score:ScoreSource):ChartSpec[]{return [ ];} const start=(label:string)=>``; const empty=(label:string)=>`${start(label)}No measurements available yetMissing values are not plotted as zero.`; -export function renderChart(rows:ChartRow[],spec:ChartSpec,score:ScoreSource='objective'):string{ +export function renderChart(rows:ChartRow[],spec:ChartSpec,score:ScoreSource='objective',xAxis:XAxis='concurrency'):string{ const scatter=spec.key==='scatter'; const val=(r:ChartRow):number|null=>{const n=scatter?r[score]:r[spec.key as Metric];return typeof n==='number'&&Number.isFinite(n)?n*(spec.scale??1):null;}; const usable=rows.filter(r=>val(r)!==null&&(!scatter||r.generationTps!==null)); if(!usable.length)return empty(spec.title); const yMax=scatter?100:spec.fixedMax??Math.max(1,...usable.map(r=>val(r)!*1.08)); - const allXs=[...new Set(rows.map(r=>r.concurrency))].sort((a,b)=>a-b); + const allXs=[...new Set(rows.map(r=>xValueOf(r,xAxis)))].sort((a,b)=>a-b); const xMin=scatter?0:Math.min(...allXs),xMax=scatter?Math.max(1,...usable.map(r=>r.generationTps!))*1.08:Math.max(...allXs); const x=(v:number)=>xMax===xMin?320:72+(v-xMin)/(xMax-xMin)*496; const y=(v:number)=>230-v/yMax*184; @@ -36,15 +48,15 @@ export function renderChart(rows:ChartRow[],spec:ChartSpec,score:ScoreSource='ob // Limit labels to a readable subset without changing numeric axis spacing. const ticks=scatter?[0,.25,.5,.75,1].map(t=>t*xMax):allXs.filter((_,i)=>allXs.length<=8||i===allXs.length-1||i%Math.ceil(allXs.length/7)===0); for(const t of ticks)svg+=`${fmt(t)}`; - svg+=`${scatter?'Generation tokens / second':'Concurrent requests'}`; + svg+=`${scatter?'Generation tokens / second':xAxisLabels[xAxis]}`; const keys=[...new Set(rows.map(r=>r.modelKey))].sort(); keys.forEach((key,i)=>{ - const rs=rows.filter(r=>r.modelKey===key).sort((a,b)=>a.concurrency-b.concurrency),color=colorFor(key); + const rs=rows.filter(r=>r.modelKey===key).sort((a,b)=>xValueOf(a,xAxis)-xValueOf(b,xAxis)),color=colorFor(key); let segment:string[]=[]; const flush=()=>{if(segment.length>1)svg+=``;segment=[];}; - if(!scatter){for(const r of rs){const v=val(r);if(v===null){flush();continue;}segment.push(`${x(r.concurrency)},${y(v)}`);}flush();} - for(const r of rs){const v=val(r);if(v===null||(scatter&&r.generationTps===null))continue;const cx=x(scatter?r.generationTps!:r.concurrency),cy=y(v);const title=`${r.modelKey} • concurrency ${r.concurrency} • ${scatter?fmt(r.generationTps!)+' tok/s • ':''}${fmt(v)} ${scatter?'score':spec.unit} • ${r.requests} requests, ${r.failures} failures`; - svg+=`${escapeHtml(title)}`; + if(!scatter){for(const r of rs){const v=val(r);if(v===null){flush();continue;}segment.push(`${x(xValueOf(r,xAxis))},${y(v)}`);}flush();} + for(const r of rs){const v=val(r);if(v===null||(scatter&&r.generationTps===null))continue;const cx=x(scatter?r.generationTps!:xValueOf(r,xAxis)),cy=y(v);const title=`${r.modelKey} • ${xAxis==='mtpDepth'?(r.mtpDepth?`${r.mtpDepth} draft token${r.mtpDepth===1?'':'s'}`:'MTP off')+` • concurrency ${r.concurrency}`:`concurrency ${r.concurrency}`} • ${scatter?fmt(r.generationTps!)+' tok/s • ':''}${fmt(v)} ${scatter?'score':spec.unit} • ${r.requests} requests, ${r.failures} failures`; + svg+=`${escapeHtml(title)}`; } });return svg+''; } diff --git a/src/defaults.ts b/src/defaults.ts index 3cd5059..f9b3301 100644 --- a/src/defaults.ts +++ b/src/defaults.ts @@ -14,7 +14,12 @@ const rubric='Score correctness, completeness, clarity, and instruction followin const rule=(id:string,label:string,type:Rule['type'],extra:Partial={}):Rule=>({id,label,type,weight:1,...extra}); const test=(id:string,name:string,category:string,prompt:string,rules:Rule[]=[],answerKey='',maxTokens=1024):TestCase=>({id,name,category,prompt,rules,answerKey,maxTokens,version:1,rubric,kind:'quality'}); import { accountingTests } from './accounting'; -export const starterTests:TestCase[]=[ +// These three ask for "only JSON" and never mention code fences, so a fenced answer is the model +// following the instruction in the usual markdown way rather than disobeying it. The accounting +// pack is deliberately not here: its prompt says "no prose or code fences", which makes the fence +// part of what that pack tests. +const fenceTolerant=new Set(['subnet','facts','json-extract']); +export const starterTests:TestCase[]=([ test('report-general','Explain renewable energy','Report','Write a 350–500 word report comparing solar and wind power from your own knowledge. Use the headings Overview, Tradeoffs, and Recommendation. Explain intermittency, storage, and one appropriate use case for each. State uncertainties; do not invent citations.',[rule('length','350–500 words','words',{min:350,max:500}),...['Overview','Tradeoffs','Recommendation'].map(x=>rule(x,x+' heading','heading',{expected:x}))]), test('report-network','Design a small office network','Networking','Write a 350–500 word report recommending a network for a 30-person office with staff devices, guest Wi-Fi, and an internal file server. Use the headings Design, Security, and Operations. Explain VLANs, DHCP, DNS, firewall rules, backups, and monitoring. Work from your own knowledge; do not invent references.',[rule('length','350–500 words','words',{min:350,max:500}),...['Design','Security','Operations'].map(x=>rule(x,x+' heading','heading',{expected:x}))]), test('report-security','Respond to a phishing incident','Security','Write a concise incident-response report for an employee who entered their password on a phishing page. Use the headings Immediate actions, Investigation, and Prevention. Distinguish password reset, session revocation, MFA, and evidence preservation. Work from your own knowledge.', ['Immediate actions','Investigation','Prevention'].map(x=>rule(x,x+' heading','heading',{expected:x}))), @@ -26,7 +31,7 @@ export const starterTests:TestCase[]=[ test('instructions','Follow exact constraints','Instructions','Return exactly two lines and no other text. First line: TCP is connection-oriented. Second line: UDP is connectionless.',[rule('exact','Exact required response','exact',{expected:'TCP is connection-oriented.\nUDP is connectionless.'})],'TCP is connection-oriented.\nUDP is connectionless.',128), test('debug','Find a boundary bug','Coding','Explain the bug in this JavaScript function and give a corrected version. Include headings Bug, Fix, and Example. Do not execute code.\nfunction total(xs) { let sum = 0; for (let i = 0; i <= xs.length; i++) sum += xs[i]; return sum; }',[...['Bug','Fix','Example'].map(x=>rule(x,x+' heading','heading',{expected:x}))],'The <= condition accesses xs[xs.length], which is undefined, making sum NaN. Use i < xs.length. Empty input should return 0.',768), ...accountingTests -]; +] as TestCase[]).map(t=>fenceTolerant.has(t.id)?{...t,allowCodeFence:true}:t); export function performanceTest(length:string):TestCase { const count=length==='long'?180:length==='medium'?55:4; const content=Array.from({length:count},(_,i)=>`Record ${i+1}: A regional office operates a router, two switches, staff workstations, and a file service. Backups complete nightly. Monitoring tracks latency, availability, capacity, and authentication errors.`).join('\n'); diff --git a/src/endpoint.ts b/src/endpoint.ts new file mode 100644 index 0000000..0867cac --- /dev/null +++ b/src/endpoint.ts @@ -0,0 +1,28 @@ +// Where this app sends prompts. +// +// Loopback is the default and the private case, but the endpoint is just an OpenAI-shaped HTTP API, +// so it can be LM Studio on another machine, a server on the LAN, or anything else speaking the same +// protocol. The host is therefore not restricted. What stays rejected is everything that would make +// the address mean something other than an origin: credentials embedded in the URL, a path, a query, +// a fragment. +// +// This lives in src/ rather than beside the LM Studio client because the window needs it too — it +// has to say plainly when prompts are leaving the machine — and that module reaches for node:fs and +// node:child_process, which cannot be bundled for the renderer. +export const loopbackHosts = ['127.0.0.1', 'localhost', '[::1]', '::1']; + +// Reported, never enforced: the window uses it to replace a privacy claim it can no longer make. +export function isLoopbackUrl(url: string): boolean { + try { return loopbackHosts.includes(new URL(url).hostname); } catch { return false; } +} + +export function validateUrl(url: string): string { + let u: URL; + try { u = new URL(url); } catch { throw Error('Enter a full address, such as http://127.0.0.1:1234 or http://192.168.1.50:1234.'); } + if (!['http:', 'https:'].includes(u.protocol)) throw Error('The endpoint must be an http:// or https:// address.'); + if (!u.hostname) throw Error('The endpoint needs a host, such as 127.0.0.1 or 192.168.1.50.'); + // A token belongs in the token field, where it is encrypted at rest and never crosses to the window. + if (u.username || u.password) throw Error('Put credentials in the API token field rather than in the address.'); + if (u.pathname !== '/' || u.search || u.hash) throw Error('Give only the server address, with no path, query or fragment — for example http://192.168.1.50:1234.'); + return u.origin; +} diff --git a/src/history.ts b/src/history.ts index 8b38b00..8fa5dfe 100644 --- a/src/history.ts +++ b/src/history.ts @@ -1,6 +1,7 @@ import type {ChartRow} from './charts'; import type {ModelProfile} from './model-profile'; import {percentile} from '../electron/metrics'; +import {mtpDepthText,rowDepth} from './mtp-sweep'; // One saved measurement group (model x test x concurrency) from one run, carrying the facts needed to pool // it with the same measurement taken in a different run. It extends the summary row the single-run screens // already use, so no measurement is recomputed here. @@ -11,6 +12,12 @@ export type HistoryRow=ChartRow&{ family:string;kind:ModelProfile['kind'];sizeLabel:string;sizeBucket:string;quantization:string;quantTier:string; publisher:string;architecture:string;totalB:number|null;activeB:number|null; promptSize:string|null;testKind:string;benchmarkPack:string|null; + // Which llama.cpp build produced the numbers. See electron/runtime.ts backendOf for where it + // comes from and why a run that cannot say reports Unknown rather than today's selection. + backend:string;backendRef:string|null; + // The calibrated prompt-processing rate for the model this row measured, copied from the run's + // own record so a row pooled into another screen can still report it. + prefillCalibratedTps:number|null;prefillOverheadMs:number|null; reasoning:string;temperature:number;contextLength:number;maxTokens:number;waves:number; completed:number;durationsMs:number[];objectiveCount:number;localJudgeCount:number;externalJudgeCount:number; }; @@ -54,19 +61,23 @@ export const facetValue={ promptSizes:(r:HistoryRow)=>r.promptSize??otherPrompt, benchmarks:(r:HistoryRow)=>r.benchmarkPack??ownTests, mtp:(r:HistoryRow)=>r.mtp, + // The depth a measurement was taken at, so a sweep step pools with an ordinary run that used + // the same depth and never with one that used a different depth. + mtpDepth:(r:HistoryRow)=>mtpDepthText(rowDepth(r)), vision:(r:HistoryRow)=>r.vision, reasoning:(r:HistoryRow)=>r.reasoning, statuses:(r:HistoryRow)=>r.runStatus, + backends:(r:HistoryRow)=>r.backend, } as const; export type FacetKey=keyof typeof facetValue; export const facetKeys=Object.keys(facetValue) as FacetKey[]; -export const facetLabels:Record={families:'Model family',kinds:'Architecture',sizes:'Parameters',quantTiers:'Quantization tier',quants:'Exact quantization',concurrency:'Concurrent requests',promptSizes:'Prompt size',benchmarks:'Benchmark pack',mtp:'Native MTP',vision:'Vision',reasoning:'Reasoning',statuses:'Run status'}; +export const facetLabels:Record={families:'Model family',kinds:'Architecture',sizes:'Parameters',quantTiers:'Quantization tier',quants:'Exact quantization',concurrency:'Concurrent requests',promptSizes:'Prompt size',benchmarks:'Benchmark pack',mtp:'Native MTP',mtpDepth:'MTP depth',vision:'Vision',reasoning:'Reasoning',statuses:'Run status',backends:'Inference engine'}; export const modelFacets:FacetKey[]=['families','kinds','sizes','quantTiers','quants']; -export const conditionFacets:FacetKey[]=['benchmarks','concurrency','promptSizes','mtp','vision','reasoning','statuses']; -export const groupOptions={model:'Model',family:'Model family',kind:'Dense vs MoE',size:'Parameters',sizeBucket:'Size range',quantTier:'Quantization tier',quantization:'Exact quantization',benchmark:'Benchmark pack'} as const; +export const conditionFacets:FacetKey[]=['backends','benchmarks','concurrency','promptSizes','mtp','mtpDepth','vision','reasoning','statuses']; +export const groupOptions={model:'Model',family:'Model family',kind:'Dense vs MoE',size:'Parameters',sizeBucket:'Size range',quantTier:'Quantization tier',quantization:'Exact quantization',benchmark:'Benchmark pack',backend:'Inference engine'} as const; export type GroupBy=keyof typeof groupOptions; -export const groupValue:Recordstring>={model:r=>r.modelKey,family:r=>r.family,kind:r=>kindLabel(r.kind),size:r=>r.sizeLabel,sizeBucket:r=>r.sizeBucket,quantTier:r=>r.quantTier,quantization:r=>r.quantization||unknownFacet,benchmark:r=>r.benchmarkPack??ownTests}; -export const sortOptions={label:'Name',runs:'Runs',requests:'Requests',generationTps:'Generation tok/s',estimatedPrefillTps:'Prefill tok/s',throughput:'Total tok/s',medianMs:'Median latency',p95Ms:'p95 latency',failureRate:'Failure rate',objective:'Objective score',localJudge:'Local judge',gpuHotSpotMax:'GPU hot spot peak'} as const; +export const groupValue:Recordstring>={model:r=>r.modelKey,family:r=>r.family,kind:r=>kindLabel(r.kind),size:r=>r.sizeLabel,sizeBucket:r=>r.sizeBucket,quantTier:r=>r.quantTier,quantization:r=>r.quantization||unknownFacet,benchmark:r=>r.benchmarkPack??ownTests,backend:r=>r.backend}; +export const sortOptions={label:'Name',runs:'Runs',requests:'Requests',generationTps:'Generation tok/s',estimatedPrefillTps:'Prefill tok/s',prefillCalibratedTps:'Prefill tok/s (calibrated)',throughput:'Total tok/s',medianMs:'Median latency',p95Ms:'p95 latency',failureRate:'Failure rate',objective:'Objective score',localJudge:'Local judge',gpuHotSpotMax:'GPU hot spot peak'} as const; export type SortKey=keyof typeof sortOptions; export type HistoryView={search:string;groupBy:GroupBy;sort:SortKey;descending:boolean}&Record; const emptyFacets=()=>Object.fromEntries(facetKeys.map(k=>[k,[] as string[]])) as Record; @@ -81,7 +92,7 @@ export function matchesHistory(row:HistoryRow,view:HistoryView,ignore?:FacetKey) } const terms=view.search.toLocaleLowerCase().trim().split(/\s+/).filter(Boolean); if(!terms.length)return true; - const haystack=[row.modelKey,row.model,row.quantization,row.family,row.architecture,row.publisher,row.runName,row.test].join(' ').toLocaleLowerCase(); + const haystack=[row.modelKey,row.model,row.quantization,row.family,row.architecture,row.publisher,row.runName,row.test,row.backend].join(' ').toLocaleLowerCase(); return terms.every(t=>haystack.includes(t)); } const promptOrder=['short','medium','long',otherPrompt]; @@ -118,11 +129,11 @@ export type HistoryGroup={ runs:{id:string;name:string;created:string;status:string}[]; first:string;last:string; requests:number;completed:number;failures:number;failureRate:number|null; - generationTps:Spread;estimatedPrefillTps:Spread;throughput:Spread;ttftMs:Spread; + generationTps:Spread;estimatedPrefillTps:Spread;prefillCalibratedTps:Spread;throughput:Spread;ttftMs:Spread; medianMs:number|null;p95Ms:number|null; objective:number|null;localJudge:number|null;externalJudge:number|null; gpuHotSpotAvg:number|null;gpuHotSpotMax:number|null; - conditions:Record<'concurrency'|'promptSizes'|'mtp'|'vision'|'reasoning',string[]>; + conditions:Record<'concurrency'|'promptSizes'|'mtp'|'mtpDepth'|'vision'|'reasoning',string[]>; mixed:boolean; }; const distinct=(rows:HistoryRow[],pick:(r:HistoryRow)=>string)=>[...new Set(rows.map(pick))].filter(Boolean); @@ -136,10 +147,13 @@ function aggregate(key:string,label:string,rows:HistoryRow[]):HistoryGroup{ const conditions={ concurrency:distinct(rows,r=>String(r.concurrency)).sort((a,b)=>+a-+b), promptSizes:distinct(rows,r=>r.promptSize??otherPrompt), - mtp:distinct(rows,r=>r.mtp),vision:distinct(rows,r=>r.vision),reasoning:distinct(rows,r=>r.reasoning)}; + mtp:distinct(rows,r=>r.mtp),mtpDepth:distinct(rows,r=>mtpDepthText(rowDepth(r))),vision:distinct(rows,r=>r.vision),reasoning:distinct(rows,r=>r.reasoning)}; return {key,label,rows,models:distinct(rows,r=>r.modelKey).sort(),runs,first:created[0]??'',last:created[created.length-1]??'', requests,completed:requests-failures,failures,failureRate:requests?failures/requests:null, generationTps:weighted(rows,r=>r.generationTps,ok),estimatedPrefillTps:weighted(rows,r=>r.estimatedPrefillTps,ok), + // Measured once per loaded model rather than per request, so every row of one model carries the + // same number; weighting by requests still pools two runs of that model in the right proportion. + prefillCalibratedTps:weighted(rows,r=>r.prefillCalibratedTps,ok), throughput:weighted(rows,r=>r.throughput,ok),ttftMs:weighted(rows,r=>r.ttftMs,ok), medianMs:percentile(durations,.5),p95Ms:percentile(durations,.95), // Scores weight by responses that actually carried a score, so an ungraded run never dilutes a graded one. @@ -164,7 +178,7 @@ const sortValue=(g:HistoryGroup,key:SortKey):number|string|null=>{ case 'label':return g.label; case 'runs':return g.runs.length; case 'requests':return g.requests; - case 'generationTps':case 'estimatedPrefillTps':case 'throughput':return g[key]?.value??null; + case 'generationTps':case 'estimatedPrefillTps':case 'prefillCalibratedTps':case 'throughput':return g[key]?.value??null; default:return g[key]; } }; diff --git a/src/main.tsx b/src/main.tsx index 0750190..57cfc98 100644 --- a/src/main.tsx +++ b/src/main.tsx @@ -18,8 +18,20 @@ import {LibraryControls} from './LibraryControls'; import {BenchmarkLibrary,BenchmarkResults} from './BenchmarkPanels'; import {benchmarkConfig,summaryFor} from './benchmarks'; import {visibleModels,selectVisible,defaultLibraryView,restoreLibraryView} from './library'; +import {appVersion} from './version'; +import type {Twin} from '../electron/text-only'; import {HistoryPanel} from './HistoryPanel'; import {defaultHistoryView,restoreHistoryView,shortDate} from './history'; +import {defaultSweep,maxMtpDepth,measuredDepths,mtpDepthText,sweepSteps,sweepVerdicts} from './mtp-sweep'; +import {calibrationFor,focusBlurb,focusColumns,resultsFocuses,type ResultsFocus} from './results-focus'; +import {backendRows,comparableBackends} from './chart-compare'; +import {planTextOnlySwap,textOnlyIncomplete,textOnlySummary} from './text-only-swap'; +import type {Twin as TextOnlyTwin} from '../electron/text-only'; +import {backendOf} from '../electron/runtime'; +import type {HistoryRow} from './history'; +import {runtimeChoices,type Runtime} from '../electron/runtime'; +import {isLoopbackUrl} from './endpoint'; +import {cacheQuantLabel,cacheQuants,cacheScalePair,flashOn,type CacheQuant} from './cache-quant'; import type {UpdateState} from '../electron/update'; const api=window.bench; @@ -32,7 +44,11 @@ const toggle=(list:string[],id:string)=>list.includes(id)?list.filter(x=>x!==id) // into a real setting, silently and with no way to tell from the box. const numOr=(raw:string,previous:number)=>raw.trim()===''||!Number.isFinite(Number(raw))?previous:Number(raw); const modelLabel=(m:Model)=>`${m.nativeMtp?.supported===true?'MTP · ':''}${m.display_name} · ${m.quantization?.name??'Unknown quantization'}${m.capabilities&&'vision' in m.capabilities&&m.capabilities.vision?' · Vision':''}`; -function Badge({children,tone=''}:{children:React.ReactNode;tone?:string}){return {children}} +function Badge({children,tone='',title}:{children:React.ReactNode;tone?:string;title?:string}){return {children}} +// MTP arrives in two shapes and the difference is worth seeing on the card: heads built into +// the model file, and heads stored beside it as their own file. Both are usable; naming which +// one a model has is what stops "no MTP" from being read as "this model cannot do MTP". +const mtpBadge=(m:Model)=>m.nativeMtp?.supported===true?{tone:'green',text:m.nativeMtp.kind==='sidecar'?'MTP head':'MTP built in'}:m.nativeMtp?.supported===false?{tone:'',text:'No MTP'}:{tone:'',text:'MTP unknown'}; function Field({label,children,hint}:{label:string;children:React.ReactNode;hint?:string}){return } // Field renders a