chore(release): 0.5.0, with fresh benchmarks and a stateless smoke - #14
Merged
Merged
Conversation
The counter was wired to OpenRouter's URL, so the token table needed an account there and nothing else. Take the endpoint, the model and the key from BENCH_TOKEN_* instead, defaulting to DeepSeek, and keep reading the OPENROUTER_* pair so an older setup still runs. Subtract the chat template while we are here. The provider bills ~83 tokens before the payload starts, which a raw prompt_tokens reading charged to the response: a 74-character sample came back as 112 tokens, a ratio below one. Take that floor once per endpoint and take it off every sample.
The published numbers dated from v0.4.0 and the tool-surface row had been edited by hand since, in KiB while its neighbours were in KB, off a tool count its own summary contradicted. Rerun everything on the 2026-07-28 build, same host, same key, same 50 iterations, and take every row from that one run. First run with a token table. Cold start earned a warning in the results README: it is three spawns and nothing else, so a run taken while the workstation was busy doubled it for all three servers at once.
The PowerShell smokes all open an initialize handshake, so nothing exercised the stateless path outside unit tests: a confirmation that survives the retry and then runs the command, a decline that fails closed, and a task polled until it hands back its output. Two things it pinned down that were guesses before. A finished task has no fetch method. tasks/result answers -32601; a completed tasks/get inlines the CallToolResult next to the status. The decline case needs its own command. Reusing the approved one passes for the wrong reason: confirm_ttl still remembers the yes.
Version in the three places the check job compares, plus a CHANGELOG the generated release notes cannot produce: what the 2026-07-28 support changes for a client, and why tools/list moved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark refresh, a smoke test for the 2026-07-28 path, and the 0.5.0 version bump.
The published benchmark numbers dated from v0.4.0, and the tool-surface row had been edited by hand since: in KiB while its neighbours were in KB, against a tool count its own
summary.mdcontradicted. Everything is rerun on this build, same host, same key, same 50 iterations, and every row comes from that one run. Receipts inbenchmark/results/v0.5.0/.The token counter was wired to OpenRouter's URL. It now takes the endpoint, model and key from
BENCH_TOKEN_*, defaulting to DeepSeek, and still reads theOPENROUTER_*pair. It also subtracts the provider's chat template, roughly 83 tokens billed before the payload starts, which a rawprompt_tokensreading was charging to the response: a 74-character sample came back as 112 tokens, a ratio below one.scripts/stateless-smoke.pydrives the stateless path against a real host, which nothing did before: the PowerShell smokes all open aninitializehandshake. It covers a confirmation that survives the retry and then runs the command, a decline that fails closed, and a task polled until it returns its output. Two things it settled: a finished task has no fetch method,tasks/resultanswers -32601 and a completedtasks/getinlines theCallToolResult; and the decline case needs its own command, since reusing the approved one passes for the wrong reason whileconfirm_ttlstill remembers the yes.Version bumped in
Cargo.toml, bothserver.jsonfields and the lockfile, with aCHANGELOG.mdcovering 0.5.0 and the 0.4.x line.