Repository navigation
mr-hook: skip the embed wait when the embedder is not loaded (F42) - v0.42.2 - #75
Merged
Merged
Conversation
mr-hook gives the embedding call 950 ms of its one-second deadline. The embedder sits behind a model router that unloads it after a few idle minutes and needs seconds to load it again, so the first prompt after every idle spell waited the whole budget, was cancelled, and fell back to BM25 anyway. Before embedding, the hook now reads the router's GET /running (alias resolved through /v1/models only when the name is not listed) under a 100 ms ceiling. If the model is not loaded, or the router cannot be read, it uses the existing BM25 fallback at once and sends ONE warm-up request (fixed token, never prompt text) that it abandons as soon as the bytes are on the wire; abandoning it does not stop the router's load. A load already in flight is skipped without a second request. A loaded model embeds as before. Not read as cold, so these embed exactly as before: a server with no /running route, a 4xx or non-list answer, a model the router does not list, and a refused connection (a dead embedder keeps its real diagnosis). Rows log cold-skip-bm25 / cold-skip-empty instead of embedder-down, with the reason and whether a warm-up was sent in err. The probe ceiling is carved out of the embed budget so probe + embed cannot overrun the deadline. -cold-skip=false restores the old path. Tests drive the built binary against a fake router: cold latency 952 ms -> 8 ms, warm unchanged (5 ms vs 4 ms). Red against the previous code first; 15 mutations of the new code each turn the suite red. B12 budget raised to 31317 (+365 non-test LOC), version 0.42.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g (F42) The first cut abandoned the warm-up request the instant its bytes were on the wire. Against a live router that started the load in one trial and was dropped in the next (the router logged a client-closed request after 30 ms and stayed unloaded), so a cold hook could skip the embed wait and still never warm the model. The warm-up now holds its request open only until the router's /running lists the model as starting (6 ms in the live trial), then abandons it: the load survives the abandoned request and the process exiting. A 150 ms ceiling bounds the wait on a busy or silent router; a /running that cannot be read ends it at once. The usage row records which happened (load confirmed / load not confirmed within 150ms / the error). An alias is watched under the router's real id. Tests: a fake router that lists the load once the embeddings call arrives, a silent one that never does (the hook still exits inside the ceiling), the alias, stopping-is-not-loading, unobservable and refused cases. B12 budget 31337. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d requests, status check, clamp, stuck mode (F42)
1. The warm-up fakes now behave like the live router: a load begins only for
a request still open when the router admits it (25 ms in the fakes, about
6 ms live); a client that left first is dropped and nothing loads. A
warm-up that writes, cancels at once and keeps polling /running can no
longer pass: that mutant is red in the unit and the binary-level tests.
2. WarmUpEmbedder reads the HTTP status of the warm-up answer. A 4xx/5xx is an
error ("embeddings answered status N", recorded in the row), never "ready".
3. The warm-up ceiling is clamped to the deadline like the probe's (a quarter
of the embed budget, at most 150 ms), so probe + warm-up stay under half
of it for any -timeout-ms; the row names the budget it actually had.
4. A reachable router that never gets the embedder loaded used to be loud
(the embed call erroring) and is now an ordinary-looking cold skip forever.
Three earlier consecutive cold rows in the hook's own usage log (one bounded
64 KiB tail read, cold path only), the oldest two minutes old, now log
cold-stuck-bm25 / cold-stuck-empty. Limit: counted per log, not per model.
5. README: the warm-up is held until the router admits it (not "does not hold
it open"), and the rollback is shown as -cold-skip=false (the = is
required by Go's flag package).
6. The binary-level tests use the 5000 ms deadline of the older e2e tests and
bounds relative to the fake embedding's 3 s, not absolute 200/800 ms.
33 mutations (the original 20 plus 13 for these items) each turn the suite
red. B12 budget 31434 (+97).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mr-hook: skip the embed wait when the embedder is not loaded (F42) - v0.42.2
The defect. The prompt hook embeds every prompt through a model router with a ~1000 ms budget. When the embedding model was not loaded (it unloads after a few idle minutes), the hook waited out ~950 ms on the cold load and logged
embedder-down, on every first prompt after an idle spell.The fix.
GET /runningfirst (100 ms cap, a quarter of the embed budget at most), walking the endpoint chain the wayembed()does and resolving aliases through/v1/models.cold-skip-bm25/cold-skip-empty), and fire one fixed-word warm-up (never prompt text) that is held only until the router lists the model asstarting(cap clamped to the deadline), then abandoned; the load survives the hook's exit.cold-stuck-*, so a router that can never load the embedder stays visible (counted per usage log, not per model: a named limit)./running, a non-list 200, an unlisted model, a refused connection) are not read as cold: they embed exactly as before.-cold-skip=falserestores the old path byte for byte.Measured. Fake router, built binary: cold 951 ms latency before, 50 ms after; warm 5 ms with the probe, 4 without. Live router, cold: two trials at 60 and 52 ms with the load confirmed
startingwithin ~6 ms and surviving the hook's exit.Verification. Red tests first; 33 mutants, all red (including a "cancel the warm-up at once and keep polling" mutant that the first test set missed, found by an adversarial review: the fakes now drop a request that is not held open, as the real router does). Full suite: 53 packages ok. Two-lens adversarial review, every finding refute-checked; the MEDIUM and the LOWs are fixed here; four LOWs are parked in the CHANGELOG (alias records
/runningnever reports, the probe not applyingembed()'s admission test to unverified candidates, no hysteresis on a slow/running, a 5xx during the hold abandoning the POST).🤖 Generated with Claude Code