fix(loki): shed facts on a 413 instead of abandoning the tool loop - #223
Merged
Merged
Conversation
Browser-tested the previous fix and read the box log. The 429 step-down
fired exactly as designed:
[loki] llama-3.3-70b-versatile rate-limited, stepping down to
llama-3.1-8b-instant
[loki] tool loop failed: groq 413: Request too large for model
`llama-3.1-8b-instant`
A prompt sized for a 128k context does not fit a small one. So the loop
still gave up and fell back to the weaker-retrieval path, and the
step-down that was supposed to keep tools working achieved nothing — it
just moved the failure one line down the log.
The answer the operator saw was correct, but only because the FALLBACK
path now has fixed people lookup. The tool loop itself was not serving
production turns at all.
Facts are the elastic part of the prompt (projects plus accumulated tool
results) and the first ones are the most relevant, so a 413 now halves
the fact budget and retries: 40 -> 20 -> 10, which fits every model in
the chain. Non-413 errors still propagate immediately — retrying a 401
or a 500 only burns latency for a guaranteed second failure.
Both behaviours pinned in scripts/test/agent-tool-loop.ts (11 checks).
67/67 unit files pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Browser-tested #222 and read the box log. The 429 step-down fired exactly as designed — and then hit the next layer:
A prompt sized for a 128k context does not fit a small one. So the loop still gave up and fell back to the weaker-retrieval path — the step-down achieved nothing except moving the failure one line down the log.
The answer the operator saw was correct, but only because the fallback path now has fixed people lookup (#222). The tool loop itself was not serving production turns at all.
Fix
Facts are the elastic part of the prompt (projects + accumulated tool results), and the first ones are the most relevant. A 413 now halves the fact budget and retries:
which fits every model in the chain. Non-413 errors still propagate immediately — retrying a 401 or a 500 only burns latency for a guaranteed second failure.
Verification
Both behaviours pinned in
scripts/test/agent-tool-loop.ts(now 11 checks): a scripted model that 413s twice must produce a shrinking fact set and still answer, and a 401 must propagate on the first attempt without a retry. 67/67 unit files pass, tsc clean, 0 lint errors.🤖 Generated with Claude Code