llm_hub: add upfront WARN-ONLY input-overflow check against max_context - #1943
llm_hub: add upfront WARN-ONLY input-overflow check against max_context#1943arash77 wants to merge 2 commits into
Conversation
This is all automatics, we can not do that :( |
Is it worth doing, then? Our providers could actually supply this via an API endpoint like |
6803a4f to
ebe1618
Compare
|
Is CZ supporting it and UFR not? I would ask UFR to add it. |
Neither supports it right now. |
|
Interesting, ok lets ask them and see how complicated that is. |
|
ok RZ looked into it and it seems customizing the api pass through routes is a premium freature in LiteLLM! |
Estimate the input token count locally with tiktoken (cl100k_base, offline) and compare it to a new max_context column in the genai_models data table. When the estimate exceeds the model's context window, warn on stderr but still proceed -- the check never blocks the job. Image tokens are not counted, so the estimate is a lower bound for multimodal inputs. The check is skipped silently when max_context is empty/0/unknown or tiktoken is unavailable, so it is purely advisory. Add a 7th max_context column to the genai_models data table (sample, conf, and .loc documentation), a tiktoken conda dependency, and bump the Galaxy version suffix. Document that tool_data_table_conf.xml and genai_models.loc must be updated together: a row with fewer than 7 tab-separated fields is rejected (pad max_context with an empty value if unknown), and a 6-column declaration makes the tool form fail to render.
ebe1618 to
ffb37c0
Compare
The max_context column cannot be added to genai_models in place: that .loc file is generated automatically, and the 6- and 7-column layouts are mutually incompatible (a 7-column <columns> declaration rejects every 6-field row, and a 6-column declaration against a 7-field row makes the tool form fail to render). Extending it would therefore require every deployment to update tool_data_table_conf.xml and the generated .loc in lockstep. Rename the table instead, so the 7-column table lives alongside the untouched genai_models. Existing deployments keep working with no coordinated change, and admins opt in by generating the new file with the max_context field. Rename tool-data/genai_models.loc.sample accordingly, repoint the three from_data_table references, and document the migration path in the .loc sample header and README.
2 participants
Summary
Adds an upfront, WARN-ONLY input-overflow check to the
llm_hubtool. Before sending the request to the LLM, the tool now estimates the input token count locally withtiktoken(cl100k_base, offline — no proxy calls) and compares it to the selected model's context window. If the estimate exceeds the window, it prints aWARNINGto stderr but still proceeds — the check never blocks the job.This surfaces the most common failure mode (input too large → silently truncated/empty output) before the user waits on a long generation, while remaining purely advisory.
What changed
llm_hub.py: newestimate_input_tokens()+ a module-level guard. Counts text content only; image tokens are not counted (no reliable local estimate), so the count is a lower bound for multimodal inputs. Usesdisallowed_special=()so literal<|endoftext|>-style sequences don't crash the counter. All failure modes (tiktoken unavailable, network-restricted node unable to fetch the BPE vocab on first use, malformedmax_context) are swallowed silently — the check is advisory and never aborts the job.max_contextcolumn (7th) in thegenai_modelsdata table: sample.loc,tool_data_table_conf.xml.sample, and.locdocumentation updated.max_contextis the model's full context window in tokens (e.g.128000); leave empty or0if unknown, in which case the check is skipped for that model.macros.xml/llm_hub.xml: add atiktokenconda dependency (@TIKTOKEN_VERSION@0.13.0) and bump the Galaxy version suffix3 → 4.Deployment note (verified against a live Galaxy)
tool_data_table_conf.xmlandgenai_models.locmust be updated together. This was tested end-to-end against a running Galaxy (ToolDataTableManager+ CheetahNameMapper):<columns>declaration against a.locrow with fewer than 7 tab-separated fields causes Galaxy to reject the entire row (Line N ... is invalid). Pad themax_contextfield with an empty value if unknown — that is the supported "I don't know the window" state..locmakes$model.fields.max_contextraiseNotFound, so the tool form fails to render.The
.loc.samplenow documents this coupling explicitly.Testing
planemo test—All 1 test(s) executed passed. llm_hub (Test #1): passed128000,128,000,0, empty, whitespace, non-numeric), token estimation (text, multimodal-with-image, special-token safety), and gracefulNonereturn when tiktoken is unavailable.Backward compatibility
Existing 6-column
genai_models.locdeployments keep working provided admins updatetool_data_table_conf.xmlto the 7-column declaration and pad each row to 7 fields (emptymax_contextis fine). The existing tool test (no API key configured) is unchanged and still passes.FOR CONTRIBUTOR: