High token use?

[amotl]

Problem

According to this comment, CrateDB's llms-full.txt weighs in rather heavy, with a token usage on OpenAI of:

{"input": 212817, "output": 1139, "total": 213956}

Solution

Instead of feeding the full uncompressed knowledge context, either serve individual elements on demand per MCP, or try to compress it.

• Outline: Expand into MCP server #6
• llm-min.txt: Min.js-style compression to reduce LLM context size #30

[amotl]

Probing open models

In order to investigate reducing service costs, we are evaluating to run the language model on hardware instead of consuming it through a remote API.

In order not to cause high input token counts, some implementations limit the context window. With Ollama in its default configuration on some contemporary standard model ¹², it looks like this:

level=WARN source=runner.go:128 msg="truncating input prompt" limit=4096 prompt=17769 keep=5 new=4096

In order to accompany the relatively large llms-full.txt the machinery is currently producing, the corresponding configuration setting would need to be adjusted.

export OLLAMA_CONTEXT_LENGTH=250000

On an ill-fitted machine just bearing AMD Radeon Pro 5600M and Intel UHD Graphics 630 which does not support any kind of GPU number crunching, CPU-bound inference, specifically when using large context windows, is not feasible, even when selecting variants with small parameter sizes: Bigger irons are needed for such purposes - Cloud Run GPUs to the rescue?

Thoughts

Most probably, consuming documenation snippets through MCP on demand is way to go, so the llms.txt route can be easily abandoned.

Footnotes

1. $ ollama list
   NAME               ID              SIZE      MODIFIED
   deepseek-r1:7b     0a8c26691023    4.7 GB    21 hours ago
   llama3.2:latest    a80c4f17acd5    2.0 GB    21 hours ago
   qwen3:0.6b         3bae9c93586b    522 MB    21 hours ago
   gemma3:1b          8648f39daa8f    815 MB    21 hours ago
   

   ↩

2. https://ollama.com/search ↩

[amotl]

Cloud Run GPUs to the rescue?

Cloud Run GPUs, now GA, makes running AI workloads easier for everyone.
https://cloud.google.com/blog/products/serverless/cloud-run-gpus-are-now-generally-available
https://news.ycombinator.com/item?id=44178468

[amotl]

Enumerating a few GPU-for-rent offerings.

• https://www.hetzner.com/dedicated-rootserver/matrix-gpu/
• https://www.ori.co/dedicated-endpoints
• https://www.runpod.io/

[amotl]

Maximum context sizes of default LLMs are 128000 (?) or 200000 (o4-mini). Only gpt-4.1 accepts our llms-full.txt bearing 212786 tokens. Maybe decrease size?

• Query: Use llm program instead of cratedb-about ask #45 (comment)

[amotl]

Also about token budgets, we found that both claude-4-sonnet and claude-4-opus will reject the large blob of the current knowledgebase representation (207831 tokens), also maxing out at 200000 input tokens. That certainly needs to be shrinked.

llm chat --model claude-4-sonnet --option temperature 0.0 \
  --fragment https://cdn.crate.io/about/v1/llms-full.txt \
  --system-fragment https://cdn.crate.io/about/v1/instructions-general.md

anthropic.BadRequestError: Error code: 400 - 
{'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'prompt is too long: 207831 tokens > 200000 maximum'}}

So, currently, only GPT-4.1 works:

Today, we’re launching three new models in the API: GPT‑4.1, GPT‑4.1 mini, and GPT‑4.1 nano. These models outperform GPT‑4o and GPT‑4o mini across the board, with major gains in coding and instruction following. They also have larger context windows—supporting up to 1 million tokens of context—and are able to better use that context with improved long-context comprehension. They feature a refreshed knowledge cutoff of June 2024.

[amotl]

Even when omitting the "Optional" section of the llms.txt, decreasing the file size from ~1.1 MB to ~850 kB, and also the system fragment, it doesn't reduce the token count. Why?

prompt is too long: 208075 tokens > 200000 maximum

[amotl]

@WalBeh just suggested to look into fine-tuning LLMs in order to decrease context sizes for knowledgebase curation and documentation inquiry purposes. Today, there is probably a plethora of possibilities, e.g. by using PyReft, LoRA, or Unsloth (video).