[Bug]: tokenizer_config max_length caps several models at 128 tokens

[Dylancouzon]

What happened?

load_tokenizer takes min(model_max_length, max_length) when both keys exist in tokenizer_config.json (fastembed/common/preprocessor_utils.py:47). Several Qdrant-hosted ONNX repos carry a "max_length": 128 that the upstream model repo does not, so input is truncated at 128 tokens with no warning.

For two models this contradicts the shipped description:

model	description says	actual limit
thenlper/gte-large	512 input tokens truncation	128
sentence-transformers/all-MiniLM-L6-v2	256 input tokens truncation	128

Two more repos carry the same key where the upstream repo has none: Qdrant/Splade_PP_en_v1 (upstream sentence_bert_config.json is max_seq_length: 512) and Qdrant/all_miniLM_L6_v2_with_attentions (BM42). Their descriptions carry no token count.

qdrant/paraphrase-multilingual-MiniLM-L12-v2-onnx-Q also caps at 128, and there it matches the upstream. Whether the four above are deliberate is a question for maintainers.

What is the expected behaviour?

The effective limit matches the description, and a limit below the upstream default is a deliberate choice.

Related: there is no way to change the limit through the API. enable_truncation is called once from the config file, and no max_length kwarg reaches the tokenizer, so the only override today is specific_model_path with an edited tokenizer_config.json.

A minimal reproducible example

from fastembed import TextEmbedding, SparseTextEmbedding

print(TextEmbedding("sentence-transformers/all-MiniLM-L6-v2").model.tokenizer.truncation["max_length"])  # 128, description says 256

m = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
doc = "lorem ipsum " * 200 + "zebra"
print(m.model.tokenizer.token_to_id("zebra") in list(m.embed([doc]))[0].indices)  # False

The second check shows the effect: a term past position 128 is absent from the sparse vector.

What Python version are you on? e.g. python --version

Python 3.11, uv

FastEmbed version

0.8.0

What os are you seeing the problem on?

MacOS

[Dylancouzon]

Related open PRs in this code path. None of them covers the repo configs above.

• Fix unsafe max context #500 touches the same enable_truncation call, for the sys.maxsize sentinel case. The review there asks for the real limit to be read from config.json (max_position_embeddings) instead of clamped, which would give thenlper/gte-large its 512.
• Handle model_max_length / max_length = 0 in tokenizer config (fixes #685) #687 handles max_length: 0 in the same function ([Bug]: models with max_length=0 in tokenizer config break #685).
• docs: align MiniLM truncation metadata #639 corrects the description for sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 to 128, matching that model's upstream max_seq_length: 128. That fix is right for that model. The two models in the table here are the reverse case: their descriptions match upstream and the config sits below it.

[denis-samatov]

Thanks for the detailed reproducer and the cross-links. I traced the same loader path and see three distinct choices here: correct the affected model metadata, expose an explicit max_length override while preserving the current default, or change the automatic precedence using config.json:max_position_embeddings. The last option looks most likely to change existing behavior unexpectedly. Would maintainers prefer a focused PR for the metadata path or for a validated public override? I’d be happy to implement the chosen scope with regression tests covering the affected models and the neighboring #500/#687 cases.