[Bug]: gte-base mixed-length batches fail with fixed padding in 0.8.0

[Dylancouzon]

What happened?

thenlper/gte-base fails on 0.8.0 when a batch contains texts with different final encoded lengths; it worked on 0.7.4.

load_tokenizer sets truncation unconditionally but padding only when absent (fastembed/common/preprocessor_utils.py:52-56):

tokenizer.enable_truncation(max_length=max_context)
if not tokenizer.padding:
    tokenizer.enable_padding(...)

gte-base ships fixed padding 128 with model_max_length: 512 (revision c078288). Fixed padding never shortens an encoding already longer than it, so rows come back ragged and fail at fastembed/text/onnx_text_model.py:82.

Version	Padding	Example lengths	Result
0.8.0	fixed 128	[128, 202]	ValueError
0.7.4	dynamic	[202, 202]	works

Regression introduced by commit 800f388 (#588), tagged only in v0.8.0/v0.8.0-gpu.

Scope: Of 34 TextEmbedding models with an HF source, four use fixed padding; only gte-base sets it below truncation. Sparse, late-interaction, cross-encoder, and multimodal registries were not audited. add_custom_model repositories are also exposed.

The failure requires two items with differing final encoded lengths; a single document or batch_size=1 works. Distinct from #689, which concerns how max_context is selected.

What is the expected behaviour?

Embedding should support mixed-length batches. If fixed padding is below the truncation limit, use batch-longest padding while preserving padding direction and metadata. min(p, max_context) does not help.

I have a patch and regression tests ready and can open a PR.

A minimal reproducible example

from fastembed import TextEmbedding

model = TextEmbedding("thenlper/gte-base")
print(model.model.tokenizer.padding["length"])          # 128
print(model.model.tokenizer.truncation["max_length"])   # 512

list(model.embed(["hello world", "retrieval " * 200]))  # ValueError
list(model.embed(["retrieval " * 200]))                 # fine on its own

What Python version are you on? e.g. python --version

Python 3.12.14, uv

FastEmbed version

0.8.0

What os are you seeing the problem on?

Linux — Ubuntu 26.04 LTS under WSL2. Also tokenizers==0.22.2, numpy==2.3.5, onnxruntime==1.29.0.

Relevant stack traces and/or logs

File "fastembed/text/onnx_text_model.py", line 82, in onnx_embed
  input_ids = np.array([e.ids for e in encoded])
ValueError: setting an array element with a sequence. The requested array has an
inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.