Local mode: storage.sqlite is ~2.3x the size of the vectors it holds

[Gronoxx]

Current behavior

CollectionPersistence.persist() pickles the whole PointStruct. By then pydantic has already coerced the vector to list[float] or list[list[float]], and a Python float costs about 9 bytes in a pickle against 4 as float32. The file on disk ends up roughly 2.3x the size of the data it holds.

import pickle, numpy as np
from qdrant_client import models

rng = np.random.default_rng(0)
for label, shape in [("dense 384-d", (384,)), ("dense 1024-d", (1024,)), ("multivector 390x1024", (390, 1024))]:
    v = rng.normal(size=shape).astype(np.float32)
    vec = v.tolist() if len(shape) == 1 else {"colbert": v.tolist()}
    n = len(pickle.dumps(models.PointStruct(id=1, vector=vec)))
    print(f"{label:<22} pickle={n:>9d} B | float32={v.nbytes:>8d} B | {n/v.nbytes:.2f}x | {n/v.size:.2f} B/float")

On 1.18.0:

dense 384-d            pickle=     3653 B | float32=    1536 B | 2.38x | 9.51 B/float
dense 1024-d           pickle=     9415 B | float32=    4096 B | 2.30x | 9.19 B/float
multivector 390x1024   pickle=  3597276 B | float32= 1597440 B | 2.25x | 9.01 B/float

End to end, 60 points of 200x512 produced a 53.0 MiB storage.sqlite for 23.4 MiB of float32 payload.

The caller cannot work around it

All three ways of passing the vector give byte-identical pickles, because pydantic coerces every one of them to list[list[float]] before serialization:

tolist()        internal type=list   pickle=3.43 MiB
ndarray         internal type=list   pickle=3.43 MiB
list(ndarray)   internal type=list   pickle=3.43 MiB

Expected behavior

A collection on disk stays close to the size of the vectors in it. Storing the vector as a float32 buffer with its shape, rather than a pickle of Python floats, would do that.

Scope, honestly

This is a design limitation of the persistence format, not a behavior defect, and fixing it changes the on-disk format, so it needs a migration path. I am raising it as an issue with the numbers rather than a PR, since the format decision is yours.

It also matters less than the byte count suggests, for two reasons worth stating:

• Local mode is documented for development, prototyping and testing, and 2.3x of a prototype-sized collection is not a problem.
• The memory side is being addressed separately. Deserializing those Python float lists dominates the peak when reopening a collection, and fix: rebuild local-mode vectors on reload the way the write path stores them #1381 plus a follow-up on CollectionPersistence.load() take that peak from 482 MiB to 54 MiB on a 47 MiB collection. Disk size is what remains after those.

Where it did bite: 2,734 chunks with ColBERT multivectors produced 8.5 GB of storage.sqlite for about 3.8 GB of float32. That still falls under "switch to server mode when you need to scale", which is what I did.

Related, though neither identifies this cause: #239 (sqlite file not pruned after deletes), #570 (upsert slow for sparse embeddings).

[generall]

Qdrant local mode is not designed for performance. It's function is to serve as a qdrant mock for small-scale experiments, CI/CD, tests e.t.c.

Please prefer to use docker version of qdrant or Qdrant Edge to get best optimizations and performance

[Gronoxx]

Understood, thanks for clarifying the scope. Closing this one.