[QUESTION] - Search Optimization for 40 millions vectors in 3072 dimensions in 1 collection

[nicmax09]

Hi,

I hope you are doing well.

I am building a Qdrant database locally, with at least 30 millions vectors embedded in 3072 dimensions, using OpenAI embedding-large-3 model after having made a few tries on local models. Each vector represent a phrase scraped from forums.

I found Qdrant truly amazing, as it has a large number of features and optimization settings. I am quite new to AI with python.

My current configuration is the following :
{ "params":{ "vectors":{ "size":3072 "distance":"Cosine" "on_disk":true } "shard_number":1 "replication_factor":1 "write_consistency_factor":1 "on_disk_payload":true } "hnsw_config":{ "m":16 "ef_construct":100 "full_scan_threshold":10000 "max_indexing_threads":0 "on_disk":false } "optimizer_config":{ "deleted_threshold":0.2 "vacuum_min_vector_number":1000 "default_segment_number":0 "max_segment_size": NULL "memmap_threshold": NULL "indexing_threshold":20000 "flush_interval_sec":5 "max_optimization_threads": NULL } "wal_config":{ "wal_capacity_mb":32 "wal_segments_ahead":0 } "quantization_config":{ "binary":{ "always_ram":true } } "strict_mode_config":{ "enabled":false } }

When I try to retrieve the top 1000 comments similar to a search vectorized query, this setup makes the search last only 10 seconds. But when I try to search for the top 5000, the search lasts 86 seconds and for the top 10000, more than 120 seconds. The search time is not linear to the search limit.

My current search query is the following :
search_result = qdrant_client.query_points( collection_name=REVIEWS_COLLECTION_NAME, query=embedded_query, with_payload=True, limit=top_k, search_params = models.SearchParams(quantization=models.QuantizationSearchParams( ignore=False, rescore=True, oversampling=2.0, )), score_threshold=0.6, timeout=120 )

And I am running qdrant on a VM of 64 gb of RAM, 16 vCPUs and 1TB of memory.

My question is : Do you have any tip on how to improve my configuration, increasing the speed of search and not decreasing too much the accuracy ? I have already tried to put my HNSW on RAM, but it takes too much space.

Thanks a lot for you help.

[eeshsaxena]

the tests folder is actually super useful for this kind of thing. the unit tests basically double as usage examples and are sometimes more up to date than the docs.

also worth searching closed issues with the same keywords, someone has usually asked something similar and a maintainer replied with the right pattern.

what have you tried so far? if you drop a snippet here it is easier to spot what is off.

[denis-samatov]

The non-linear jump is expected with this combination:

• a very large limit;
• binary-quantized search;
• oversampling=2.0;
• rescore=True;
• original vectors stored on disk;
• payload returned for every hit.

With 3,072-dimensional float32 vectors, one original vector is 12 KiB. A limit=10_000 query with oversampling=2.0 can preselect and rescore about 20,000 candidates, which means reading roughly:

20,000 × 3,072 × 4 bytes ≈ 234 MiB

of original vector data per query, before HNSW work, payload reads, result serialization, and network transfer. At limit=5_000, the corresponding amount is about 117 MiB. Because those original vectors are on disk, page-cache misses and random I/O can create the sharp latency threshold you are seeing.

The Qdrant quantization documentation explicitly notes that rescoring with original on-disk vectors can be slow; binary quantization accelerates candidate selection, but rescoring still needs the original vectors.

I would benchmark these factors independently:

search_params=models.SearchParams(
    quantization=models.QuantizationSearchParams(
        ignore=False,
        rescore=False,  # fastest path; measure recall separately
    )
)

Then compare:

1. rescore=False versus True;
2. oversampling=1.0, 1.2, and 2.0;
3. with_payload=False versus True;
4. smaller result limits.

If recall requires rescoring, reduce oversampling and keep the rescored candidate/result set small. Fetch IDs and scores first with with_payload=False, then retrieve payload only for the final selected IDs. score_threshold usually does not eliminate the candidate-collection and rescoring cost, so it is not a substitute for reducing limit.

There is also a hard sizing constraint here: 40 million × 3,072 × 4 bytes is about 458 GiB (491.5 GB decimal) of raw original vectors. They cannot remain memory-resident on a 64 GB machine. Keeping binary vectors in RAM helps the first stage, but it does not remove original-vector I/O during rescoring. More RAM/faster NVMe, or a lower-dimensional embedding model (including a suitable Matryoshka model), can materially change this profile.

Finally, if you genuinely need 5,000–10,000 results per request for analytics/export, an interactive ANN top-k query may be the wrong primitive. Consider batching that work offline or filtering/scrolling candidates and scoring them in a dedicated pipeline.

Reference: https://qdrant.tech/documentation/manage-data/quantization/

If this solves your issue, please consider marking this comment as the answer!