perf: add benchmark for ChromaDB retrieval latency (#311)#328
perf: add benchmark for ChromaDB retrieval latency (#311)#328Aharshi3614 wants to merge 1 commit into
Conversation
imDarshanGK
left a comment
There was a problem hiding this comment.
@Aharshi3614 Thanks for the contribution. The PR is generally aligned with Issue #311, but I noticed a few things that need to be addressed before it can be merged:
- In
test_single_query_latency, the_retrieve()arguments appear to be reversed.
Current code:
context, docs = _retrieve("document information topic", session_id, top_k=4)
But _retrieve() is defined as:
def _retrieve(session_id: str, query: str, top_k: int = 4):
It should be:
context, docs = _retrieve(session_id, "document information topic", top_k=4
-
The collection cleanup should use the same collection naming logic as creation to avoid potential mismatches in the future.
-
If possible, avoid loading
SentenceTransformerat module import time, since it can slow down test collection and CI runs.
Please update the PR and let me know once the changes are pushed.
What this does
Adds a benchmark test suite for ChromaDB retrieval latency in
backend/tests/test_chromadb_benchmark.py.Why
Closes #311
Benchmark Results
| Test | Result |
| Single query latency | 11.886ms |
| Mean over 10 runs | 20.248ms |
| Small collection (10 chunks) | 17.206ms |
| Large collection (200 chunks) | 19.899ms |
| Empty collection | 7.822ms |
| Latency stdev (consistency) | 1.922ms |
Test Coverage
Testing
pytest tests/test_chromadb_benchmark.py -v -s
7 passed in 21.33s