Skip to content

perf: add benchmark for ChromaDB retrieval latency (#311)#328

Open
Aharshi3614 wants to merge 1 commit into
imDarshanGK:mainfrom
Aharshi3614:perf/chromadb-retrieval-benchmark
Open

perf: add benchmark for ChromaDB retrieval latency (#311)#328
Aharshi3614 wants to merge 1 commit into
imDarshanGK:mainfrom
Aharshi3614:perf/chromadb-retrieval-benchmark

Conversation

@Aharshi3614

Copy link
Copy Markdown
Contributor

What this does

Adds a benchmark test suite for ChromaDB retrieval latency in backend/tests/test_chromadb_benchmark.py.

Why

Closes #311

Benchmark Results

| Test | Result |
| Single query latency | 11.886ms |
| Mean over 10 runs | 20.248ms |
| Small collection (10 chunks) | 17.206ms |
| Large collection (200 chunks) | 19.899ms |
| Empty collection | 7.822ms |
| Latency stdev (consistency) | 1.922ms |

Test Coverage

  • Single query latency
  • Repeated query latency (10 runs with stats)
  • Small collection (10 chunks)
  • Large collection (200 chunks)
  • top_k variation (1, 4, 8)
  • Empty collection handling
  • Latency consistency check

Testing

pytest tests/test_chromadb_benchmark.py -v -s
7 passed in 21.33s

@imDarshanGK imDarshanGK left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Aharshi3614 Thanks for the contribution. The PR is generally aligned with Issue #311, but I noticed a few things that need to be addressed before it can be merged:

  1. In test_single_query_latency, the _retrieve() arguments appear to be reversed.

Current code:
context, docs = _retrieve("document information topic", session_id, top_k=4)

But _retrieve() is defined as:
def _retrieve(session_id: str, query: str, top_k: int = 4):

It should be:
context, docs = _retrieve(session_id, "document information topic", top_k=4

  1. The collection cleanup should use the same collection naming logic as creation to avoid potential mismatches in the future.

  2. If possible, avoid loading SentenceTransformer at module import time, since it can slow down test collection and CI runs.

Please update the PR and let me know once the changes are pushed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a benchmark for ChromaDB retrieval latency

2 participants