Improve retrieval quality with reranking layer

[Ayush-kathil]

Improve retrieval quality with simple reranking

Currently we rely on vector similarity alone to pick top-k documents. In some cases this returns chunks that are close semantically but not the most useful for answering the query.

A simple improvement would be to introduce a reranking step after retrieval:

• expand the candidate pool slightly
• rerank using additional signals (e.g. keyword overlap, metadata)
• select top-k after reranking

This should improve answer quality without adding much complexity.

I can work on an initial version of this if this direction makes sense.

[Flux07-gezz]

Hey @Ayush-kathil !
I'd like to work on this improvement.

I explored the current retrieval flow and found search_kubeflow_docs() where Milvus currently returns the top-k results directly. My plan is to expand the retrieval candidate pool, add a lightweight reranking step using semantic similarity + keyword overlap (and potentially metadata signals later), then select the final top-k results after reranking.

Starting with a simple implementation first to keep complexity low.