Skip to content

score_threshold is applied backwards for Recommend/Discover/Context queries on Euclidean or Manhattan collections (local mode) #1370

Description

@VANDRANKI

Describe the bug

In local/in-memory mode, score_threshold is compared in the wrong direction for Recommend (best_score / sum_scores strategy), Discover and Context queries whenever the collection's vector distance is Euclidean or Manhattan. This makes query_points (and the recommend/discover local implementations that feed it) silently return an empty (or truncated) result set even when the threshold should trivially be satisfied.

Root cause

In qdrant_client/local/local_collection.py, LocalCollection.search():

required_order = distance_to_order(distance)

if required_order == DistanceOrder.BIGGER_IS_BETTER or isinstance(
    query_vector,
    (
        DiscoveryQuery,
        ContextQuery,
        RecoQuery,
        MultiDiscoveryQuery,
        MultiContextQuery,
        MultiRecoQuery,
    ),  # sparse structures are not required, sparse always uses DOT
):
    order = np.argsort(scores)[::-1]
else:
    order = np.argsort(scores)

...

if score_threshold is not None:
    if required_order == DistanceOrder.BIGGER_IS_BETTER:
        if score < score_threshold:
            break
    else:
        if score > score_threshold:
            break

Recommend (best_score/sum_scores), Discover and Context queries always compute a sigmoid-based synthetic score where bigger is always better, regardless of the collection's underlying distance metric - calculate_recommend_best_scores, calculate_discovery_scores, calculate_context_scores (and their multi-vector counterparts) all pass their raw distances through scaled_fast_sigmoid/fast_sigmoid. This is exactly why the sort-order check above special-cases these query types with an isinstance(...) override on top of required_order.

The score_threshold check a few lines below does not repeat that override - it branches solely on required_order, which is derived only from distance_to_order(distance) (EUCLID/MANHATTAN => SMALLER_IS_BETTER). So on a Euclidean/Manhattan collection, the threshold ends up compared with the operator meant for raw distances, even though the actual score in hand is the "bigger is better" synthetic one. Since points are iterated best-first, the loop breaks immediately on the very first (best-scoring) point, discarding the whole result set.

Reproduction (verified against current master)

qdrant_client/local/local_collection.py was fetched fresh from https://raw.githubusercontent.com/qdrant/qdrant-client/master/qdrant_client/local/local_collection.py and diffed against the file used for this reproduction - identical (only CRLF/LF whitespace differences). Reproduced with qdrant-client==1.17.1 from PyPI, in local/in-memory mode (no server required).

Minimal repro:

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")
client.create_collection(
    "recommend_euclid",
    vectors_config=models.VectorParams(size=2, distance=models.Distance.EUCLID),
)
client.upsert(
    "recommend_euclid",
    points=[
        models.PointStruct(id=1, vector=[0.0, 0.0]),
        models.PointStruct(id=2, vector=[1.0, 0.0]),
        models.PointStruct(id=3, vector=[2.0, 0.0]),
        models.PointStruct(id=4, vector=[3.0, 0.0]),
        models.PointStruct(id=5, vector=[10.0, 0.0]),
    ],
)
query = models.RecommendQuery(
    recommend=models.RecommendInput(positive=[1], strategy=models.RecommendStrategy.BEST_SCORE)
)

print(client.query_points("recommend_euclid", query=query, limit=10).points)
print(client.query_points("recommend_euclid", query=query, limit=10, score_threshold=0.05).points)

Output:

Recommend (BEST_SCORE) on a EUCLID collection, no score_threshold:
  id=2 score=0.25
  id=3 score=0.09999999403953552
  id=4 score=0.050000011920928955
  id=5 score=0.004950493574142456

Same query with score_threshold=0.05:
  -> 0 result(s) returned
  Expected: the points scoring >= 0.05 (ids 2, 3, 4) should be kept.
  Actual:   0 results - the loop breaks on the very first (best) point.

The same mismatch reproduces for a Discover query on a Manhattan collection:

Discover query on a MANHATTAN collection, no score_threshold:
  id=4 score=1.2272727191448212
  id=5 score=1.1666666567325592

Same query with score_threshold=-100:
  -> 0 result(s) returned
  Expected: both points should trivially pass such a low threshold.
  Actual:   0 results.

As a control, the identical BEST_SCORE recommend query against a Cosine collection (already bigger-is-better, so required_order happens to match) behaves correctly:

Control: same BEST_SCORE recommend query on a COSINE collection,
score_threshold=0.5 (bigger-is-better metric, no mismatch possible):
  id=2 score=0.7492331266403198
  id=3 score=0.5497245788574219
  -> 2 result(s) returned, correctly non-empty.

Full standalone reproduction script (self-contained, only needs qdrant-client installed):

from qdrant_client import QdrantClient, models


def recommend_on_euclidean():
    client = QdrantClient(":memory:")
    client.create_collection(
        "recommend_euclid",
        vectors_config=models.VectorParams(size=2, distance=models.Distance.EUCLID),
    )
    client.upsert(
        "recommend_euclid",
        points=[
            models.PointStruct(id=1, vector=[0.0, 0.0]),
            models.PointStruct(id=2, vector=[1.0, 0.0]),
            models.PointStruct(id=3, vector=[2.0, 0.0]),
            models.PointStruct(id=4, vector=[3.0, 0.0]),
            models.PointStruct(id=5, vector=[10.0, 0.0]),
        ],
    )
    query = models.RecommendQuery(
        recommend=models.RecommendInput(
            positive=[1], strategy=models.RecommendStrategy.BEST_SCORE
        )
    )

    no_threshold = client.query_points("recommend_euclid", query=query, limit=10).points
    print("Recommend (BEST_SCORE) on a EUCLID collection, no score_threshold:")
    for p in no_threshold:
        print(f"  id={p.id} score={p.score}")

    threshold = 0.05  # permissive: 3 of the 4 candidates score >= 0.05
    filtered = client.query_points(
        "recommend_euclid", query=query, limit=10, score_threshold=threshold
    ).points
    print(f"\nSame query with score_threshold={threshold}:")
    print(f"  -> {len(filtered)} result(s) returned")


def discover_on_manhattan():
    client = QdrantClient(":memory:")
    client.create_collection(
        "discover_manhattan",
        vectors_config=models.VectorParams(size=2, distance=models.Distance.MANHATTAN),
    )
    client.upsert(
        "discover_manhattan",
        points=[
            models.PointStruct(id=1, vector=[0.0, 0.0]),
            models.PointStruct(id=2, vector=[1.0, 0.0]),
            models.PointStruct(id=3, vector=[5.0, 0.0]),
            models.PointStruct(id=4, vector=[1.2, 0.0]),
            models.PointStruct(id=5, vector=[2.0, 0.0]),
        ],
    )
    query = models.DiscoverQuery(
        discover=models.DiscoverInput(
            target=1, context=[models.ContextPair(positive=2, negative=3)]
        )
    )

    no_threshold = client.query_points("discover_manhattan", query=query, limit=10).points
    print("\nDiscover query on a MANHATTAN collection, no score_threshold:")
    for p in no_threshold:
        print(f"  id={p.id} score={p.score}")

    threshold = -100
    filtered = client.query_points(
        "discover_manhattan", query=query, limit=10, score_threshold=threshold
    ).points
    print(f"\nSame query with score_threshold={threshold}:")
    print(f"  -> {len(filtered)} result(s) returned")


if __name__ == "__main__":
    recommend_on_euclidean()
    discover_on_manhattan()

Expected behavior

For Recommend (best_score/sum_scores), Discover and Context queries, score_threshold should always filter out points whose synthetic score is below the threshold (bigger-is-better semantics), regardless of the collection's underlying distance metric - matching the same isinstance(...) override that already governs the sort direction a few lines above.

Actual behavior

On a Euclidean or Manhattan collection, score_threshold uses the comparison direction meant for raw distances instead, so the result-building loop breaks on the very first (best-scoring) candidate, and the query returns far fewer results than it should - typically zero, for any threshold that isn't extreme.

Environment

  • qdrant-client 1.17.1 (installed from PyPI)
  • Verified the exact code (qdrant_client/local/local_collection.py) is unchanged on current master by diffing against the file fetched fresh from https://raw.githubusercontent.com/qdrant/qdrant-client/master/qdrant_client/local/local_collection.py
  • Reproduced entirely in local/in-memory mode (QdrantClient(":memory:")), no server needed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions