Skip to content

fix: meaning search compares every vector, so a search is the same every time - #99

Merged
alkhatibdev merged 1 commit into
mainfrom
exact-meaning-search
Nov 14, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
exact-meaning-search

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

Meaning search compares every vector

Meaning search had an approximate index (HNSW) over the clause vectors, built by
dpolens index build. Postgres chose, query by query, between that index and comparing every
vector, from its statistics about the table. The two do not return the same clauses, so the
same question could come back with different results, and a score measured on it could not be
reproduced.

This pull request removes the index. A question is compared with every clause's vector, which
is exact and the same on every run.

What was measured

On main, with GDPR and the UAE law loaded (1,326 vectors) and nothing forced:

The table Postgres uses Meaning search's top five against an exact scan, 314 eval questions
One model, just embedded, not yet analysed every vector the same for all 314
One model, analysed every vector the same for all 314
A second model's vectors added, not yet analysed the index, for that model differ for 55, and the right clause leaves the top five for 13 and enters it for 1
The same, once Postgres analyses the table every vector the same for all 314

So with one model and two laws, searches were already exact, and the published scores do not
move. The index came into play while the table held vectors the statistics had not caught up
with, which is the state a second model is in when two are compared. An index is also what a
planner prefers as a table grows, so a larger corpus would have crossed to it. That was not
measured.

Comparing every vector takes about 4 ms for the 1,326 on a laptop.

What changes

  • vector_search no longer sets hnsw.ef_search and hnsw.iterative_scan, and build_index
    no longer creates an index. Its create_index argument is gone, since the worker has no
    index to leave alone.
  • Migration 0021 drops the approximate index of every model. Nothing is embedded again. On
    the way back, the earlier version's index build makes its index again at the next start.
  • dpolens index build writes rows and creates nothing, so it connects as the application
    role, like every other command, and maintenance_scope is removed.
    DPOLENS_MIGRATION_DATABASE_URL is read by migrations alone, as its description says.
  • A search kept to some laws returns every clause those laws hold, with no setting to make it
    so.
  • evals/retrieval.md says a question is compared with every clause's vector, and why.

Checked

On 9fe00ea with this change, with GITHUB_ACTIONS=true, CI=true and every cache off:

  • ruff check, ruff format --check, mypy, lint-imports, scripts/api_contract.py,
    scripts/client_text.py, docker compose config
  • pytest: 1439 passed. At the database: each score meaning search returns is the clause's
    own closeness to the question, and no clause is missing; embedding builds no index beside the
    table's key; a search kept to one law returns all of its clauses; 0021 drops an index an
    earlier version built and leaves the table's own; dpolens index build runs as the
    application role with no owner connection
  • The retrieval evals job, step for step: every published run at its published figure
  • The upgrade from main, as setup runs it: a database built by main, with its approximate
    index, then this branch's migration and start-up steps. The index is gone, nothing is
    embedded again, every published run is at its published figure, and all 314 eval questions
    return the same clauses before and after ANALYZE. Downgrading and starting main again
    rebuilds its index

…ery time

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit e2471e0 into main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant