Repository navigation
fix: meaning search compares every vector, so a search is the same every time - #99
Merged
Merged
Conversation
…ery time Signed-off-by: alkhatibdev <me@alkhatib.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Meaning search compares every vector
Meaning search had an approximate index (HNSW) over the clause vectors, built by
dpolens index build. Postgres chose, query by query, between that index and comparing everyvector, from its statistics about the table. The two do not return the same clauses, so the
same question could come back with different results, and a score measured on it could not be
reproduced.
This pull request removes the index. A question is compared with every clause's vector, which
is exact and the same on every run.
What was measured
On
main, with GDPR and the UAE law loaded (1,326 vectors) and nothing forced:So with one model and two laws, searches were already exact, and the published scores do not
move. The index came into play while the table held vectors the statistics had not caught up
with, which is the state a second model is in when two are compared. An index is also what a
planner prefers as a table grows, so a larger corpus would have crossed to it. That was not
measured.
Comparing every vector takes about 4 ms for the 1,326 on a laptop.
What changes
vector_searchno longer setshnsw.ef_searchandhnsw.iterative_scan, andbuild_indexno longer creates an index. Its
create_indexargument is gone, since the worker has noindex to leave alone.
0021drops the approximate index of every model. Nothing is embedded again. Onthe way back, the earlier version's
index buildmakes its index again at the next start.dpolens index buildwrites rows and creates nothing, so it connects as the applicationrole, like every other command, and
maintenance_scopeis removed.DPOLENS_MIGRATION_DATABASE_URLis read by migrations alone, as its description says.so.
evals/retrieval.mdsays a question is compared with every clause's vector, and why.Checked
On
9fe00eawith this change, withGITHUB_ACTIONS=true,CI=trueand every cache off:ruff check,ruff format --check,mypy,lint-imports,scripts/api_contract.py,scripts/client_text.py,docker compose configpytest: 1439 passed. At the database: each score meaning search returns is the clause'sown closeness to the question, and no clause is missing; embedding builds no index beside the
table's key; a search kept to one law returns all of its clauses;
0021drops an index anearlier version built and leaves the table's own;
dpolens index buildruns as theapplication role with no owner connection
main, as setup runs it: a database built bymain, with its approximateindex, then this branch's migration and start-up steps. The index is gone, nothing is
embedded again, every published run is at its published figure, and all 314 eval questions
return the same clauses before and after
ANALYZE. Downgrading and startingmainagainrebuilds its index