Skip to content

Speed up cosine scoring without changing results #23

Description

@himanshu231204

Problem

_cosine() is a pure-Python loop over 384 floats, called once per stored memory per search. It also recomputes the norm of every stored vector on every query, even though those norms never change.

This may be the entire scaling problem — and fixing it is far simpler than adding a vector index (#22).

Ideas, roughly in order of effort

  1. Normalise vectors at write time. If stored vectors are unit length, cosine similarity is just a dot product. This removes a square root and a full pass per row, per query, and costs one column or a convention.
  2. Vectorise the scan. Load embeddings into a single array and score with one matrix multiply rather than N Python loops.
  3. Skip the round trip. Unpacking every blob into a Python list allocates heavily; operating on buffers directly avoids it.

Idea 1 is likely the best ratio of gain to complexity, and it's backward-compatible if existing vectors are normalised on read.

Acceptance criteria

  • Benchmarks from Build a benchmark harness for store and search at 1k / 10k / 100k memories #21 before and after, posted on this issue
  • Ranking is unchanged — same corpus and query produce the same order. Existing tests should pass untouched; add one that asserts scores match the current implementation within a small tolerance
  • Existing databases keep working. If vectors become normalised at write time, old un-normalised rows must still score correctly
  • No new required dependency. If numpy is used, it must be optional with a pure-Python fallback — this project's install weight is a feature

Why this matters more than it looks

If this makes a 100k-row scan fast enough, #22 can be closed and the project keeps its exact search, its single-file property, and its dependency-light install. That's a considerably better outcome than adding an index.

Pointers

  • src/localmem_mcp/store.py_cosine(), _pack(), _unpack(), and the loop in search()

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededperformanceSpeed, memory use, scalingv0.3: scaleBenchmarks, vector index, batch embedding

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions