embedkit: geteilte Embedding-Mechanik aus symseek und symmemory extrahieren

[danieljustus]

Status: tracking issue

This issue no longer has direct implementation work — it was split into 4 right-sized sub-issues (see comment below). Closes automatically once all four are merged.

• embedkit: Result type and quantization-aware SameSpace guard #81 — Result type + quantization-aware SameSpace guard (merged)
• embedkit: provider abstraction over ollamakit with deterministic hash fallback #82 — Provider abstraction over ollamakit + deterministic hash fallback
• embedkit: configurable-size embedding cache #83 — Configurable-size embedding cache
• embedkit: dimension pinning with truncation and renormalization #84 — Dimension pinning with truncation and renormalization

Goal

New package embedkit with the embedding mechanics duplicated today across two repos: hash fallback, cache, dimension resolution, and provenance metadata.

Why now

Two near-identical implementations of the same concept, one of which bypasses ollamakit entirely:

• symaira-seek/internal/engine/embeddings.go — ollamakit.Embed, FNV-1a fallback (local-hash), own LRU (map+list), dimension autodetect, defaultEmbeddingDim = 768
• symaira-memory/internal/extractor/embeddings.go — raw net/http, FNV-1a fallback, hashicorp/golang-lru, DefaultDimensions = 768

EmbeddingDim = 768 is additionally hard-coded in symaira-memory/internal/db/lsh.go:15 and only truncates mismatched dimensions with a stderr warning. Every model change is a risk in two independent places today.

Current state / Evidence

• symaira-seek/internal/engine/embeddings.go (dimension autodetect, localHashModelName)
• symaira-memory/internal/extractor/embeddings.go (own http.Client, EmbeddingResult{Vector,Source,Model})
• symaira-memory/internal/db/lsh.go:15 — EmbeddingDim = 768
• Both repos document a mixed-embedding-space guard; provenance fields exist twice

Out of Scope

• Ranking, index, chunking (symseek)
• Memory scopes, consolidation (symmemory)
• Concrete model selection — separate issues in the consumer repos
• Migrating the consumers themselves (their own issues)

Risks / Dependencies

Migrating the consumers requires a reindex and is tracked separately in symaira-seek and symaira-memory. This issue (and its sub-issues) deliver only the library.

[danieljustus]

This issue bundles several independently-implementable pieces of work: a new embedkit package, provider abstraction, cache, deterministic hash fallback, dimension pinning with renormalization for Matryoshka-capable models, and a quantization-aware SameSpace guard. Correctly getting the space-identity guard right (matching model and quantization level) is the crux of the whole issue — getting it wrong would silently let mismatched embedding spaces compare as equal, which is a data-correctness risk in the consuming tools.

Given that scope, this is better handled as a dedicated, focused implementation pass (or split into a few smaller sub-issues) rather than folded into a larger batch session. Suggested breakdown for a follow-up:

1. Core Result{Vector, Source, Model, Dim, Quant} type + SameSpace guard (model + quantization identity), with tests covering the exact confusion case named in scope (4-bit MLX vs. unquantized Ollama).
2. Provider abstraction wrapping ollamakit.Client as the default provider, plus deterministic hash fallback (mirroring the FNV-1a approach in symaira-seek/internal/engine/embeddings.go and symaira-memory/internal/extractor/embeddings.go).
3. Cache with configurable size (replacing the two independent LRU implementations).
4. Dimension pinning: truncation + renormalization for Matryoshka-capable models, with tests.

No code change made in this pass — leaving the issue open and unclaimed so it can be planned into the right-sized units.

[danieljustus]

Split into 4 right-sized sub-issues per the breakdown above:

1. embedkit: Result type and quantization-aware SameSpace guard #81 — Result type + quantization-aware SameSpace guard (foundation, lands first)
2. embedkit: provider abstraction over ollamakit with deterministic hash fallback #82 — Provider abstraction over ollamakit + deterministic hash fallback
3. embedkit: configurable-size embedding cache #83 — Configurable-size embedding cache
4. embedkit: dimension pinning with truncation and renormalization #84 — Dimension pinning with truncation and renormalization

All four carry group: local-models and milestone v0.8.1, same as this issue. This issue now tracks the set; it closes once all four are merged. #81 should land first since #82–#84 all consume its Result type.