fix: resolve each index's own embedding model (serve queries, CLI index/stats/status)

[donbowman]

This PR fixes a family of bugs with one root cause: a code path using ModelType::default() (384-dim MiniLM) — or a hardcoded 384 — instead of resolving the embedding model/dimensions each index records in its metadata.json. The indexing side already followed the per-index contract; the query, CLI, status, and served-add paths did not.

1. Serve queries embedded with the default model

Symptom. On a codesearch serve hub whose indexes were rebuilt with embeddinggemma-q4 (768-dim), every semantic query failed with Query embedding dimension mismatch: expected 768, got 384. status(kind=index) reported the hub model as minilm-l6-q, and --model on serve did nothing.

Cause. CodesearchService::new_for_serve hardcoded model_type: ModelType::default(); its shared embedding_service was built from that; run_serve(...) has no model parameter, so --model was silently inert. The single-repo stdio path already resolved the model from metadata.json (issue #118) — serve never did.

Fix. Resolve the query model per routed repo, with one embedder per model:

• ModelType::from_index_metadata(db_path) — reader counterpart to write_metadata_fields.
• EmbeddingServicePool — lazily loads one EmbeddingService per model, shared across MCP sessions and REST handlers.
• ServeState::model_for_alias() + embedding_pool(); CodesearchService::query_model(alias) / embedding_service_for(model).
• semantic_search resolves the model from the routed alias; semantic_search_multi embeds once per distinct model and searches each store with its own embedding (with_vector_store_read_multi now passes the alias to its closure).
• CLI warns that serve --model is inert; README documents per-repo resolution.

2. POST /repos with --model created the store at the wrong dimension

Symptom. codesearch index add --model embeddinggemma-q4 <path> (which delegates to POST /repos when serve is running): the .codesearch.db directory is created, stats reports the new dimension, but no files are indexed.

Cause. add_repo_handler opened the store inline via try_open_stores, which read the dimension from metadata.json (absent → 384 default), and parsed body.model only afterwards — applying the override to metadata but never to the store. The background reindex then embedded 768-dim vectors into a 384-dim store and indexed nothing.

Fix. try_open_stores takes an optional dimension override; add_repo_handler parses the model first and passes model.dimensions() for a fresh/being-rebuilt store. All other call sites pass None and keep reading metadata.json.

3. CLI index downgraded non-default indexes

Symptom. codesearch index <path> on a repo indexed with embeddinggemma-q4 embedded against it with the 384-dim default (FileMetaStore logged "Model changed, full re-index required" and wiped its metadata); codesearch index add on an already-registered repo did nothing (it only registers).

Cause. index_with_options did let model_type = model.unwrap_or_default(); and never called resolve_embed_model(db_path) — the helper the serve/watcher paths use whose own doc says "Every embedding path resolves the model through this helper instead of ModelType::default()".

Fix. New resolve_index_model(db_path, force, requested): on an existing index the recorded model wins; an explicit --model that disagrees is rejected with a pointer to --force. resolve_embed_model is now pub(crate) so the CLI reuses it.

4. stats hardcoded 384 dims

codesearch stats, get_db_stats and the repo listing opened the vector store as VectorStore::new(&db_path, 384), so Dimensions: always read 384 and the store was opened with the wrong dimension. They now read the recorded dimensions via recorded_dimensions(db_path); the delete-old-chunks path uses model_type.dimensions().

5. status(kind="index") reported the service default

The model field was self.model_type (the hardcoded default in serve mode) while dimensions came from live store stats, so every repo read minilm-l6-q. It now resolves per repo for project=, and reports the common model — or mixed — for a group (group_model_label).

6. status reported ready for an index that cannot be searched

Symptom. On a 27-repo hub partway through a model migration, status(kind="index", project="py-opa") reported indexed: false but status: "ready" / "Index is ready for searching." — while search(project="py-opa") failed with Index not built. Call build_index() after inserting chunks.

Cause. Readiness keyed only off total_chunks > 0; the HNSW graph state (stats.indexed) was ignored. Chunks-inserted-but-graph-not-built is exactly the mid-rebuild window an operator watches.

Fix. Single-store path uses a new single_index_status(total_chunks, indexed); the group path's index_status_summary takes all_indexed. A store that failed to report stats is tracked separately, so a failure is never misread as "not built" (it still surfaces as degraded-ready with warnings).

Tests

• Serve: model_for_alias_reads_the_index_metadata_model, ..._is_none_without_index_metadata, ..._is_per_repo_not_hub_wide, serve_service_uses_repo_model_not_default, group_status_model_label_is_common_or_mixed, try_open_stores_honours_dimension_override_for_a_fresh_repo.
• CLI/index: index::index_model_resolution_tests::{recorded_dimensions_reads_metadata_dimensions, recorded_dimensions_falls_back_to_model_then_default, resolve_index_model_prefers_recorded_model_on_existing_index, resolve_index_model_rejects_disagreeing_override_without_force, resolve_index_model_uses_requested_model_for_a_fresh_index}.
• Embed: test_model_type_round_trips_through_index_metadata.
• Status: index_status_summary_reports_building_when_chunks_exist_but_graph_is_not_built, single_index_status_requires_a_built_graph_to_report_ready.

Proof rule. Reintroducing each defect and watching the relevant test fail before reverting:

• query_model returning self.model_type → serve_service_uses_repo_model_not_default fails (left: AllMiniLML6V2Q, right: EmbeddingGemma300MQ4).
• resolve_index_model ignoring the recorded model → resolve_index_model_prefers_recorded_model_on_existing_index and ..._rejects_disagreeing_override_without_force fail.
• try_open_stores ignoring the dimension override → try_open_stores_honours_dimension_override_for_a_fresh_repo fails (left: 384, right: 768).
• Readiness ignoring the built-graph signal → the two status tests fail (left: ready, right: building).

Validation

cargo check --all-targets, cargo clippy --all-targets -- -D warnings, cargo test --lib --bins. 702 passed, 13 ignored. The single failure, watch::tests::test_no_ignore_files_returns_none, is unrelated and pre-existing on this host (also fails on unmodified develop). The pre-push QC hook could not run here because pwsh is not installed; the equivalent cargo fmt + clippy + test gate was run manually.

[flupkede]

A big thank-you for this PR, Don — truly excellent work. 🙇

One root cause (the per-index model contract in metadata.json) traced through six symptoms, each fixed with a regression test that fails on reintroduction, with README and CHANGELOG kept honest along the way. We walked the whole thing through a full human audit today: every criterion the PR states is implemented and pinned — 4 of 5 items cleared as ok.

Two notes from the walkthrough, neither merge-blocking:

1. Needs evaluation — the new serve --model default vs. legacy unrecorded indexes. When a repo's metadata.json records no model (an index built before the recording contract existed), the query fallback now follows the serve default. Concretely: setting serve --model embeddinggemma-q4 on a hub holding such a legacy repo makes every search on that repo fail with expected 384, got 768 — while it worked before the flag was set. And with a same-dimension model the failure is silent: search "works", rankings quietly degrade across incomparable vector spaces. What would settle it: a one-line fallback-time warning (repo, assumed model, re-index command) — or an explicit decision that unrecorded indexes are unsupported.

2. Deferred by design (recorded, revisit at the next fastembed bump): the first cold-model load holds the pool's map mutex and briefly stalls all queries — the deliberate trade against doubling the ~2GB ONNX arena.

Also noted, pre-existing: index <path> --force without --model still rebuilds at the built-in default, which can silently downgrade a non-default index on that path. Good candidate for a follow-up.

And finally: an invitation to join the repo as a collaborator is on its way — work of this quality is very welcome here. Thanks again! 🚀

[donbowman]

A big thank-you for this PR, Don — truly excellent work. 🙇

One root cause (the per-index model contract in metadata.json) traced through six symptoms, each fixed with a regression test that fails on reintroduction, with README and CHANGELOG kept honest along the way. We walked the whole thing through a full human audit today: every criterion the PR states is implemented and pinned — 4 of 5 items cleared as ok.

Two notes from the walkthrough, neither merge-blocking:

1. Needs evaluation — the new serve --model default vs. legacy unrecorded indexes. When a repo's metadata.json records no model (an index built before the recording contract existed), the query fallback now follows the serve default. Concretely: setting serve --model embeddinggemma-q4 on a hub holding such a legacy repo makes every search on that repo fail with expected 384, got 768 — while it worked before the flag was set. And with a same-dimension model the failure is silent: search "works", rankings quietly degrade across incomparable vector spaces. What would settle it: a one-line fallback-time warning (repo, assumed model, re-index command) — or an explicit decision that unrecorded indexes are unsupported.

2. Deferred by design (recorded, revisit at the next fastembed bump): the first cold-model load holds the pool's map mutex and briefly stalls all queries — the deliberate trade against doubling the ~2GB ONNX arena.

Also noted, pre-existing: index <path> --force without --model still rebuilds at the built-in default, which can silently downgrade a non-default index on that path. Good candidate for a follow-up.

And finally: an invitation to join the repo as a collaborator is on its way — work of this quality is very welcome here. Thanks again! 🚀

OK I will take a look at the serve model when there is some index already.

[donbowman]

Thanks for the careful read, and for walking the whole thing through the audit.

We addressed the first note in 654fd6d, "fix(serve): query unrecorded indexes with the built-in model".

You were right about the fallback. When a repo's metadata.json records no model, the model it was built with is unknowable, and we were resolving that case to the serve default. That turned serve --model into an override of an existing index, which contradicts what the flag help and the CLI comment both promise. It failed loudly on a dimension change, and quietly on a same-dimension change.

What we did:

• An unrecorded index is now queried with the built-in 384-dim default. That is the historical behaviour, and it is the value every other reader already assumes for metadata without a model_short_name (the CLI index path, the incremental refresh, resolve_embed_model). serve --model keeps doing what its name says: it selects the model for newly created indexes, and it still backs the scope-free status summary.
• The assumption now reaches the caller: each search response carries a warnings entry naming the repo, the assumed model and the re-index command.
• The server logs the same line once per repo, so a busy hub does not repeat it on every query.
• Tests: unrecorded_index_is_queried_with_builtin_default_not_serve_default and legacy_model_warning_is_logged_once_per_alias. We reintroduced the defect, watched the first test fail, then reverted it.

The second note, where the first cold-model load holds the pool's map mutex, is deferred by design. We have not touched it here. We agree it belongs with the next fastembed bump.

The pre-existing index <path> --force downgrade is also untouched, and we agree it is a good follow-up.

[flupkede]

Audited end-to-end (5 items walked, every criterion pinned by a regression test) and validated on the final head: fmt clean, clippy -D warnings clean, 716+712 tests green. The needs-evaluation point is settled by 654fd6d - the serve default no longer leaks into the query fallback for unrecorded indexes, and the caller-facing warning plus once-per-repo log are exactly right. Excellent work - thank you!