Model/provider architecture, local inference lifecycle, and eval harness

[zeapsu]

The Great Sage has rewritten this quest for the current Sage roadmap. 🧙‍♂️

The original version of this issue was about setting up an ML model integration structure in FastAPI. Sage has evolved since then into a local-first desktop knowledge agent, so this issue should now track the model/provider architecture more deliberately.

Goal

Build a clean model integration layer for Sage that supports local-first usage, configurable providers, retrieval improvements, and measurable quality.

Scope

Provider architecture

• Keep provider access behind a small abstraction
• Support local-first defaults where possible
• Allow configured providers such as Ollama, OpenAI-compatible APIs, Anthropic, DeepSeek, etc.
• Make provider/model selection visible to the UI where useful

Local model lifecycle

• Avoid loading expensive models unnecessarily
• Make embedding model initialization predictable
• Prepare for future local rerankers or specialized models
• Keep desktop startup reliable

Retrieval quality

• Support stronger embedding models
• Consider hybrid search with dense retrieval plus keyword/BM25
• Add reranking as an optional second stage
• Preserve Tome-scoped retrieval as a first-class concept

Grounding and citations

• Improve source attribution from document-level to chunk-level or span-level where possible
• Make generated artifacts cite their source chunks
• Ensure reports, quizzes, flashcards, and audio remain grounded in selected sources

Evaluation harness

Add small repeatable evals so changes can be measured instead of judged by vibes:

• retrieval recall@k
• answer faithfulness
• citation correctness
• generated quiz answer validity
• report grounding quality

Non-goals for first pass

• Full ML platform infrastructure
• Cloud-only model serving
• Complex agent orchestration
• Heavy dependencies that make the local desktop app fragile

Acceptance criteria

• There is a documented provider/model abstraction
• Local model initialization is predictable and testable
• Retrieval behavior can be evaluated with a small fixture set
• Future model upgrades can be compared against baseline results

[zeapsu]

Great Sage update: I rewrote this issue to match the current Sage architecture. 🧙‍♂️

The old framing was from the earlier FastAPI/arXiv era. The new framing tracks provider architecture, local model lifecycle, retrieval quality, grounding/citations, and an eval harness for measuring model improvements.