refactor: sage app foundations with real chat, KB UI, generation endpoints, audio, history#37
Closed
spencerwozniak wants to merge 11 commits into
Closed
refactor: sage app foundations with real chat, KB UI, generation endpoints, audio, history#37spencerwozniak wants to merge 11 commits into
spencerwozniak wants to merge 11 commits into
Conversation
Provides a pip-installable dependency list alongside the uv-managed pyproject.toml for environments that don't use uv. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds POST /api/generate/{flashcards,quiz,report} on a new router. Each
endpoint pulls excerpts from the knowledge base (semantic search when a
topic is provided, random sample otherwise — optionally scoped to a
tome), prompts the configured LLM with response_format=json_object on
OpenAI-compatible providers, and validates the parsed payload before
responding. An empty KB returns 400 with a hint to upload documents.
Also exposes an llm_text(...) helper used by the audio endpoint for
plain-text completions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/generate/audio retrieves KB context (or randomly samples it),
prompts the configured provider for a spoken-style narration script,
and segments it into sentence-level chunks with word-rate-estimated
timings for transcript scroll-lock.
When OPENAI_API_KEY is configured, the script is also synthesized to
MP3 via OpenAI's tts-1 voice (default: alloy) and cached to disk; the
response includes audio_url. Without a key, audio_url is null and the
frontend falls back to the browser's SpeechSynthesis voice.
GET /api/audio/{audio_id}.mp3 streams the cached file with a regex
guard against directory traversal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Adds KnowledgeStore.list_sessions(): one SQL pass joining sessions, messages, and tomes that returns derived title (first user message), tome name, message count, and last activity timestamp. Ordered by most-recent activity. - Adds GET /api/chat/sessions?limit=<n> as a thin wrapper for the History panel. - Extends the chat SYSTEM_PROMPT with an "App capabilities you should know about" section so the model can describe its own routed features (quiz / flashcards / audio / report / history / tomes / knowledge base / upload) accurately when the user asks "what can you do?", and skips replying when the frontend has already routed away. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds typed helpers covering every new backend surface: - Knowledge base: listDocuments, getDocument, deleteDocument plus DocumentSummary / DocumentDetail. - Generation: generateFlashcards, generateQuiz, generateReport, generateAudio with shared GenerateOpts and GeneratedSource. - Audio: AudioGeneration / AudioSegment plus resolveAudioUrl() that joins a relative audio path against the active API base (so it works from both the Next.js dev server and the Tauri sidecar). - History: listChatSessions and ChatSessionSummary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two new components reachable from the round button next to the command-bar pill: - UploadModal: paste / drag-drop / file-picker ingest of text-like files (.txt, .md, .markdown, .rst, .csv, .json, .log). Backend dedupes by content hash so re-uploads are safe. - KnowledgeBaseWidget: two-pane list+detail browser over every ingested document. Lists titles with doc_type/source/timestamp, shows chunk count, content hash, linked tomes, and a 200-char preview. Per-row delete. An "Add document" button in the header opens UploadModal and refreshes the list on close. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the hard-coded SAMPLE_* placeholders that the original FlashcardWidget / QuizWidget / AudioPlayerWidget / ReportViewWidget rendered with real, KB-grounded views that POST to the new /api/generate/* endpoints. Each view shares the same shape: loading card, error card with retry, content card, and a "Grounded in: ..." source footer listing the distinct source documents. AudioPlayerWidget gains two new props — audioUrl and script — and switches between two playback modes: - <audio> element when the backend synthesized an MP3 (OpenAI TTS). Timeupdate events drive currentTime; a Download link appears. - SpeechSynthesis fallback when no audio_url. Seek bar restarts speech from the nearest segment; a "Browser TTS" badge appears in place of Download. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- ChatWidget now POSTs to /api/chat via sendChat(), tracks sessionId across turns, surfaces backend errors inline as assistant-style messages, and accepts an initialQuery prop so the first command-bar prompt seeds the conversation. - The verb router used to live only in the command bar. detectView() is now lifted to page.tsx and shared with ChatWidget through an onCommand callback, so typing "make flashcards on attention" from inside the chat view routes correctly instead of being answered as prose. Supported verbs: quiz / test / flashcards / audio / podcast / listen / report / study guide / summary / history / tome / collection / library / knowledge / kb / docs. - CommandBar gets a circular knowledge-base button (icon: library_add) to the right of the pill that opens the KB browser; the model badge is now driven by getRuntimeConfig() instead of being hard-coded. - page.tsx threads generationPrompt through the routed views so the user's full command text becomes the topic for /api/generate/*. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the SAMPLE_HISTORY array with a fetch against
GET /api/chat/sessions. Filters out stub sessions (no user message
or zero messages), derives a row title from the first user message
(first line, 90 char cap), and groups rows into Today / Yesterday /
Earlier this week / Older buckets using a shared formatTimestamp
helper that handles both `datetime('now')` and ISO timestamp formats.
Search input filters by title or tome name client-side. Loading,
error, and empty states are inline. The filter pills (chat / quiz /
flashcard / etc.) are removed for now — only chat rows exist until
the other generations gain server-side persistence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Installs the agentation npm package as a devDependency and mounts a small <AgentationDev /> wrapper at the end of <body> in the root layout. The wrapper only renders when NODE_ENV === "development", so the overlay never ships in a production build. The optional MCP server (which forwards feedback into a CLI agent session) is set up out-of-tree via `npx agentation-mcp init` — no config is committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One markdown file per feature area introduced on this branch:
- README.md — index
- agentation.md — dev-only feedback overlay
- chat-backend-integration.md — ChatWidget ↔ /api/chat
- knowledge-base-ui.md — upload modal + KB browser
- in-chat-command-routing.md — shared detectView() and the
capability section of the chat system prompt
- generation-endpoints.md — /api/generate/{flashcards,quiz}
- audio-generation.md — /api/generate/audio + TTS playback
- history-panel.md — live session list
Each doc has a "Files touched" section so reviewers can map features
to diffs quickly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
spencerwozniak
changed the title
Owner
|
Great Sage review has entered the chat. 🧙♂️ First off, thank you for the thoughtful work here. The direction is very much aligned with where Sage is going: local-first knowledge base, uploaded sources, grounded generated artifacts, chat/history, and richer study views. I also tested the branch locally in an isolated checkout: Backend: Frontend: So this is not a "this does not work" review. It is more that the branch is too large and overlaps with recent work on The current
This PR also adds useful new pieces that are not currently on main, including:
Those are worth considering, but I do not think we should merge this whole branch as one PR because it mixes several architectural and product decisions at once. Specific things I would like to avoid merging as-is:
I also hit a local existing-DB issue while testing related branches:
That suggests we should add a small migration/schema-hardening step before relying on newer fields against existing local user databases. My recommendation: let’s split or cherry-pick this branch into smaller PRs. Suggested chunks:
There is good work here. I just want to preserve the current shape of Sage and bring the best pieces in deliberately instead of merging one large parallel implementation. |
This was referenced May 22, 2026
Contributor
Author
|
Closing in favor of a 4-PR split:
Why this split and not a finer oneA finer chunking was considered along these lines:
Rejected because it doesn't match the actual diff on this branch:
The 4-PR split keeps each PR self-contained, lets the backend / chores / docs merge in any order, and only one PR (#43, frontend) carries the full feature payload — which is unavoidable given how tightly the frontend commits coordinate on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch replaces every placeholder fixture in the Sage app with a real backend round-trip and adds the foundational endpoints, UI surfaces, and developer tooling that the rest of the product can build on. The original
SAMPLE_*arrays for flashcards, quizzes, chat, history, and audio are gone — every view now fetches live, KB-grounded data from the FastAPI backend.There is one feature doc per area under
docs/, and each commit message corresponds to one of those docs.What changed
Backend (FastAPI)
POST /api/generate/{flashcards,quiz,report}. Each retrieves excerpts from the user's KB (semantic search when a topic is provided, random sample otherwise — optionally scoped to a tome), calls the configured LLM withresponse_format=json_objecton OpenAI-compatible providers, and validates the parsed payload before responding. Empty KB →400with a hint to upload documents.POST /api/generate/audioreturns a spoken-style narration script with sentence-level segments and word-rate-estimated timings. WhenOPENAI_API_KEYis set, it also synthesizes an MP3 viatts-1and caches it;GET /api/audio/{id}.mp3streams it back with a regex guard against directory traversal.GET /api/chat/sessionsbacked by a newKnowledgeStore.list_sessions()that joinssessions × messages × tomesin one SQL pass and returns derived titles, tome name, message count, and last-activity timestamp.SYSTEM_PROMPTnow lists every routing keyword and the view it opens, so the model can describe its own routed features when asked "what can you do?" and stays silent when the frontend has already routed away.Frontend (Next.js / Tauri)
UploadModal(paste / drag-drop / file-picker for text-like files, deduped server-side by content hash) andKnowledgeBaseWidget(two-pane list + detail browser with per-row delete). Reachable via a new circular button next to the command-bar pill and via the verb router (knowledge,kb,docs, …).FlashcardsView,QuizView,AudioView,ReportVieware all real fetch wrappers around the matching/api/generate/*endpoint. Each renders loading / error-with-retry / content states and a "Grounded in: …" source footer.AudioPlayerWidgetnow supports two playback modes: a hidden<audio>element when the backend returned an MP3 (with a Download link), and aSpeechSynthesisfallback when no key is configured (with a "Browser TTS" badge). Transcript scroll-lock works in both modes.ChatWidgetcallssendChat(), trackssessionIdacross turns, surfaces backend errors inline, and acceptsinitialQueryso the user's first command-bar prompt seeds the conversation.detectView()is lifted intopage.tsxand exposed toChatWidgetviaonCommand, so typing "make flashcards on attention" from inside the chat view routes to the flashcards view instead of being answered as prose. The model badge inCommandBarandChatWidgetis driven bygetRuntimeConfig()instead of being hard-coded.HistoryPanelnow readsGET /api/chat/sessions, filters out empty stub sessions, derives titles from the first user message, and groups rows into Today / Yesterday / Earlier this week / Older.src/lib/sage-api.tsgets typed helpers for every new endpoint (knowledge, generation, audio, history) plus aresolveAudioUrl()that works from both the Next.js dev server and the Tauri sidecar.Tooling
agentationadded as a devDependency; a small<AgentationDev />wrapper mounted at the end of<body>renders only whenNODE_ENV === "development", so the in-app feedback overlay never ships in a production build.Docs
Per-feature reference notes under
docs/— one file per feature area, each with a "Files touched" section so reviewers can map features to diffs quickly:README.md(index)agentation.mdchat-backend-integration.mdknowledge-base-ui.mdin-chat-command-routing.mdgeneration-endpoints.mdaudio-generation.mdhistory-panel.mdTest plan
cd backend && uvicorn main:appboots without import errors.GET /returns OK;GET /api/configreturns provider/model..mdfile via the round button → it appears inKnowledgeBaseWidgetwith the expected chunk count and content hash.flashcards on attentionin the command bar with a non-empty KB → real cards appear with a "Grounded in: …" footer; empty KB → an error card pointing at upload.quiz on transformers,report on attention,audio on backpropagation.OPENAI_API_KEYset, the audio view streams an MP3 and exposes a Download link; without one, the SpeechSynthesis fallback runs and the badge reads "Browser TTS".flashcardsinside the chat view → it routes to the flashcards view, no chat reply is generated.HistoryPanelshows real conversations, grouped by Today / Yesterday / Earlier this week / Older; search filters by title and tome name.npm run buildinfrontend/succeeds;npx tsc --noEmitis clean.Known leftovers / explicitly out of scope
frontend/src-tauri/tauri.conf.json(removal ofidentifier) is not included — it would break Tauri builds, so it's kept in the working tree for the author to triage.HistoryItem["type"]union and ICON map are pre-shaped to extend.multipart/form-dataendpoint that calls the existingPDFService.extract_textand forwards into the ingest pipeline.Where to go next
All the groundwork is in place to take Sage from "real, working scaffold" to a genuinely exceptional product. The two highest-leverage follow-ups:
1. Deploy on Vercel
Make an account on https://vercel.com/ and connect it to this repository so that you can host the web app publicly. (This is free!)
2. Make Tomes a real, first-class feature
The schema, store methods, and tome-aware retrieval already exist (
create_tome,link_to_tome,get_tome_documents,get_tome_document_ids,tome_idonSearchDocsSkill, etc.), and every/api/generate/*endpoint already accepts atome_id. What's missing is the application layer on top:CommandBar,ChatWidget,UploadModal, and all*Viewcomponents, so uploads and generations are scoped by default instead of polluting the global KB.TomeSelectorwith create / rename / delete / set-active, drag-to-link documents from the KB browser, and an "active tome" pill in the header.list_sessionsquery already filters bytome_id), and one-click "make a quiz / flashcards / report from this tome".HistoryItem["type"]will start carrying real diversity.3. Begin integrating more advanced ML models
The provider abstraction (
providers/factory.py,providers/base.py) and the skill registry (skills/base.py,SearchDocsSkill) are deliberately model-agnostic. That makes the next ML upgrades incremental, not architectural:text-embedding-3-largeorbge-large-en-v1.5), add a re-ranker (bge-rerankerorcohere/rerank) betweenSearchDocsSkilland the LLM, and add hybrid BM25 + dense scoring on top of the existing chunk table./api/generate/*endpoint already passeschunk_indexthrough to the response; the next step is to have the LLM emit[doc:chunk]markers and render real footnotes inFlashcardsView/QuizView/ReportView.PDFService.extract_text(already used by the arXiv flow) into amultipart/form-dataendpoint, then layer in OCR (Tesseract or a vision LLM) and table-aware parsers (unstructured,pdfplumber).SkillRegistryis the right place to addweb_fetch,arxiv_search,wolfram_alpha, etc. The chat orchestrator already supports tool-calls; new skills are additive.KnowledgeStore, it's now cheap to wire a small eval set per skill (retrieval recall@k, quiz answer accuracy, report faithfulness) so each model upgrade can be measured rather than vibed.