Keep Buddy's action and mentor requests out of the FAQ - #210
Merged
Merged
Conversation
Every question in a project-scoped Buddy conversation will feed the FAQ
once the chat is retired (Wiki#319, backend#214), in both capability
modes, and only the AI classifier filters. Chat only ever received
documentation questions, so both FAQ prompts described their input as
questions "asked to a docs chatbot" and discarded nothing but greetings,
smalltalk and thanks. Buddy is also a mentor: it is asked to act ("move
my card to done", "remind my PM about the review"), to report on the
asker ("what should I work on next?", "summarise my onboarding path")
and to continue a conversation ("and the second one?"). Under the old
rules each of those would have become an FAQ entry for the PM.
Both prompts now discard four kinds of text: smalltalk (as before),
requests to do something, questions about the asker's own onboarding
state, and follow-ups that only make sense inside their conversation.
The rule closes with the test that keeps a personal phrasing from being
mistaken for a personal question: it stays a documentation question when
the same documentation would answer anyone who asks it, so "how do I get
VPN access?" still counts even though it says "I".
The categories live in one constant, insights.faq.NON_FAQ_KINDS, that
the live classifier (classify) and the full rebuild (group) both render.
Two hand-kept copies would drift, and a rebuild that disagreed with the
classifier would resurrect entries the live path had dropped, or drop
entries it had kept, the next time a PM pressed refresh.
"Documentation chatbot" / "docs chatbot" become "the project's
assistant" in both prompts. The FaqClassifyRequest.question and
FaqClassifyResponse.relevant descriptions, the classify route
description, the module docstrings and docs/faq-grouping-concept.md say
the same. No schema change: the backend keeps sending question text
only, and strips quoted selections before sending (backend#214).
Tests pin what a stub can prove. With a scripted model the verdict is
whatever the script says, so asserting relevant=false for "move my card"
would only test parsing. Instead the tests capture the system prompt
each entry point actually sends and assert every category and example
is in it, that the chatbot wording is gone, and that the classify prompt
contains the shared constant verbatim. The parsing side is covered too:
an irrelevant verdict yields no group, no documents and no second
redaction call, and group drops discarded Buddy requests while keeping
the mentor-toned documentation question. The capturing stubs override
generate with its real signature, so they need no type: ignore.
Verification: ruff format --check, ruff check and pyright src/ (0
errors) clean; pytest 980 passed, 8 skipped (963 before, +17 new).
Reverting the two insights modules fails the new tests at collection.
Closes #209
Closed
4 tasks
Afif-del
approved these changes
Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of SprintStartProject/Wiki#319 (chat → Buddy migration). Closes #209. Unblocks the FAQ half of SprintStartProject/sprintstart-backend#214.
Once
/chatis retired, every question in a project-scoped Buddy conversation feeds the FAQ, in both capability modes, and only this classifier filters (Wiki#319, decision 12). Chat only ever got documentation questions; Buddy is also a mentor, so it also gets requests to act, questions about the asker's own onboarding, and context-only follow-ups. The old prompts discarded only greetings/smalltalk/thanks, so all of those would have become FAQ entries for the PM.Changes
insights.faq.NON_FAQ_KINDS, rendered verbatim by both_CLASSIFY_SYSTEM(live, per question) and_GROUPING_SYSTEM(full rebuild). If the two prompts drifted, a PM's refresh would bring back entries the live path had dropped. Categories:FaqClassifyRequest.question,FaqClassifyResponse.relevant, the/insights/faq/classifyroute description, module docstrings, anddocs/faq-grouping-concept.md(new "classifier decides" bullet).Tests (+17)
A scripted model returns whatever the script says, so asserting
relevant=falsefor "move my card" only tests parsing. So the tests check both sides:NON_FAQ_KINDSverbatim (shared-list guarantee).group_faqsdrops the discarded Buddy requests and keeps the mentor-toned doc question.generatewith its real signature, so no newtype: ignore.Verification
uv run ruff format --check .✅uv run ruff check .✅uv run python -m pyright src/: 0 errors ✅uv run python -m pytest: 980 passed, 8 skipped (dev baseline 963 passed) ✅insightsmodules fails the new tests.Not covered: real-model classification quality. That needs a live LLM (an
llm_requiredtest would be skipped in CI), so it's worth a manual spot check with the examples above once backend#214 feeds Buddy questions.Base
Branched from
dev. It doesn't overlap with #208 (Buddy tutor), which touches noinsights/files, so it can merge in either order.