CiteDesk is a citation-first RAG assistant for BrightCare Medical Billing Office, a fictional healthcare billing/admin team. It helps staff answer internal billing policy questions using approved knowledge-base sources, and refuses to answer when the evidence is missing or insufficient.
Core rule: No source, no answer.
Healthcare billing/admin teams often need fast, consistent guidance before replying to a patient, updating a billing case, or escalating a concern. A generic chatbot can sound confident even when it is wrong, which is risky for policy-based work.
CiteDesk was designed around one core decision: the system should not answer unless approved internal policy evidence supports the response. When evidence is missing, weak, or unrelated, CiteDesk returns a not-found/manual-review response instead of guessing.
This project demonstrates how to build an AI workflow with controls around:
- Source-grounded answering
- Mandatory citations
- Evidence-quality checks
- Refusal behavior
- QA logging
- Manual verification
- Failure-type tracking
This pattern applies anywhere a team needs consistent, sourced answers from internal documents.
CiteDesk is an internal policy assistant for healthcare billing/admin staff. It receives a staff question, retrieves relevant approved policy chunks, checks whether the evidence is strong enough to answer, and then either returns a cited response or refuses with a manual-review message.
The project uses fictional healthcare billing/admin data only. It is not medical, legal, financial, or patient-specific advice.
CiteDesk v1 uses LLM-assisted keyword extraction + keyword-filtered retrieval instead of vector search.
This was an intentional decision. The knowledge base is small, structured, and highly inspectable, so keyword retrieval is easier to debug, easier to explain, and easier to validate in a portfolio setting. Vector search is listed as a planned v2 upgrade once test results show real retrieval failures that justify the extra complexity.
The workflow does not send retrieved chunks directly into the final answer step. It first uses a separate LLM evidence-reranking step to decide:
- Which chunks actually support the question
- Whether the evidence is strong, weak, or missing
- Whether CiteDesk should answer or refuse
This separation makes the workflow easier to audit because retrieval quality and answer quality can be reviewed independently.
CiteDesk does not treat all refusals the same. It logs two distinct failure types:
| Failure Type | Meaning |
|---|---|
no_candidates_retrieved |
No approved policy chunk matched retrieval |
insufficient_evidence |
Related chunks were retrieved, but the evidence was too weak to support an answer |
This distinction matters because it shows whether the issue is a retrieval gap or an evidence-quality gap.
The frontend is designed like a real internal BrightCare staff tool, while the Google Sheets log keeps the QA details visible for review. This keeps the user experience clean while preserving auditability behind the scenes.
Example staff questions:
- When should a billing dispute be escalated?
- Can we send a receipt for a partial payment?
- Can we resend an invoice if the patient asks for another copy?
- What should staff do if a patient says they paid but the payment is missing?
- What should staff do if a patient requests a refund?
- What should staff do if a patient asks for a corrected invoice?
Example refusal cases:
- Can we charge a late payment penalty?
- Can staff offer a 10% discount for patients who pay today?
- Can we waive a patient’s remaining balance as a courtesy?
| Tool | Purpose |
|---|---|
| n8n | Main workflow automation engine |
| Supabase | Stores approved knowledge-base chunks |
| OpenAI | Keyword extraction, evidence reranking, and cited answer generation |
| Google Sheets | QA logs, manual review, and metrics tracking |
| HTML/CSS/JavaScript | Staff-facing frontend interface |
| ngrok | Local webhook testing for Docker-based n8n |
CiteDesk uses a retrieval-and-verification workflow instead of a generic chatbot flow.
Frontend staff question
↓
n8n Webhook
↓
LLM extracts retrieval keyword
↓
Supabase retrieves approved knowledge chunks
↓
Code node filters candidate chunks
↓
IF: candidates found?
├── No → Not Found response + QA log
└── Yes → LLM reranks evidence
↓
IF: answerable?
├── Yes → Generate cited answer + QA log + frontend response
└── No → Insufficient evidence response + QA log
CiteDesk has three major output paths:
Used when the retrieved policy chunks strongly answer the staff question.
Output:
- Staff-facing answer
- Source cards with document, section, and chunk ID
grounding_status = Groundedfailure_type = none
Used when no approved knowledge-base chunk matches the question.
Output:
- Manual review message
- No citations shown
grounding_status = Not Foundfailure_type = no_candidates_retrieved
Used when related chunks are found, but they do not fully support a reliable answer.
Output:
- Manual review message
- No final policy answer
grounding_status = Not Foundfailure_type = insufficient_evidence
The Supabase knowledge table stores approved billing/admin policy chunks.
Example fields:
| Field | Description |
|---|---|
chunk_id |
Unique source reference, e.g. ESC-001 |
document_name |
Source policy document |
section_title |
Policy section |
chunk_text |
Approved policy text |
keywords |
Retrieval keywords |
approved_status |
Approval status |
document_category |
Billing, refund, escalation, receipt, etc. |
version |
Knowledge-base version |
last_updated |
Policy update date |
Every question is logged to Google Sheets for review.
Tracked fields include:
- Question ID
- Staff question
- Retrieved chunk IDs
- Retrieved documents
- Retrieval confidence
- Answer returned
- Citation included
- Not-found flag
- Grounding status
- Failure type
- Manual review result
- Error type
- QA notes
This makes the system auditable and helps identify whether failures come from retrieval, evidence quality, or answer generation.
| Failure Type | Meaning |
|---|---|
none |
A grounded answer was returned |
no_candidates_retrieved |
No approved policy chunk matched retrieval |
insufficient_evidence |
Related chunks were found but were too weak to answer |
This failure-type tracking is important because it separates true knowledge-base gaps from evidence-quality issues.
Final QA testing used 10 staff-style questions.
| Metric | Result |
|---|---|
| Total tests | 10 |
| Grounded answers | 7 |
| Correct refusals | 3 |
| Citation compliance | 100% |
| Manual review passed | 10/10 |
| Logged errors | None |
Example tested cases:
| Staff Question | Expected Result |
|---|---|
| When should a billing dispute be escalated? | Grounded answer |
| Can we charge a late payment penalty? | No matching policy found |
| Can staff offer a 10% discount for patients who pay today? | Insufficient evidence |
| Can we send a receipt for a partial payment? | Grounded answer |
| Can we resend an invoice if the patient asks for another copy? | Grounded answer |
| What should staff do if a patient says they paid but the payment is missing? | Grounded answer |
| What should staff do if a patient requests a refund? | Grounded answer |
| What should staff do if a patient asks for a corrected invoice? | Grounded answer |
| Should we stop payment reminders if the account is under dispute review? | Grounded answer |
| Can we waive a patient’s remaining balance as a courtesy? | No matching policy found |
citedesk-rag-assistant/
├── README.md
├── workflow/
│ └── citedesk-rag-workflow.json
├── frontend/
│ └── citedesk.html
├── screenshots/
│ ├── workflow-architecture.png
│ ├── frontend-grounded-answer.png
│ ├── frontend-not-found.png
│ ├── frontend-insufficient-evidence.png
│ ├── portfolio-qa-view.png
│ └── qa-summary.png
└── docs/
└── architecture.md
- Staff asks a question through the CiteDesk frontend.
- The frontend sends the question to an n8n webhook.
- OpenAI extracts a short retrieval keyword.
- Supabase returns approved knowledge chunks.
- A Code node filters candidate chunks using keyword and metadata fields.
- OpenAI reranks the evidence and decides whether the question is answerable.
- If answerable, CiteDesk generates a concise answer and returns source cards.
- If not answerable, CiteDesk refuses and explains that manual review is needed.
- The result is logged to Google Sheets for QA review.
CiteDesk v1 is intentionally scoped.
Current limitations:
- Uses keyword-filtered retrieval instead of vector search
- Depends on quality of the approved knowledge chunks
- Requires manual review of test outputs
- Does not include user authentication
- Does not sync policies automatically from documents
- Does not provide patient-specific, legal, medical, or financial advice
Possible future upgrades:
- Vector search using embeddings
- Supabase pgvector retrieval
- Admin dashboard for adding/editing policy chunks
- Role-based access control
- Document upload and chunking pipeline
- Automatic stale-policy detection
- More detailed evaluation dashboard
- Human approval workflow for policy changes
Although this build uses a fictional healthcare billing/admin team, the same pattern can apply to many internal knowledge workflows: HR policy assistants, customer support SOP assistants, finance procedure assistants, onboarding knowledge bases, legal operations references, and client service playbooks.
This pattern applies anywhere a team needs consistent, sourced answers from internal documents.
This is a fictional portfolio project using demo healthcare billing/admin data. It is not intended for production use and does not provide legal, medical, financial, or patient-specific advice.
Built by Alieza San Diego
LinkedIn