🩺 Clinical Triage Agent

A calm, voice-enabled first-aid triage companion that helps people decide whether a situation needs emergency care — or can be safely managed at home.

Built at the Healthcare × AI Hackathon, New York City 2026, this prototype demonstrates how a multimodal agentic AI system can reduce unnecessary emergency room visits by guiding people through first-aid assessment in plain, reassuring language — no doom-scroll of worst-case scenarios, no replacement for professional care.

---

The idea

Someone twists their ankle. Instead of panicking and driving to the ER (or Googling their way into a spiral of worst-case diagnoses), they open this app, describe what happened, optionally share a photo or voice message, and receive calm, evidence-based first-aid guidance — grounded in real clinical documents, not hallucinated advice.

If things look more serious, the agent checks in with the user before suggesting escalation: urgent care, a call to their doctor, or 911. It never jumps to conclusions, and it never lists ten terrifying possibilities when one reassuring answer will do.

This is a proof-of-concept prototype. The backend runs on your local machine — no cloud server required. It is not a medical device, and it is not a replacement for professional medical advice.

---

What it can do

• 💬 Text chat — describe symptoms in plain language
• 🎙️ Voice input — record a voice message; Whisper transcribes it
• 🔊 Voice output — agent responses spoken aloud via ElevenLabs
• 📷 Image input — share a photo for visual assessment (bruising patterns, swelling, rashes, etc.)
• 🧠 RAG-grounded answers — responses anchored in real clinical first-aid documents, not pure LLM generation
• 🚨 Escalation logic — four-tier system (monitor at home → call your doctor → urgent care → call 911), always checking with the user before escalating
• 🌐 Web frontend — clean chat UI built with React via Lovable, accessible from any browser

---

Architecture

---

Project structure

triage_agent/
├── agent/
│   └── triage_agent.py      # LangGraph agent — core brain
├── escalation/
│   └── escalation.py        # 4-tier escalation logic
├── frontend/
│   ├── app.py               # FastAPI server
│   └── TriageChat.jsx       # React frontend component (Lovable)
├── rag/
│   ├── ingest.py            # Load clinical docs into ChromaDB
│   └── retriever.py         # Query the knowledge base
├── voice/
│   └── voice.py             # ElevenLabs TTS + Whisper STT
├── data/
│   └── docs/                # Place clinical PDFs/TXTs here
├── NOTICE                   # Attribution and third-party credits
├── LICENSE                  # Apache 2.0
├── requirements.txt
└── .env.example

---

Quickstart

Prerequisites

• Python 3.11 or 3.12
• Homebrew (macOS) for system dependencies
• API keys for OpenAI and ElevenLabs
• A free ngrok account for exposing the local backend
• A Lovable account for the web frontend

1. Clone the repo

git clone https://github.com/YOUR_USERNAME/triage-agent.git
cd triage-agent

2. Install system dependencies (macOS)

# Required for audio recording (pyaudio)
brew install portaudio

3. Create and activate a virtual environment

conda create -n triage-agent-env python=3.12
conda activate triage-agent-env

4. Install Python dependencies

pip install -r requirements.txt

# Whisper installs separately to avoid build conflicts
pip install openai-whisper

# If pyaudio fails via pip, use conda instead
conda install -c conda-forge pyaudio

5. Set up environment variables

cp .env.example .env

Edit .env with your keys:

OPENAI_API_KEY=sk-proj-...
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=aKw9UnnjRq5scbeeGI7Z  # Belle B (default)
CHROMA_DB_PATH=./data/chroma_db
CHROMA_COLLECTION=clinical_triage

6. Add clinical documents

Place first-aid PDFs or .txt files in data/docs/. See the Clinical Knowledge Base section for recommended free sources.

7. Ingest documents into ChromaDB

python rag/ingest.py

8. Test the agent in the terminal (CLI mode)

python agent/triage_agent.py

Type your symptoms and press Enter. To include an image, use:

You: I twisted my ankle | /path/to/photo.jpg

Type quit to exit.

---

Running the full web interface

The web interface requires three things running simultaneously: the backend server, an ngrok tunnel to expose it, and the Lovable frontend pointing at the tunnel URL.

Terminal 1 — start the backend

uvicorn frontend.app:app --reload --port 8000

The --reload flag auto-restarts the server when you save changes. The API docs are available at http://localhost:8000/docs.

Terminal 2 — expose the backend via ngrok

ngrok http 8000

Copy the https://xxxx.ngrok-free.app URL from the output. This URL is stable as long as this terminal stays open — you only need a new one if you restart ngrok.

Lovable frontend

1. Create a new project at lovable.dev
2. Upload frontend/TriageChat.jsx as src/components/TriageChat.jsx
3. Set the environment variable in Lovable project settings:

   VITE_API_BASE_URL=https://xxxx.ngrok-free.app
   

4. Update this value each time you get a new ngrok URL

Testing individual modules

Module	Command	Also testable at
RAG ingest	python rag/ingest.py	Local computer only
RAG retrieval	python rag/retriever.py	Local computer only
Triage agent	python agent/triage_agent.py	Local computer (CLI)
Escalation logic	python escalation/escalation.py	Local computer only
Voice TTS only	python voice/voice.py --tts-only	ElevenLabs playground
Voice STT (file)	python voice/voice.py --stt-file /path/to/audio.wav	Local computer
Full API server	uvicorn frontend.app:app	Local computer → ngrok → Lovable

---

API endpoints

Once the server is running, three endpoints are available:

Endpoint	Method	Description
/chat	POST	Send a message (+ optional image)
/transcribe	POST	Upload audio, receive transcribed text
/tts	POST	Convert text to ElevenLabs speech
/health	GET	Health check

Interactive docs: http://localhost:8000/docs

---

---

Clinical knowledge base

The RAG knowledge base is built from clinical first-aid documents you provide. Due to copyright restrictions, most source documents used in the original prototype cannot be redistributed here.

Freely redistributable sources (recommended starting point):

Source	License	URL
WHO First Aid guidelines	CC BY-NC-SA 3.0 IGO	who.int
MedlinePlus (NLM/NIH authored)	US public domain	medlineplus.gov
CDC health topics	US public domain	cdc.gov/az

Topics recommended for good coverage:

• Sprains and fractures (ankle, wrist, knee)
• Lacerations and wound care
• Burns (minor vs. when to seek care)
• Head injuries and concussion red flags
• Allergic reactions and anaphylaxis
• Chest pain and cardiovascular symptoms
• Abdominal and pelvic pain
• Respiratory conditions
• Fever in adults
• Insect bites and stings
• Heat and cold exposure

Limiting the knowledge base to freely available sources will reduce retrieval breadth compared to the original deployment.

---

Limitations and future directions

This is a proof-of-concept prototype, not a production system. Current limitations include:

• Local backend only — the server runs on your machine and requires it to stay on and connected
• No persistent sessions — conversation history lives in memory and is lost when the server restarts
• No authentication — the API has no access control
• Knowledge base breadth — retrieval quality depends on the documents you ingest

Potential directions for a production system:

• Integration with patient-facing health portals as a plug-in triage chatbot, enabling access to a patient's medical history for more personalized guidance
• MCP (Model Context Protocol) integration to connect with electronic health record systems
• Persistent session storage (Redis or a database) for multi-turn conversations across devices
• Clinician-reviewed response templates for common presentations
• Multilingual support for broader accessibility
• HIPAA-compliant deployment infrastructure

---

Origin

This project originated at the Healthcare × AI Hackathon, New York City, 2026.

See NOTICE for full contributor credits and third-party attributions.

---

License

Copyright 2026 Eduardo Castro.

Licensed under the Apache License, Version 2.0. See LICENSE for the full text.

This software is provided for research and educational purposes. It is not a medical device and must not be used as a substitute for professional medical advice, diagnosis, or treatment. In an emergency, call your local emergency number immediately.