ONYX — Autonomous Medical Proxy

"Your health, handled."

An autonomous medical concierge that cross-references patient records against live conversations and webcam scans to prevent dangerous drug interactions — and places real pharmacy calls on the patient's behalf.

---

Inspiration

Managing prescriptions, refill timing, and allergy safety is overwhelming — especially for patients juggling multiple medications or relying on caregivers. A single missed interaction or an unnoticed allergen can have serious consequences.

Onyx was built to act as a always-on medical proxy: ingest a patient's medical history once, then use that dossier to power every interaction — voice, vision, and telephony — so safety checks and pharmacy logistics happen automatically, not after the fact.

---

What It Does

Onyx is a full-stack medical concierge with a luxury dashboard UI and a FastAPI backend wired to real AI and telephony services.

Onboarding — Upload a medical PDF or image. Gemini 2.0 Flash (via OpenRouter) parses it into a structured patient JSON profile: demographics, allergies, current medications, caregiver contacts, and refill metadata.

Dashboard — A glassmorphism patient command center shows the dossier, medication list, refill status, scan history, and a notifications bell. Patients speak to Onyx via the browser mic; responses are synthesized with ElevenLabs eleven_v3.

Visual Pill Scanner — Hold a medication bottle up to the webcam. GPT-4o-mini Vision identifies the drug, class, and a plain-language description, then cross-checks it against the patient's known allergies. A CRITICAL alert fires before the patient can consume a conflicting drug.

Real-Time Telephony — When a refill or prescription action is needed, Onyx places an outbound Twilio call to the pharmacy. The message is synthesized with ElevenLabs eleven_turbo_v2_5 (low-latency telephony mode), played via TwiML <Play>, and the pharmacist can confirm with speech or DTMF. If they decline, Onyx responds with a persuasion follow-up.

Refill Reminders — A daily APScheduler job flags medications whose last_fill_date + days_supply is within reminder_days_before, surfaces a banner in the dashboard, drives the notifications bell, and can trigger an automated pharmacy call.

---

Architecture

flowchart LR
    subgraph Browser["Browser (Patient UI)"]
        onboarding["onboarding.html"]
        dashboard["index.html (dashboard)"]
        webcam["Webcam pill scan"]
        mic["Mic → /api/chat → TTS"]
        bell["Notifications bell"]
    end

    subgraph Backend["FastAPI Backend (main.py)"]
        dossier["patient_profile.json"]
        process["/api/process_report"]
        scan["/api/scan"]
        tts["/api/tts"]
        alert["/api/caregiver/alert"]
        twilio_conv["/api/twilio/conversation"]
        scheduler["APScheduler"]
    end

    subgraph Services["External Services"]
        openrouter["OpenRouter\n(Gemini + GPT-4o-mini)"]
        elevenlabs["ElevenLabs"]
        twilio["Twilio Voice"]
        pharmacy["Pharmacy phone line\n(CAREGIVER_PHONE_NUMBER)"]
    end

    Browser <-->|REST / static| Backend
    process --> openrouter
    scan --> openrouter
    tts --> elevenlabs
    alert --> elevenlabs
    alert --> twilio
    twilio_conv --> elevenlabs
    twilio <-->|TwiML Gather / Play| pharmacy

Loading

All LLM calls route through OpenRouter — a single OPENROUTER_API_KEY covers both Gemini (document parsing) and GPT-4o-mini (chat + vision).

The Safety Loop

flowchart TD
    A[Upload medical report] --> B[Gemini extracts allergies + medications]
    B --> C[Save to patient_profile.json]
    C --> D[Scan pill bottle via webcam]
    D --> E[GPT-4o-mini Vision with allergy context]
    E --> F{allergy_conflict?}
    F -->|true| G[CRITICAL ALLERGEN DETECTED alert]
    F -->|false| H[Safe — show drug info]
    G --> I[Persist to scan_history]
    H --> I
    I --> J[Surface in notifications bell for 7 days]

Loading

The Pharmacy Call Loop

flowchart TD
    A[Patient requests refill via voice or UI] --> B["/api/chat sets alert_triggered: true"]
    B --> C["/api/caregiver/alert with intent + meds"]
    C --> D[ElevenLabs synthesizes pharmacy message]
    D --> E[Audio → CDN or /api/audio via BASE_URL]
    E --> F[Twilio outbound call with TwiML Play + Gather]
    F --> G{Pharmacist response}
    G -->|yes / press 1| H[Confirm refill and hang up]
    G -->|no| I["/api/pharmacy/response persuasion follow-up"]

Loading

---

Tech Stack

Layer	Tech	Purpose
Frontend	HTML, Tailwind CSS, GSAP	Onboarding flow + patient dashboard (vanilla JS, no framework)
Backend	Python, FastAPI, APScheduler	REST API, static file serving, daily refill scheduler
Document parsing	Gemini 2.0 Flash via OpenRouter	PDF/image → structured patient JSON
Chat + Vision	GPT-4o-mini via OpenRouter	Intent detection, conversational replies, pill identification
In-app TTS	ElevenLabs eleven_v3	Natural voice responses in the browser
Telephony TTS	ElevenLabs eleven_turbo_v2_5	Low-latency audio for Twilio <Play>
Telephony	Twilio Voice + TwiML	Outbound pharmacy calls, inbound conversation handler
PDF parsing	PyMuPDF (fitz)	Extract text from uploaded medical PDFs
Tunneling	localhost.run SSH / localtunnel	Expose local server so Twilio can reach webhooks

---

Project Structure

Onyx/
├── README.md
├── .env.example                 # API key template
├── patient_profile.json         # Generated at runtime (patient dossier)
├── image.png                    # Dashboard screenshot
│
├── frontend/
│   ├── onboarding.html          # Medical document upload + profile initialization
│   └── index.html                 # Main dashboard (scanner, chat, notifications, refills)
│
└── backend/
    ├── main.py                    # FastAPI app — all API routes + static serving
    ├── requirements.txt           # Python dependencies
    ├── start_tunnel.py            # localhost.run SSH tunnel → writes BASE_URL to .env
    ├── start_lt.py                # localtunnel fallback
    ├── get_models.py              # ElevenLabs voice discovery utility
    ├── sim_pharmacy.sh            # curl helper to simulate pharmacist DTMF response
    └── audio_cache/               # Generated at runtime (TTS MP3 cache for Twilio)

---

API

Patient & Onboarding

Endpoint	Method	Description
/onboarding	GET	Serve onboarding page
/api/patient	GET	Return full patient dossier
/api/process_report	POST	Upload PDF/image → Gemini extraction → save profile

Conversational

Endpoint	Method	Description
/api/chat	POST	STT transcript → intent + alert_triggered + spoken reply text
/api/tts	POST	Text → ElevenLabs MP3 audio stream
/api/convaI/message	POST	Multi-turn conversation handler (call SID + speech)

Visual Scanner

Endpoint	Method	Description
/api/scan	POST	Base64 webcam image → drug ID + allergy conflict check
/api/scan/history	GET	Scan history (?bookmarked_only=true optional)
/api/scan/history/{id}	PATCH	Toggle bookmark on a scan entry

Refills & Notifications

Endpoint	Method	Description
/api/refill/status	GET	Medications due for refill (banner data)
/api/refill/request-pharmacy	POST	Trigger pharmacy call for due medications
/api/notifications	GET	Unified bell feed (refill due, upcoming, allergy alerts)

Telephony

Endpoint	Method	Description
/api/twilio/conversation	POST	TwiML webhook — inbound call conversation loop
/api/caregiver/alert	POST	Outbound pharmacy call with synthesized message
/api/pharmacy/response	POST	TwiML webhook — handle pharmacist yes/no response
/api/audio/{audio_id}	GET	Serve cached TTS audio for Twilio <Play>

---

Setup & Run

Prerequisites

• Python 3.9+
• Node.js + npx (only if using the localtunnel fallback in start_lt.py)
• API keys for Twilio, ElevenLabs, and OpenRouter

1. Clone the repo

git clone https://github.com/Akhileshreddym/Onyx.git
cd Onyx

2. Install dependencies

pip install -r backend/requirements.txt

3. Configure environment variables

Copy .env.example to .env at the project root and fill in your keys:

OPENROUTER_API_KEY=...
ELEVENLABS_API_KEY=...
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
TWILIO_PHONE_NUMBER=...
CAREGIVER_PHONE_NUMBER=...   # pharmacy number Onyx will call

BASE_URL is written automatically by the tunnel script in step 5.

4. Start the FastAPI server

From the project root:

uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload

5. Expose the server publicly (required for Twilio)

In a second terminal:

python3 backend/start_tunnel.py

This forwards port 8000 through localhost.run and writes a public *.lhr.life URL into .env as BASE_URL. A localtunnel alternative is available at backend/start_lt.py.

Configure your Twilio phone number's voice webhook to ${BASE_URL}/api/twilio/conversation (POST) so incoming calls hit Onyx.

6. Open the app

Initialize a patient profile first:

http://localhost:8000/onboarding

Then open the main dashboard:

http://localhost:8000/

---

Design System: "Gold Concierge"

• Deep forest-black backgrounds (#050705) with subtle radial gradients
• Glassmorphism panels — frosted blur, gold wire borders, soft drop shadows
• Serif display type (Playfair Display / Cormorant Garamond) paired with Inter for body text
• Classic gold accents (#D4AF37) for CTAs, badges, and active states
• Critical alerts in deep red (#ef4444) with pulsing border animations

---

Honest Assessment

Aspect	Status	Notes
PDF parsing → patient profile	Correct	Gemini 2.0 Flash via OpenRouter extracts structured JSON from uploaded documents
Webcam pill scanner + allergy check	Correct	GPT-4o-mini Vision identifies drug and cross-references patient allergies
Twilio outbound pharmacy calls	Correct	TwiML <Play> + <Gather> with ElevenLabs synthesized audio
Persuasion follow-up on pharmacist decline	Correct	/api/pharmacy/response webhook handles no/decline responses
Browser speech-to-text	Limitation	Uses browser Web Speech API (Chrome-focused); no dedicated STT provider; silent fallback if mic captures nothing
Patient storage	Limitation	Single patient_profile.json file, not a real database or EHR
HIPAA compliance	Limitation	Not compliant — no encrypted storage, audit logs, or BAA-covered providers
Tunnel dependency	Limitation	Requires localhost.run or localtunnel for Twilio webhooks, not production-ready

---

What's Next for Onyx

• Dedicated STT provider (Deepgram / Whisper) instead of browser Web Speech API
• FHIR / EHR integration instead of PDF upload
• SMS caregiver alerts via Twilio Messaging
• Multi-patient household support
• HIPAA-compliant deployment (encrypted storage, audit logs, BAA-covered providers)
• Proactive drug interaction checks across the full medication list (not just allergies)

---

Contributors

Name	GitHub
Akhilesh Reddy Mallu	@Akhileshreddym
Shawn Madadha	@ShawnMadadha

---