PostCareAI

A Multimodal Multi-Agent System for Clinician Supervised Post-Operative Patient Follow-Up — patients submit daily wound photos and symptom logs via a mobile app; a fine-tuned MedSigLIP model assesses wounds; RAG + safety rules + LLM agents guide recovery and triage cases on a clinician web dashboard (human-in-the-loop).

Capstone prototype — not clinically validated. Model outputs and agent responses require review by qualified clinicians. Do not use for real medical decisions.

Highlights

Fine-tuned MedSigLIP (google/medsiglip-448) on SurgWound — 6-label multi-label wound classification
Macro ROC-AUC: 0.7450 on held-out test set (137 samples); 0.745 validation macro AUC at best epoch
Per-label Youden J thresholds tuned for clinical safety (high sensitivity on infection & urgency)
Multi-agent pipeline — triage → patient guidance → clinician handoff (Gemini or rule fallback)
Vector RAG over clinical care documents for 24/7 patient coaching
Dual frontend — mobile-first patient app + web clinician command center
Automated tests — pytest suite covering wound API, safety rules, triage, and RAG (26 tests)

Demo

Role	URL	Credentials
Login	http://127.0.0.1:5173/ (dev) or http://127.0.0.1:8000/ (prod build)	—
Patient	`/patient`	`patient@postcare.test` / `patient123`
Clinician	`/clinician`	`clinician@postcare.test` / `clinician123`

API docs: http://127.0.0.1:8000/docs

Screenshots

Patient app (mobile-first)

Dashboard	Daily check-in	Recovery metrics

AI Recovery Coach (RAG)

Clinician portal (web dashboard)

Command center	Case review — wound + AI scores	Case review — handoff notes

Architecture

System overview

flowchart TB
    subgraph Patient["Patient Mobile App"]
        P1[Daily check-in]
        P2[Wound photo]
        P3[Pain + symptoms + note]
        P4[Recovery chat]
    end

    subgraph Backend["FastAPI Backend"]
        API[REST API]
        CV[MedSigLIP<br/>6-label wound CV]
        SAF[Safety rules<br/>thresholds + symptoms]
        RAG[RAG engine<br/>MiniLM + care docs]
        AG[Multi-agent pipeline]
        DB[(SQLite<br/>cases + chat)]
        IMG[(Wound images<br/>data/uploads)]
    end

    subgraph Agents["Agents"]
        T[Triage agent]
        PA[Patient agent]
        CL[Clinician agent]
        GEM[PostCare-Gemini]
        RULES[PostCare-rules fallback]
    end

    subgraph Clinician["Clinician Web Dashboard"]
        Q[Priority queue]
        R[Case review modal]
        W[Wound photo + AI scores]
    end

    P1 --> P2 & P3
    P2 & P3 -->|POST /api/patients/upload| API
    P4 -->|POST /patient/case/id/chat| API

    API --> CV --> SAF --> AG
    API --> IMG
    AG --> T & PA & CL
    GEM -.->|if API key set| T & PA & CL
    RULES -.->|else| T & PA & CL
    RAG --> PA & P4

    AG --> DB
    API --> DB

    DB --> Q
    IMG --> W
    DB --> R
    Q --> R

Check-in → clinician review

sequenceDiagram
    actor Patient
    participant App as Patient App
    participant API as FastAPI
    participant Model as MedSigLIP
    participant Store as SQLite + Uploads
    participant Agents as Agent Pipeline
    actor Clinician
    participant Dash as Clinician Dashboard

    Patient->>App: Submit photo + pain + symptoms
    App->>API: POST /api/patients/upload
    API->>Model: Predict wound scores
    Model-->>API: 6-label probabilities
    API->>Store: Save case + wound image
    API->>Agents: Triage → Patient → Clinician notes
    Agents-->>API: Priority + guidance + handoff
    API-->>App: Case result + patient guidance

    Clinician->>Dash: Open review queue
    Dash->>API: GET /clinician/cases
    API-->>Dash: Prioritized case list
    Clinician->>Dash: Open case
    Dash->>API: GET /clinician/cases/{id}
    API-->>Dash: Scores + handoff + has_wound_image
    Dash->>API: GET /clinician/cases/{id}/image
    API-->>Dash: Wound photo
    Clinician->>Dash: Finalize review
    Dash->>API: POST /clinician/cases/{id}/review

Agent pipeline

flowchart LR
    A[Wound image] --> B[MedSigLIP inference]
    B --> C[6 probability scores]
    C --> D[Safety rules]
    E[Patient symptoms + pain] --> D
    D --> F[Safety flags]
    F --> G[Triage agent]
    G --> H{Priority}
    H -->|urgent| I[High priority queue]
    H -->|review| J[Needs review]
    H -->|routine| K[Routine]
    G --> L[Patient agent]
    L --> M[Recovery guidance]
    G --> N[Clinician agent]
    N --> O[Handoff summary + review note]
    P[RAG over care docs] --> L
    P --> Q[24/7 recovery chat]

Agents use PostCare-Gemini (gemini-2.0-flash) when POSTCARE_GEMINI_API_KEY is set; otherwise PostCare-rules fallback.

Data model

erDiagram
    CASE ||--o| WOUND_ASSESSMENT : has
    CASE ||--o| PATIENT_CONTEXT : has
    CASE ||--o| CLINICIAN_SUMMARY : has
    CASE ||--o| WOUND_IMAGE : stores

    CASE {
        string case_id PK
        string status
        string clinician_priority
        datetime created_at
    }

    PATIENT_CONTEXT {
        string patient_name
        int pain_score
        int post_op_day
        string procedure
        list symptoms
    }

    WOUND_ASSESSMENT {
        float healing_status
        float infection_risk
        float urgency
        float erythema
        float edema
        float exudate
    }

    CLINICIAN_SUMMARY {
        string summary
        string review_note
        json visual_findings
    }

    WOUND_IMAGE {
        string file_path
        string content_type
    }

Model performance (Run 2 — test set)

Model: medsiglip-448-surgwound-v2 · Checkpoint: checkpoint-180 · Test samples: 137

Ranking quality (AUC — threshold-independent)

Label	AUC	Notes
Exudate	0.846	Best-performing label
Edema	0.756
Healing status	0.752
Infection risk	0.730
Urgency	0.710	Rarest class (57 positives in train)
Erythema	0.676
Macro ROC-AUC	0.7450

Production operating points (Youden J thresholds)

Thresholds shift sensitivity/specificity tradeoff; macro ROC-AUC is unchanged.

Label	Threshold	Sensitivity	Specificity	Design intent
infection_risk	0.25	90.0%	45.3%	Catch infections early
urgency	0.24	87.5%	49.6%	Flag urgent cases for clinician
healing_status	0.38	83.6%	54.9%	Detect poor healing
erythema	0.51	67.5%	64.8%	Balanced redness detection
edema	0.71	33.3%	91.5%	Conservative — reduce false swelling alerts
exudate	0.78	50.0%	87.2%	Conservative — reduce false discharge alerts

{
  "healing_status": 0.38,
  "erythema": 0.51,
  "edema": 0.71,
  "infection_risk": 0.25,
  "urgency": 0.24,
  "exudate": 0.78
}

Training summary

Metric	Run 1 (baseline)	Run 2 (production)
Optimizer steps	40	300
Unfrozen blocks	4	8
Learning rate	single 5e-5	differential backbone/head
Thresholds	fixed 0.5	per-label Youden J
Val macro AUC (best epoch)	underfit	0.745
Training time	—	22.5 min

Run 1 → Run 2 (hyperparameters)

Parameter	Run 1	Run 2	Effect
`N_UNFREEZE`	4	8	~2× trainable capacity (~14% → ~28%)
`GRAD_ACCUM`	16	4	8 → 30 optimizer steps/epoch
`EPOCHS`	5	10	40 → 300 total optimizer steps (7.5×)
Learning rate	single 5e-5	differential backbone=1.5e-5 / head=8e-5	Preserves pretrained features, fast head convergence
Threshold	fixed 0.5	per-label (Youden's J)	Corrects miscalibration (e.g., healing sens=0.84/spec=0.26)

Key design decisions

Expanded selective freezing — Last 8 encoder blocks + classification head are trainable (~28% of params); deeper layers give the model more expressive capacity without saturating T4 VRAM
Differential learning rate — Backbone at BACKBONE_LR=1.5e-5 (preserves pretrained SigLIP features); head at HEAD_LR=8e-5 (fast learning from random initialization)
Masked BCE loss — 3 of 6 labels have MISSING values; loss is zeroed for those entries instead of dropping entire samples
Light augmentation — Horizontal flip, rotation, and color jitter for a small dataset (480 train images)
eval_loss for model selection — Validation set has only 69 images; per-label AUC is too noisy for checkpoint comparison
Per-label threshold tuning — Youden's J (J = sensitivity + specificity − 1) on the validation set replaces fixed threshold=0.5 after training

Full experiment lineage: training/experiments/EXPERIMENT_LOG.md

Class imbalance handling

Training set: 480 samples × 6 labels. pos_weight in BCE loss upweights rare positives:

Label	Positives	pos_weight
urgency	57	7.42
edema	50	6.56
exudate	70	5.24
infection_risk	78	5.15
erythema	129	2.59
healing_status	198	1.42

Frontends

Patient app (mobile-first)

Dark UI with bottom tab navigation:

Tab	Route	Purpose
Home	`/patient`	Recovery ring, metrics, medications, check-in CTA
Check-In	`/patient/log`	Camera upload, pain slider, symptom toggles
Recovery	`/patient/recovery`	AI guidance, wound scores, care path
Coach	`/patient/assistant`	RAG-powered recovery chat
You	`/patient/settings`	Profile and preferences

Clinician portal (web dashboard)

Desktop sidebar layout with case management:

Page	Route	Purpose
Dashboard	`/clinician`	Stats cards + recent assessments table
Patient Queue	`/clinician/queue`	Searchable, filterable case queue
Site Management	`/clinician/sites`	Ward / care site configuration
System Admin	`/clinician/system-admin`	API health and system status
User Management	`/clinician/add-user`	Add patient / clinician accounts

Priority pills: Urgent (red) · Review (amber) · Routine (green)

Quick start

Prerequisites

Python 3.11+
Node.js 18+
(Optional) CUDA GPU for faster inference
(Optional) HF_TOKEN for downloading gated MedSigLIP weights

1. Clone and install

git clone https://github.com/SyedAshhadIbrar/PostCareAI.git
cd PostCareAI

pip install -r requirements.txt

2. Export production model

After Run 2 training (or if weights already exist locally):

python training/scripts/export_production_model.py

This copies the best checkpoint and tuned thresholds.json into models/medsiglip/.

3. Run backend

uvicorn backend.main:app --host 127.0.0.1 --port 8000 --reload

Verify: http://127.0.0.1:8000/health

4. Run frontend (development)

cd frontend
npm install
npm run dev

Open http://127.0.0.1:5173 and log in with demo credentials above.

5. Production build (single server)

cd frontend && npm run build && cd ..
uvicorn backend.main:app --host 127.0.0.1 --port 8000

Serves React UI at http://127.0.0.1:8000/

6. Run tests

pytest tests/ -v

API overview

Endpoint	Method	Description
`/health`	GET	Model loaded, RAG index, Gemini status
`/wound/assess`	POST	Direct wound image assessment (MedSigLIP)
`/api/auth/login`	POST	Patient / clinician login
`/api/patients/upload`	POST	Daily check-in (photo + symptoms)
`/patient/status`	GET	Patient recovery status
`/patient/case/{id}/chat`	POST	RAG recovery chat
`/clinician/cases`	GET	Clinician case queue
`/clinician/cases/{id}`	GET	Case detail + AI handoff
`/clinician/cases/{id}/review`	POST	Mark case reviewed

Full interactive docs: http://127.0.0.1:8000/docs

Tech stack

Layer	Technologies
ML	PyTorch, Hugging Face Transformers, MedSigLIP-448, SurgWound dataset
MLOps	MLflow, YAML configs, experiment lineage (Run 1 → Run 2)
Backend	FastAPI, sqlite3, SQLite, Pydantic
RAG	sentence-transformers (`all-MiniLM-L6-v2`), pypdf, indexed from `rag/documents/`
Agents	Google Gemini 2.0 Flash (optional) + rule-based fallback
Frontend	React 18, Vite, Tailwind CSS, React Router
Testing	pytest, httpx

Project structure

PostCareAI/
├── backend/
│   ├── agents/          # Triage, patient, clinician, recovery chat
│   ├── routes/          # FastAPI routers (wound, patient, clinician, auth)
│   ├── services/        # Wound inference, RAG, safety, vector store
│   └── database/        # SQLite models and seed data
├── frontend/
│   └── src/
│       ├── components/  # Patient (mobile) + clinician (web) UIs
│       └── lib/         # API client, site config
├── models/medsiglip/    # Production inference artifacts (config, thresholds, weights*)
├── rag/documents/       # Clinical care guide PDFs for RAG
├── training/
│   ├── configs/         # run1_underfitting.yaml, run2_best.yaml
│   ├── experiments/     # EXPERIMENT_LOG.md
│   ├── scripts/       # train.py, export_production_model.py
│   └── outputs/         # Run artifacts (not in git)
├── tests/               # pytest suite (API, safety, triage, RAG)
└── requirements.txt

* Model weights are not committed to git. Export via training/scripts/export_production_model.py.

Environment variables

Variable	Default	Purpose
`HF_TOKEN`	—	Hugging Face auth for gated `google/medsiglip-448`
`POSTCARE_MODEL_DIR`	`models/medsiglip`	Override model artifact path
`POSTCARE_DEVICE`	auto	`cuda` or `cpu`
`POSTCARE_DATABASE_URL`	SQLite local	Database connection string
`POSTCARE_GEMINI_API_KEY`	—	Enable Gemini agents (optional)
`POSTCARE_GEMINI_MODEL`	`gemini-2.0-flash`	Gemini model ID
`VITE_API_BASE`	`http://127.0.0.1:8000`	Frontend API URL (dev)

MLOps workflow

Run 1 (underfit baseline)  →  Run 2 (production)  →  export_production_model.py  →  FastAPI inference
     40 steps                      300 steps                    models/medsiglip/
     fixed 0.5 thresholds          Youden J thresholds          thresholds.json

Artifact	Location
Experiment configs	`training/configs/`
Run lineage log	`training/experiments/EXPERIMENT_LOG.md`
MLflow tracking	`training/mlruns/`
Model metadata	`models/medsiglip/postcare_config.json`
Per-label thresholds	`models/medsiglip/thresholds.json`

Reproduce training:

cd training
pip install -r requirements.txt
export HF_TOKEN=...
python scripts/run_experiment_lineage.py   # Run 1 → Run 2
mlflow ui --backend-store-uri ./mlruns    # view experiments

Limitations

Not clinically validated — prototype for research and demonstration
Small dataset — 480 train / 137 test images; results may not generalize
Urgency sensitivity — 87.5% with tuned threshold, but AUC only 0.71; clinician review is essential
Edema — 102 missing labels in training; sensitivity remains low (33%)
Prototype auth — demo credentials; not production-grade security
No real-time monitoring — no alerting, FHIR integration, or EHR connectivity

License

See LICENSE.

Name		Name	Last commit message	Last commit date
backend		backend
docs/screenshots		docs/screenshots
frontend		frontend
models/medsiglip		models/medsiglip
rag		rag
scripts		scripts
tests		tests
training		training
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

PostCareAI

Highlights

Demo

Screenshots

Patient app (mobile-first)

Clinician portal (web dashboard)

Architecture

System overview

Check-in → clinician review

Agent pipeline

Data model

Model performance (Run 2 — test set)

Ranking quality (AUC — threshold-independent)

Production operating points (Youden J thresholds)

Training summary

Run 1 → Run 2 (hyperparameters)

Key design decisions

Class imbalance handling

Frontends

Patient app (mobile-first)

Clinician portal (web dashboard)

Quick start

Prerequisites

1. Clone and install

2. Export production model

3. Run backend

4. Run frontend (development)

5. Production build (single server)

6. Run tests

API overview

Tech stack

Project structure

Environment variables

MLOps workflow

Limitations

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages