FairPulse

An independent, open audit of the robustness and fairness of physiological-signal AI.

FairPulse evaluates how machine-learning models built on photoplethysmography (PPG) — the optical signal behind smartwatches and pulse oximeters — hold up when the device, the conditions, or the patient population (notably skin tone) change. It is not another PPG model: it is a neutral arbiter that takes existing open models and stress-tests them on one honest, leakage-aware protocol. A benchmark suite, not a single-score leaderboard.

Status — v1. Robustness audit on open data and a skin-tone fairness audit executed on OpenOximetry (PhysioNet restricted, used locally under a signed DUA, never redistributed). The headline finding is honest and two-level — see Results.

---

Quickstart — audit on synthetic fixtures in <1 min (no data, no weights, no DUA)

git clone https://github.com/sahimhiba-debug/fairpulse && cd fairpulse
uv sync                                  # core deps only (no torch needed for the demo)
uv run python scripts/00_demo_audit.py   # full pipeline on SYNTHETIC data -> a real report

This runs the complete audit (frozen-embedding probe → per-skin-group MAE → subject-bootstrap CIs → leave-one-skin-tone-out) on bundled synthetic fixtures and writes docs/demo_audit_report.md. No restricted data, no model weights, nothing under a DUA — it proves the pipeline and shows the report shape. (The numbers are illustrative, not real model performance.)

Audit YOUR model

The zoo interface is one method: embed(windows, fs) -> (N, embedding_dim). Wrap your model in it, then reuse the same pipeline:

import numpy as np
from fairpulse.demo import run_demo_audit                     # synthetic data, swap the model
from fairpulse.benchmark.evaluate import audit_embeddings, embed_segments

class MyModel:
    name = "my-ppg-model"
    embedding_dim = 256
    def embed(self, windows: np.ndarray, fs: int) -> np.ndarray:   # windows: (N, L) raw PPG @ fs Hz
        x = my_preprocess(windows, fs)                              # your resample / normalize
        return my_encoder(x)                                       # -> (N, 256) numpy

# (a) quickest — drop your model into the synthetic demo:
report = run_demo_audit(model=MyModel())

# (b) on YOUR data — build a list[fairpulse.data.base.Segment] (signal, fs, subject_id,
#     skin_group, label), then:
# embs   = embed_segments(MyModel(), segments)
# report = audit_embeddings(embs, labels, subjects, skin_groups=groups)   # MAE/group + CIs + fairness

Splits are always at the patient / device / skin-tone level (never the segment level), with subject-level bootstrap confidence intervals. See src/fairpulse/zoo/ for real adapters (PaPaGei-S, Pulse-PPG) as worked examples.

What v1 does

• Model zoo — uniform 512-d adapters for PaPaGei-S and Pulse-PPG (encoders vendored and attributed, see THIRD_PARTY_NOTICES.md); Tier-1 wrappers prepared for AnyPPG, SIGMA-PPG, PPG-VascularAge (weights not bundled).
• Leakage-aware protocol — frozen-embedding Ridge probe; LOSO / leave-one-device-out / leave-one-skin-tone-out; subject-bootstrap CIs.
• Robustness audit (open data) — per-condition breakdown + cross-dataset transfer.
• Fairness audit (OpenOximetry, local) — per-skin-group error + the data-level SpO2↔SaO2 bias and occult-hypoxemia rate.
• Audit tool (stretch) — a FastAPI endpoint that takes embeddings and returns a report.

Figure 2 — Leakage-aware audit protocol: frozen-embedding Ridge probe, patient/device/skin-tone splits (never segment-level), subject-bootstrap CIs, LOSTO.

Results (v1)

Robustness (open data, heart-rate MAE in bpm, LOSO ± 95% CI): Pulse-PPG leads on these wrist/finger sets (field-trained), and physical effort ~doubles the error for both models — a robustness observation, not a winner. Full tables in docs/rapport_v1.md.

Fairness by skin tone (OpenOximetry) — read at TWO levels:

1. Data level (model-independent) — robust & significant. The pulse oximeter overestimates true arterial saturation more for darker skin: SpO2−SaO2 bias light 0.67 → dark 2.21 pts (CIs disjoint), occult hypoxemia ~3% (dark) vs ~0% (light). The documented bias (Sjoding 2020, Fawzy 2022) is reproduced.
2. Model level — same direction, NOT conclusive. Both PaPaGei-S and Pulse-PPG show higher saturation-estimation error on dark skin (SpO2 MAE dark ~2.4 vs light ~1.9), but the dark group is small (11 patients with usable PPG) → wide, overlapping CIs.

Numbers, CIs and the honest reading are in docs/rapport_v1.md §5.4; machine-readable in docs/equity_results_openox.json.

Figure 1 — Data-level pulse-oximeter bias (SpO₂−SaO₂) by skin tone, OpenOximetry. Model-independent; 95% CI (subject-bootstrap). dark n=13 (small sample).

Data

• Open, downloadable (robustness): PPG-DaLiA (CC-BY-4.0), BIDMC (ODC-BY) — scripts/02_fetch_assets.py.
• OpenOximetry (fairness): PhysioNet restricted — requires a signed DUA. It stays local in data/raw/ (gitignored), is never committed/pushed, and never sent to any online service (including the FastAPI endpoint). You download it yourself; see scripts/openoximetry_download.md.
• Synthetic fixtures (no DUA): bundled in src/fairpulse/demo.py — the Quickstart path.

No raw data, model weights, or secrets are ever committed (.gitignore enforced).

Limits (the gaps map)

Honest by design — full table in docs/rapport_v1.md §6:

• Skin-tone fairness: done on OpenOximetry, but the dark group is small (13 metadata / 11 with PPG) → the model-level gap is suggestive, not conclusive; widening it means combining datasets.
• Leave-one-device-out for a PPG model: not possible even with OpenOximetry — its raw PPG comes from a single reference recorder; the 53 oximeters vary the label, not the model input.
• Saturation labels are encounter-level (the v1.1.1 PPG is not time-synced to SpO2/SaO2 samples).

Principles

• A suite, not a leaderboard. Existing models are subjects of audit, not baselines to beat.
• Splits at patient / device / skin-tone level; report leave-one-out with confidence intervals.
• Code is MIT (see LICENSE). Datasets and model weights keep their own licenses and access terms. PaPaGei weights carry no declared license → to_verify, local research use only, never redistributed (see THIRD_PARTY_NOTICES.md).
• v1 is research / evaluation tooling, not a medical device.

Layout

src/fairpulse/
  zoo/          uniform inference adapters (PaPaGei-S, Pulse-PPG -> 512-d) + _vendor encoders
  data/         loaders: PPG-DaLiA, BIDMC, OpenOximetry; common Segment schema
  benchmark/    leakage-aware splits, Ridge probe, metrics, subject-bootstrap CIs, fairness
  audit/        report rendering + FastAPI endpoint (stretch)
  demo.py       synthetic fixtures + DemoModel (the Quickstart path)
scripts/        00_demo_audit (fixtures) · 02_fetch · 03_run_audit (open) · 04 equity · 05 integrity
docs/           rapport_v1.md (results + gaps map), catalogs, check logs

Reproducibility & integrity

59 tests, ruff clean. Report figures are checked against the audit JSON. Raw data, weights and secrets are never committed.

License · Citation · Contributing

Citation. If you use FairPulse in your research, please cite:

Sahim Elmir, H. (2026). FairPulse: an independent robustness & fairness audit suite for physiological-signal AI (PPG) (v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.20723318

A CITATION.cff file is also included for citation managers.

Acknowledgements

FairPulse audits, and is grateful to, the open work it builds on — PaPaGei (Nokia Bell Labs), Pulse-PPG, the OpenOximetry repository, and the public datasets PPG-DaLiA and BIDMC. See THIRD_PARTY_NOTICES.md for attribution and license status.