Derive HR from the WHOOP 4.0 v25 PPG waveform (parity with the v26 PpgHr lane)

[ryanbr]

Summary

PpgHr + decodeWhoop5HistoricalV26 recover HR from the optical PPG waveform for v26 (WHOOP 5/MG). v25 (WHOOP 4.0) carries the same kind of waveform — PostHooks.swift:326 even labels bytes 23–73 PPG waveform (optical) — but it's only marked as a region, never extracted or fed to the estimator. So v25 straps decode motion + timestamp but no HR, even though the machinery to derive it now exists. This looks recoverable.

Evidence

24 real v25 type-47 reject frames from a user's strap log (App 1.93, WHOOP 4.0, resting), across 3 historical sessions. The log's live HR notify lines (realtime type-40/43) are an independent ground truth — they don't come from these historical records.

1. All 3 sessions autocorrelate to the right HR. Concatenating each session's per-second records into a 24 Hz stream and taking the normalised autocorrelation peak:

session (trim)	recovered	live-HR ground truth	conf
70476	60.0 bpm	~60 bpm	0.89
70480	60.0 bpm	~60 bpm	0.75
70484	60.0 bpm	~60 bpm	0.53

2. The samples are i16-LE on an ODD byte grid — a clean alignment signature (even offsets read i16s straddling sample boundaries → garbage):

offset	parity	HRs (3 sessions)	conf
15	odd	[60, 60, 60]	0.84
16	even	[69, 53, 65]	0.12
17	odd	[60, 60, 60]	0.72
18	even	[69, 32, 33]	0.12
19	odd	[60, 60, 60]	0.72
20	even	[131, 32, 44]	0.13

3. The per-record signal is visibly a pulse. One record at odd offset 15, DC-removed (first ~4 samples are header/unix-tail bleed, then a clean rise→fall→rise):

[29597, -418, -24379, -418,    <- header, waveform not started
 -1137, -999, -1035, -976, -983, -604, 525, 1219, 1452, 1179, 542, 48,
 -567, -1281, -1820, -1908, -958, 539, 1137, 1234]   <- PPG pulse

The clean pulse spans roughly bytes 25–66; reading past ~67 corrupts it (the fields before gravity@73). Consistent with — and a refinement of — the existing 23–73 region label.

What's pinned vs. what needs your corpus

• Pinned here: feasibility; odd-offset i16 encoding; pulse shape; approximate span ~25–66; the ~24 Hz / one-record-per-second model matching v26.
• Needs the v26 rig: exact start/end byte + sample count (21 vs 24) and validation across non-resting HR, multiple straps, motion artifact, using the analyze_v26_waveform.py + live-HR-ground-truth methodology. The evidence here is resting-only, one strap, one 8 s autocorrelation window per session — promising, not conclusive.

Proposed work

1. Pin the byte-exact v25 waveform layout + sample rate against a real capture corpus (v26 methodology).
2. Add decodeWhoop4HistoricalV25 PPG extraction (odd-offset i16 → ppg_waveform) mirroring decodeWhoop5HistoricalV26.
3. Route it through the existing PpgHr lane so v25 yields HR alongside the gravity it already decodes — which also backfills retroactively via the reject-archive replay (Harden the reject-archive replay gate (failure-aware + per-app-version) #152). Keep PpgHr constants in lockstep across Swift/Android.

Repro

Save the dumped rejected frame[..] hex lines from a v25 strap log to log.txt, then:

import re
sessions={}; cur=None
for line in open('log.txt'):
    t=re.search(r'trim=(\d+)\)', line)
    if 'undecodable' in line and t: cur=t.group(1); sessions[cur]=[]; continue
    m=re.search(r'rejected frame\[\d+\] \d+B: ([0-9a-f]+)', line)
    if m and cur is not None: sessions[cur].append(bytes.fromhex(m.group(1)))
def i16(b,o): return int.from_bytes(b[o:o+2],'little',signed=True)
fs=24
def hr(sig):
    n=len(sig); m=sum(sig)/n; x=[v-m for v in sig]; a0=sum(v*v for v in x)
    if a0==0: return 0,0
    lo,hi=max(1,int(60*fs/220)),min(n-1,int(60*fs/30))
    b=max(range(lo,hi+1),key=lambda l:sum(x[k]*x[k+l] for k in range(n-l)))
    return 60*fs/b, sum(x[k]*x[k+b] for k in range(n-b))/a0
for off in (15,16,17,18,19,20):           # odd = clean pulse, even = garbage
    out=[]
    for fr in sessions.values():
        out.append(hr([i16(f,off+2*k) for f in fr for k in range(24)]))
    print(off, 'ODD' if off%2 else 'EVEN', [(round(h),round(c,2)) for h,c in out])

[ryanbr]

Possible diff (illustrative — bounds need validation)

The PpgHr lane is already generic: HistoricalStreams.swift:85 collects any record that emits ppg_waveform and :140 runs PpgHr.derivePpgHr on it. So v25 needs no new HR plumbing — it only has to emit ppg_waveform the way decodeWhoop5HistoricalV26 does. (Nice cross-check: v26 already strides odd offsets stride(from: 27, to: 75, by: 2), matching the odd-grid finding above.)

The whole change is to replace the region marker in the v25 branch of PostHooks.swift with a sample extraction:

--- a/Packages/WhoopProtocol/Sources/WhoopProtocol/PostHooks.swift
+++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/PostHooks.swift
@@ v25 branch (version == 25)
             fb.parsed["rr_intervals"] = .intArray([])
-            fb.region(23, 73, "PPG waveform (optical)", "ppg")
+            // PPG waveform: i16-LE samples on an ODD byte grid (even offsets read misaligned i16s →
+            // noise), one record per second, ~24 Hz — same model as v26. Emitting `ppg_waveform` is
+            // ALL that's needed: HistoricalStreams collects it and PpgHr.derivePpgHr recovers per-second
+            // HR (its conf>=0.3 gate drops noisy windows, so a few garbage tail samples are harmless).
+            // NOTE: [25:73) recovers the strap-log live HR on real frames, but start/end/count are NOT
+            // byte-pinned — validate against a ground-truth corpus (analyze_v26_waveform.py method).
+            var ppg: [Int] = []
+            for off in stride(from: 25, to: 73, by: 2) {        // 24 samples, ends before gravity@73
+                guard let u = u16(frame, off) else { break }
+                ppg.append(u >= 32768 ? Int(u) - 65536 : Int(u)) // i16 LE
+            }
+            if !ppg.isEmpty {
+                fb.add(25, ppg.count * 2, "ppg_waveform", "ppg", value: .intArray(ppg),
+                       note: "optical PPG @~24 Hz, LE-i16 (odd grid) — bounds pending validation")
+                fb.parsed["ppg_sample_count"] = .int(ppg.count)
+            }

That's the entire functional change — extractHistoricalStreams + PpgHr do the rest, and it backfills retroactively through the reject-archive replay (#152).

Caveats baked into the comment, repeated for emphasis: the [25:73) window (24 odd-offset samples ending at gravity@73) is the structurally-clean hypothesis — it recovers 60 bpm on the resting frames here, but my data shows offsets ≥21 starting to degrade, so the true tail may be a couple samples shorter (~byte 66). Pin start/end/count against non-resting, multi-strap captures before trusting it.