Make the face lane work on a real face - #28
Merged
Conversation
Found by a real 54-second session, which reported zero blinks. detectBlinks sat inside the branch that only ran when the blink metric passed its gates. The metric requires 60 seconds of frontal exposure; the session had 54, so it correctly abstained -- and took every blink actually observed down with it. A publication threshold was suppressing the extraction beneath it, which is the exact inversion of what Tier 2 is for. An abstaining metric must still leave its observations behind, or the substrate only records sessions that did not need it. Detection now runs whenever there are bins, and returns empty rather than reaching a percentile of an empty set when there are none. Also instruments what the diagnostics could not previously explain: - per-bin usable fraction and the largest gap between usable frames - an acceptance curve simulating a fractional frame gate - the voice lane, which had no diagnostics at all The acceptance curve exists because a fractional gate is NOT obviously the fix. Dropping a burst of unusable frames leaves a hole where they were, and the 200 ms gap rule may simply become the new binding constraint. The curve reports bins accepted, bins lost to that gap, and bins lost to sample count at each candidate threshold, so the threshold gets chosen from a real session instead of from my guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A real 70-second session with a face in frame throughout produced 0 of 14 qualifying bins and, consequently, not one blink or expression. All 27 metrics correctly reported Not measurable. The reporting was right; the coupling was not. Blink and expression detection read screening.bins, so they inherited a gate built for a different purpose. The pose limits exist so CROSS-FRAME GEOMETRIC COMPARISON stays valid -- comparing one corner against the other, measuring asymmetry. A blink is not that. It is a relative aperture change within one eye over about 150 ms, during which head pose is essentially constant, so it survives a 15-degree turn intact. Holding detection to a standard designed for asymmetry cost that session everything it contained. Detection now runs on a looser stream: attribution, image quality, and geometry completeness, without the pose and calibrated-scale gates. Every event carries poseWithinMeasurementLimits, so rate and timing can use all of them while anything comparing the two sides can filter to the trustworthy set. Detectors emit a Detected* shape and whatever holds the pose stream annotates it -- geometry alone cannot assert that flag. Published metrics are untouched. The blink rate stays bin-derived and remains explicitly a rate over pose-qualified windows; these events answer the separate question of what the session actually contained. The session's other findings are recorded but not yet acted on: a fractional frame gate will not rescue Tier 3 on its own, because at the ~50% usable fraction the data shows, bins clear the sample-count minimum and then die on the 200 ms gap rule instead -- one of them by 2 ms. Also renames the e2e mount path and the audio worklet processor, which the PhenoMetrix rename missed because neither matched its patterns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tifiers Live telemetry stopped after the previous commit. The worklet is served from public/ at a fixed unhashed URL, so browsers cache it hard: a stale copy registers "phenometric-voice-capture" while the fresh bundle asks for "phenometrix-voice-capture", AudioWorkletNode throws, and the voice lane dies without a visible error. The 70-second session before that commit produced 6972 voice frames, so the break is squarely that rename. Renaming it was wrong on principle, not just in effect. It is a runtime registration identifier -- the same category as the "phenometric.<name>.vN" schema strings that were deliberately left alone during the rename, for exactly this reason. The rule was stated and then applied inconsistently. The worklet file is restored byte-for-byte from before the rename, so a browser holding a cached copy matches the manifest again rather than failing its integrity check. The explanation lives in voice-capture.ts, which Vite content-hashes, instead of in the file that must not change. The same blanket substitution had also corrupted two wire identifiers into "phenometrix.static-assets.v1" and "phenometrix.voice-worklet-message.v1". Both restored; all twenty schema identifiers verified intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Across three real sessions the all-or-nothing frame gate was the single reason nothing was ever measurable: it discarded 82% of one session's bins to exclude 22% of its frames, and 15 of 15 in another. One glance away cost the surrounding five seconds. No new threshold replaces it, because the pack already carried one. minimumDataPerBinMs of 4000 in a 5000 ms bin IS an 80% requirement; the all-or-nothing rule was redundant with it and far stricter. Dropping bad frames and letting the published requirement do its job makes the code enforce what the pack always said. Every retained frame is individually pose-valid, so the geometry stays sound -- the bin rests on less of it, which is precisely what minimumDataPerBinMs and minimumSamplesPerBin exist to bound. durationMs had to become honest first. It summed raw inter-frame gaps, so two frames 200 ms apart contributed 200 ms of "data" while carrying two samples -- harmless while the gate guaranteed no holes, wrong the moment it stopped. Each retained frame now represents one nominal step of observation. CORRECTS AN EARLIER PROJECTION. The acceptance curve reported that a fractional gate at 0.80 would have published metrics from the measured session. It would not have: the simulation omitted the actualSpanMs >= 4800 check. Losing frames from a bin EDGE shortens the usable span one-for-one, and the span rule allows 200 ms of slack against the data rule's 1000 ms. The curve now simulates span too, and reports bins lost to it. So the honest scope of this change is narrower than claimed: it recovers bins that lost frames in a SCATTERED pattern, and nothing else. An edge burst is rejected by span; a mid-bin burst is rejected by the gap rule. The measured sessions lost frames in multi-second bursts, so this alone will not make them measurable. The tests say so explicitly rather than using a loss pattern that flatters the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three measured sessions failed the face lane on pose, and the third made the
cause plain: median pitch 7.4 degrees against a limit of 10, with the median
sitting at the ceiling before any head movement at all. That is a camera
mounted below eye level, which is constant and harmless, and the gate could not
tell it from the subject turning away, which is neither.
Limits now measure deviation from the session's resting pose -- the per-axis
median, computed from the session itself so it needs no capture-path or
contract change and adapts to how the participant actually sat.
How far that reference may itself sit from frontal is bounded per axis, on
geometric grounds rather than preference:
yaw 10 rotation about the vertical axis foreshortens one side of the
face and not the other, so a constant offset biases every
left-versus-right measurement this system exists to make
pitch 20 symmetric across the midline; moves both sides together and
leaves asymmetry largely alone. Also the axis camera placement
actually offsets
roll 15 in-plane, and already cancelled by aligning the coordinate
x-axis to the inter-eye line before anything is measured
A session outside those bounds has no admissible reference and falls back to
frontal rather than accepting an arbitrary baseline.
Fixes an incomplete decoupling from two commits ago: evaluateVisualQuality
carries a SECOND pose gate of its own, looser but still absolute, and Tier-2
detection read only its `.usable` flag. Events still vanished past 15 degrees
for the same reason and with the same consequence. Pose is now excluded
explicitly there while lighting, sharpness, and framing remain required, since
those do corrupt the landmarks a blink is measured from.
Pack moves to 3.2.0 and the policy fields are renamed from maximumAbsolute* to
maximumYawDeviationDegrees and siblings, because the old names now describe
something the code does not do. Face measurements under 3.1.0 and 3.2.0 are not
interchangeable.
Also adds a copy-diagnostics button to the report view. Calibration is an
iterate-and-rerun loop and console-only output cost a hand-copy or the whole
session each round. Clipboard on an explicit click is the same act as selecting
the text by hand: no storage, no file, no network.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five fixes driven by three real capture sessions. Every one was found by
instrumenting the pipeline and running it against an actual face rather than a
fixture, and each commit records the measurement that motivated it.
The starting symptom: a live 478-point mesh tracking a face for 70 seconds,
and a report where all 27 metrics read
Not measurable. The report was correct.The gates were not.
What the sessions measured
The fixes
dd670d6— Tier-2 extraction was gated behind Tier-3 publication.detectBlinkssat inside the branch that only ran when the blink metric passedits 60-second exposure gate. A 54-second session correctly abstained and took
every blink it observed down with it. A publication threshold was suppressing
the extraction beneath it — the exact inversion of what the substrate is for.
079aed8— Tier-2 detection read the pose-gated bins.So when no bin qualified, nothing was extracted. The pose limits exist so
cross-frame geometric comparison stays valid; a blink is a relative aperture
change within one eye over ~150 ms and survives a 15° turn intact. Detection now
runs on a looser stream, and every event carries
poseWithinMeasurementLimitsso anything comparing left to right can stillfilter to the trustworthy set.
Result: 55 blinks ≈ 22/min over 74 s — squarely inside the normal
conversational range, and the first real-world validation of a threshold that
had only ever seen fixtures.
6d4e247— restored the worklet processor name and two schema identifiers.Renaming a runtime registration identifier broke live telemetry via browser
cache. Same category as the 20
phenometric.*schema strings deliberately leftalone during the rename; the rule was stated and then applied inconsistently.
6b758ee— unusable frames are dropped, not the whole bin.The all-or-nothing gate discarded 82% of one session's bins to exclude 22% of
its frames. No new threshold replaces it:
minimumDataPerBinMsof 4000 in a5000 ms bin already is an 80% requirement. This also corrects an earlier
projection in this branch — the acceptance curve omitted the span check, so
the claimed benefit was overstated. The curve now simulates span, and the tests
state the change's real (narrower) scope.
c3558f5— pose gates measure deviation from the session's resting pose.A camera below eye level reads as constant pitch that no amount of sitting still
removes. Resting bounds differ per axis on geometric grounds: yaw foreshortens
one side of the face and so biases asymmetry (10°); pitch is symmetric across
the midline (20°); roll is already cancelled by the coordinate system (15°).
Also fixes an incomplete decoupling —
evaluateVisualQualitycarries a second,looser absolute pose gate that Tier-2 still inherited.
Contract changes
Pack 3.1.0 → 3.2.0. Policy fields renamed
maximumAbsolute*→maximumYawDeviationDegreesand siblings, because the old names describesomething the code no longer does, plus three new resting bounds. Face
measurements are not interchangeable across this bump.
Also
A copy-diagnostics button on the report screen — calibration is an
iterate-and-rerun loop and console-only output cost a hand-copy or the whole
session each round. Clipboard on explicit click; no storage, no file, no
network.
Still open
The voice lane shows 0 segments across all three sessions: 32% of
speech-active frames fail the 15 dB SNR floor, fragmenting segmentation below
the 2-second minimum. Consistent proportion each run, so environmental rather
than code.
Verification
274 unit tests, typecheck, structure + digest check, production build, 6 browser
e2e.
🤖 Generated with Claude Code