Person tracker for flowse.
RealSense D435 hand/body tracking app built with openFrameworks. Detects foreground blobs via depth background subtraction, finds the closest contour point to a configurable 3D query line, and sends results via OSC.
RealSense D435
(depth + color, pixel-aligned)
│
▼
┌─────────────────────┐
│ Color frame │
│ │
│ Ring buffer (N │
│ frames), per-pixel │
│ max → smoothColor │ ← colorMedianN, colorSmooth, colorMaxDecay
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Depth frame │
│ │
│ Temporal smoothing │ ← depthMedianN (median ring buffer)
│ → smoothDepth │ or smoothAlpha (EMA)
│ │
│ Spatial box blur │ ← depthBlur (px)
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Background │
│ subtraction │ ← bgThresh (mm), minDepth, maxDepth
│ │
│ pixel foreground │
│ if: captured_bg - │
│ smoothDepth > │
│ bgThresh │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Erode / dilate │ ← erode, dilate
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Temporal mask │
│ hysteresis │ ← maskGrow, maskShrink
│ │
│ Slow the mask to │
│ appear/disappear │
│ to prevent holes │
│ from splitting │
│ blobs │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Color BFS fill │ ← colorFill threshold
│ │
│ Seeds from depth │
│ foreground pixels, │
│ expands to color- │
│ similar neighbours │
│ to merge blobs │
│ split by depth │
│ noise │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Contour finder │ ← minBlobArea, maxBlobArea
│ → blob list │
└──────────┬──────────┘
│
┌───┘ for each blob
│
▼
┌─────────────────────┐
│ blobZ │
│ │
│ Median of │ ← depthPatchR
│ smoothDepth in a │
│ patch around the │
│ blob centroid │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Contour walk │
│ │
│ Deproject each │ ← blobZ used for all points
│ contour pixel to │
│ 3D world space │
│ via camera │
│ intrinsics + pose │
│ │
│ Find contour point │ ← maxLineDist
│ closest to query │
│ line │
│ │
│ Average near- │ ← closestAvgMargin
│ closest candidates │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Inset │ ← insetPixels
│ │
│ Shift closest 2D │
│ point toward blob │
│ centroid, re- │
│ deproject with │
│ blobZ │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Compute result │
│ │
│ dist3D = distance │
│ from blobPt3D to │
│ query line │
│ │
│ height = signed │
│ projection along │
│ line direction │
│ from lineStart │
│ │
│ angleDeg = bearing │
│ around line axis │
│ relative to world │
│ up │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ EMA smoothing │ ← blobSmoothAlpha, distMaxDelta
│ of blob results │
│ over time │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ OSC send │ ← oscMaxRate
│ │
│ /leap/stats │
│ /leap/blob/N/ │
│ position │
│ /leap/blob/N/ │
│ velocity │
└─────────────────────┘
| Key | Action |
|---|---|
b |
Capture background (averages 30 frames) |
s |
Save settings to settings.json |
↑ / ↓ |
Move GUI cursor |
← / → |
Decrease / increase selected parameter |
| Parameter | Default | Description |
|---|---|---|
maxDepth (mm) |
1500 | Pixels deeper than this are discarded. Limits the tracking range. |
minDepth (mm) |
100 | Pixels closer than this are discarded. Removes objects too close to the camera. |
bgThresh (mm) |
30 | A pixel is considered foreground only if it is at least this many mm closer than the captured background. |
smoothAlpha |
0.3 | EMA blend weight for temporal depth smoothing. 1 = no smoothing, lower = more smoothing. |
blobSmooth |
0.5 | EMA blend weight for smoothing blob result values (position, distance) over time. Lower = smoother but more lag. |
erode |
0 | Number of erosion passes applied to the threshold image. Shrinks blobs and removes small noise. |
dilate |
0 | Number of dilation passes applied to the threshold image. Expands blobs; use after erode to fill holes. |
insetPixels |
0 | Shifts the closest contour point this many pixels inward toward the blob centroid before reporting it. Avoids noisy silhouette edge readings. |
contourMaxJump |
0.1 | Rejects a world-space contour point if its distance to both neighbours exceeds this value (metres). Removes depth outliers. |
worldInsetPx |
3 | (Legacy, kept for reference.) EParlier per-point depth inset before the flat-depth approach was adopted. |
minBlobArea |
500 | Minimum blob area in pixels. Blobs smaller than this are ignored. |
maxBlobArea |
200000 | Maximum blob area in pixels. Blobs larger than this are ignored. |
maxLineDist (m) |
0.5 | Only contour points within this distance (metres) of the query line are considered for the closest-point result. Also shown as a band in the world views. |
The query line is defined by two points in world space (metres). The closest blob contour point to this line is computed each frame and sent via OSC.
| Parameter | Description |
|---|---|
start.x/y/z (m) |
World-space position of the line start point. Height along the line is 0 here. |
end.x/y/z (m) |
World-space position of the line end point. |
Defines where the RealSense camera sits in world space. Rotation order is YXZ Euler (degrees).
| Parameter | Description |
|---|---|
cam.pos.x/y/z (m) |
Camera position in world space. |
cam.rot.x/y/z (deg) |
Camera orientation in world space. |
Sent as a bundle once per frame to the configured host/port (default 127.0.0.1:9000).
| Address | Arguments | Description |
|---|---|---|
/leap/stats |
occupation (f), activity (f) |
Fraction of foreground pixels (0–1) and fraction of pixels that changed since last frame (0–1). |
/leap/blob/N/position |
x (f), y (f), z (f) |
World-space position of the closest contour point of blob N to the query line (metres). |
/leap/blob/N/velocity |
x (f), y (f), z (f) |
Frame-to-frame velocity of that point (metres/second). |
Settings are loaded from and saved to data/settings.json. Press s to save the current GUI state.
The background depth image is saved to data/background.bin and loaded automatically on startup.