ESP32 Wi-Fi CSI + Audio Streaming Pipeline

Real-time telemetry pipeline that streams two sensor modalities — Wi-Fi Channel State Information (CSI) and 16 kHz PCM audio — from an ESP32-S3 microcontroller, through a Raspberry Pi ingestion server, to a browser-based visualization and recording client. The CSI stream additionally drives a live presence/motion detector based on per-subcarrier background subtraction and an adaptive noise floor.

A full technical write-up (architecture, wire format, backpressure policies, and a debugging case study of a silently corrupted CSI source) is in docs/esp32-stream.tex.

Architecture

ESP32-S3 (C++ / FreeRTOS)          Raspberry Pi (Python)            Browser (JS)
┌──────────────────────┐        ┌───────────────────────┐        ┌──────────────────┐
│ INMP441 mic ── I2S   │ UDP    │ aiohttp + asyncio     │  WS    │ Canvas waterfall │
│ audio task (core 1)  │ :5006  │ validate + fan-out    │ /ws/*  │ motion detector  │
│ CSI callback (core 0)│ :5005  │ hub, WAV recorder,    │        │ WebAudio player  │
│ sender task (core 0) │◄:5008──│ keepalive control loop│        │ record UI        │
└──────────────────────┘        └───────────────────────┘        └──────────────────┘

• Edge — acquisition only: samples are framed into compact binary datagrams (20-byte little-endian CSI header, magic 0xC5110001, sequence numbers for loss measurement) and sent over UDP. Freshness is preferred over completeness: no retransmission, drop-at-the-producer queues.
• Ingest — a single-process asyncio server: UDP ingestion with magic-number gating, a bounded drop-oldest pub/sub hub, server-side WAV recording with a REST API, and a demand-driven keepalive loop that generates the CSI stream (the ESP computes CSI only for frames it receives).
• Client — all DSP in the browser: zero-copy binary parsing, per-subcarrier amplitude extraction, background-subtracted waterfall (viridis), and an asymmetric-EMA motion detector.

Repository layout

Path	Contents
src/main.cpp	ESP32-S3 firmware: I2S audio capture + CSI acquisition (Arduino-ESP32 3.x / IDF 5.5)
webapp/server/	aiohttp ingest server (hub, recorder, keepalive, UDP ingestion)
webapp/static/	browser client (vanilla JS, no framework)
webapp/tests/	pytest suite (hub semantics, frame gating, recorder lifecycle, keepalive)
tools/	offline CSI data validators (pcap statistical analysis)
docs/	LaTeX source of the technical report

Firmware

1. Copy include/secrets.h.example to include/secrets.h and fill in your Wi-Fi credentials and ingest server IP.

2. Build and flash with PlatformIO:

   pio run -t upload && pio device monitor
   

Hardware: ESP32-S3 devkit + INMP441 I2S microphone (WS=GPIO6, SCK=GPIO5, SD=GPIO4). The platform is pinned to Arduino-ESP32 3.x (IDF 5.5) because the older 2.x core returned a stub CSI buffer on the S3.

Server

See webapp/README.md — run locally with two commands, or deploy to a Raspberry Pi as a systemd unit.

Data validation tools

tools/csi_analyze.py tests the statistics of a captured CSI stream rather than its syntax (per-subcarrier temporal variance, unique-frame count, RSSI spread) and ends with a LIVE / STILL FROZEN verdict — built while debugging a CSI source that produced syntactically valid but physically meaningless data. tools/csi_raw.py dumps raw frames from the same pcap captures.

Per-stream characteristics

Stream	Rate	Datagram	Bandwidth	Loss policy
Audio (PCM)	31.25 pkt/s	1024 B	32 kB/s	send-and-forget, TX backoff
CSI	~50 frames/s	148–276 B	~10–14 kB/s	queue drop (16) + hub drop (32)
Keepalive	50 pkt/s (on demand)	1 B	negligible	none needed