This project presents a wearable embedded system designed to assist deaf and hard-of-hearing individuals in detecting critical sound-based events in their environment.
The solution combines embedded systems, real-time operating systems (RTOS), and cloud-based machine learning inference to provide reliable hazard detection and user feedback.
The system focuses especially on industrial and urban safety scenarios, where sound awareness is crucial.
- ~865,000 Brazilians with hearing impairment in working age
- Only ~8% are formally employed
- Industrial environments present additional safety challenges
This project explores how AI + Embedded Systems can bridge this gap through assistive technology.
- Lightweight embedded device
- Audio processing + transmission
- Cloud-based neural network inference
- Fast feedback to the user
- Captures environmental audio
- Performs signal conditioning
- Sends data to the server via WiFi
- Receives classification results
- Notifies the user
- Receives audio samples
- Applies signal processing pipeline
- Extracts features (MFCC)
- Performs classification using CNN
- Returns hazard label
- Microcontroller: ESP32-WROOM
- Microphone: INMP441 (I2S MEMS)
- Display: SSD1306 OLED (I2C)
- Actuator: Vibration Motor
- Connectivity: WiFi
The firmware is structured using FreeRTOS, enabling modular and concurrent execution:
- Audio Capture Task
- Signal Processing Task
- Communication Task (HTTP/MQTT)
- Order: 100
- Window: Hamming
- Noise reduction
- MFCC (Mel Frequency Cepstral Coefficients)
- Efficient representation for classification
- Model: CNN
- Dataset: UrbanSound8K
- 80 epochs
- Categorical Cross-Entropy
- Adam optimizer
- Vibration motor → tactile alert
- OLED display → hazard identification
- Audio capture
- Transmission to server
- Signal processing + MFCC
- CNN classification
- Response to device
- User alert
- Network dependency
- Latency
- Security considerations
- Trade-off vs TinyML
- Edge inference (TinyML)
- Hybrid architecture
- Latency optimization
- Industrial validation
- Author: Paulo H. Langone Miranda
- Email: phlangone@gmail.com
- GitHub: https://github.com/phlangone
Educational and research purposes