Document Audio_ML repository structure and architecture

[Copilot]

Repository analysis per user request to understand the Audio_ML codebase.

Repository Overview

Embedded audio classification system for STM32F746G-Discovery board implementing real-time speaker recognition pipeline: audio capture → MFCC extraction → quantized neural network inference → LCD display.

Key Components Documented

• Embedded C Application (~7.3K LOC)

  • Custom MFCC implementation with naive O(N²) DFT (candidate for CMSIS-DSP optimization)
  • FreeRTOS-based audio recording/playback
  • Display and serial communication interfaces

• AI Model (X-CUBE-AI)

  • Quantized int8 TFLite: 40-dim input → 100→200→100→5 dense layers
  • 45KB weights, 2.5KB activations
  • Input quantization: scale=2.787, zero_point=99

• Dataset: 416MB WAV files across 5 speaker classes

• Python Training Pipeline: Jupyter notebooks for model training and audio preprocessing

Architecture

• Target: STM32F746NGH6 (Cortex-M7)
• Build: STM32CubeIDE with .ioc configuration
• Total: 1.6GB, 3,643 files
• License: STMicroelectronics SLA0044

Original prompt

read my repo

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[MehediEEE45]

@copilot # 🎤 Audio Classification on STM32F746G-DISCO

Real-Time Speaker Identification using MFCC + TensorFlow Lite Micro

Author: Mehedi
Group: ODD
Course: EEE 322 — Digital Signal Processing I Lab
Department: Electrical & Electronic Engineering, SUST
Submission Date: 9 December 2025

---

📌 Project Overview

This project implements a real-time speaker recognition system on the STM32F746G-DISCO board.
Speech is recorded using the on-board MEMS microphone, processed using DSP techniques (MFCC), and classified using a TensorFlow Lite Micro (TFLM) neural network deployed on the microcontroller.

🔥 Key Features

• Real-time audio capture using WM8994 codec
• PDM → PCM conversion + DMA
• MFCC extraction (40-coefficient version)
• Neural network model trained in Python (Keras)
• Quantized TFLite model for embedded inference
• Speaker name displayed on the TFT LCD
• Dataset: 2595 audio clips (1-second each) from 5 speakers

---

🛠️ Hardware Used

• STM32F746G-DISCO (Cortex-M7 @ 216 MHz, 1MB Flash, 320 KB SRAM)
• MP34DT01 MEMS microphone
• WM8994 Audio codec (ADC/DAC)
• 4.3" TFT LCD
• SD card (optional for storage)
• ST-Link for flashing & debugging

---

🧰 Software & Libraries

Component	Version
STM32CubeIDE	v1.19
STM32CubeF7 HAL	v1.26.0
Python	3.8+
Librosa	v0.10.0
NumPy	v1.25.0
TensorFlow	2.15.0
TensorFlow Lite Micro	Built manually
CMSIS-DSP	Optional
Jupyter Notebook	v6.5+

---

📂 Project Structure

.
├── README.md
├── dataset/
│ ├── fa_him/
│ ├── imran/
│ ├── nayeem/
│ ├── shahed/
│ └── talukder/
├── python/
│ ├── split_audio.py
│ ├── make_metadata.py
│ ├── extract_mfcc.py
│ ├── train_model.py
│ ├── convert_to_tflite.py
│ └── ai_test.py
├── stm32/
│ ├── Core/
│ ├── Drivers/
│ ├── Middlewares/
│ └── xx_model_data.cc
└── report.tex

yaml
Copy code

---

🎙️ Dataset Collection

• 10 minutes of speech collected per speaker
• Converted to WAV format
• Split into 1-second clips using Python (pydub)
• Total clips: 2595
• Structured as:
  dataset/
  ├── fa_him/
  ├── imran/
  ├── nayeem/
  ├── shahed/
  ├── talukder/

pgsql
Copy code

🔧 Split Audio into 1-Second Clips

from pydub import AudioSegment
import math, os

audio = AudioSegment.from_file("fa him.wav")
chunk_length_ms = 1000
os.makedirs("fa_him", exist_ok=True)

for i in range(math.ceil(len(audio)/chunk_length_ms)):
    start = i * chunk_length_ms
    end = start + chunk_length_ms
    audio[start:end].export(f"fa_him/clip_{i+1:03}.wav", format="wav")
🎵 MFCC Feature Extraction
Extracted 40 MFCC coefficients per clip

Used librosa.feature.mfcc()

MFCC extraction pipeline:

Pre-emphasis

Framing

Hamming window

FFT

Mel filter banks

Log energy

DCT → MFCC

🤖 Model Training (Python)
Neural Network Architecture
text
Copy code
Input: 40 MFCC features
Layer 1: Dense(100) + ReLU + Dropout(0.20)
Layer 2: Dense(200) + ReLU + Dropout(0.20)
Layer 3: Dense(100) + ReLU + Dropout(0.20)
Output: Softmax (5 classes)
Training Script (simplified)
python
Copy code
model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

model.fit(X_train, y_train, epochs=100, batch_size=32)
model.save("audio_classification_100.keras")
Achieved Accuracy
Type	Accuracy
Offline training	99.8%
On-device	~70%

🧪 Converting Keras Model → TFLite Micro
Int8 Quantization
python
Copy code
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = rep_dataset_gen
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model = converter.convert()
open("audio_class_quant.tflite", "wb").write(tflite_model)
TFLite Model Size
Copy code
audio_class_quant.tflite → 57.3 KB
Converted to C array:

Copy code
xx_model_data.cc
🧵 FreeRTOS Integration
Major Tasks
record.c → audio recording from WM8994

mfcc.c → MFCC extraction on MCU

ai_on() → run inference

display.c → show predicted speaker on LCD

role.c → application state machine

System Workflow
css
Copy code
Microphone → WM8994 → PCM → MFCC → NN Model → LCD Display
📊 Results
Metric	Value
Real-time latency	~300 ms
Embedded accuracy	~70%
Max distance tested	~1 meter

Challenges
Background noise

Inconsistent speech loudness

RAM limitations for MFCC buffer + TFLite model

📸 Project Output
(Add images to docs/ and link here)

scss
Copy code
![Output 1](docs/output.jpg)
![Output 2](docs/1.jpg)
![Output 3](docs/2.jpg)
![Output 4](docs/3.jpg)
📎 References
STM32F746G-DISCO Reference Manual

TensorFlow Lite Micro Documentation

CMSIS-DSP

STM32CubeIDE, STM32CubeMX

📬 Contact
For contributions, issues, or discussions, feel free to open a GitHub issu