Vibro-Sense: Robust Vibration-based Impulse Response Localization and Trajectory Tracking for Robotic Hands

Wadhah Zai El Amri   ·   Nicolás Navarro-Guerrero

Paper page can be found here.

Dataset can be found here.

Abstract

Rich contact perception is crucial for robotic manipulation, yet traditional tactile skins remain expensive and complex to integrate. VibroSense offers a scalable and affordable alternative: high-accuracy whole-body touch localization via vibro-acoustic sensing. By equipping a robotic hand with seven low-cost piezoelectric microphones and leveraging an Audio Spectrogram Transformer (AST), VibroSense decodes the vibrational signatures generated during physical interaction.

Extensive evaluation across stationary and dynamic tasks reveals a localization error of under 5 mm in static conditions. The system demonstrates robustness to the robot’s own motion, maintaining effective tracking even during active operation. Analysis highlights the influence of material properties: stiff materials (e.g., metal) excel in impulse response localization, while textured materials (e.g., wood) provide superior friction-based features for trajectory tracking.

Our primary contribution is demonstrating that complex physical contact dynamics can be effectively decoded from simple vibrational signals, offering a viable pathway to widespread, affordable contact perception in robotics. To accelerate research, we provide our full datasets, models, and experimental setups as open-source resources.

Overview

The framework is organized into several Hydra-configured modules for training, evaluation, and inference:

• Localisation: Predicts the contact location from sensor data.
• Material Classification: Identifies the material being touched.
• Quickdraw: Handles trajectory-following task and datasets.
• Noise Separation: Separates touch/contact signals from noise using STFT-domain source separation (see src/noise_separation/README.md for details).

Project Structure

vibrosense/
├── configs/                # Hydra configs for all modules
│   ├── localisation/
│   ├── material/
│   ├── noise_separation/
│   └── quickdraw/
├── src/
│   ├── dataloaders/        # Data loading utilities for images and spectrograms
│   ├── models/             # Model definitions (AST, ResNet, ViT, etc.)
│   ├── noise_separation/   # Noise separation module (see sub-README)
│   ├── utils/              # DSP, demo scripts, and helpers
│   ├── train_network_localisation.py
│   ├── train_network_material.py
│   └── train_network_quickdraw.py
├── environment.yml         # Conda environment
├── requirements.txt        # Python dependencies
└── README.md               # Project overview (this file)

Key Modules

Localisation

Predicts the location of contact on a the hand surface using deep learning models. Configurable via configs/localisation/. Training entrypoint: src/train_network_localisation.py.

Material Classification

Classifies the material type (e.g., wood, metal, plastic) from vibro-acoustic localisation data. Configurable via configs/material/. Training entrypoint: src/train_network_material.py.

Quickdraw

Handles trajectory-following task and datasets. Configurable via configs/quickdraw/. Training entrypoint: src/train_network_quickdraw.py.

Noise Separation

See src/noise_separation/README.md for a detailed description. Implements STFT-domain source separation using models such as masking transformers and TF-Locoformer.

Configuration

All modules use Hydra for configuration management. Default configs are under configs/, and can be composed or overridden for custom experiments.

Installation

Install dependencies using Conda and pip:

conda env create -f environment.yml
conda activate vibrosense
pip install -r requirements.txt

Training & Evaluation

Each module has its own training script (see src/). Example for localisation:

python src/train_network_localisation.py

Adjust configs as needed using Hydra's command-line overrides or by editing the YAML files in configs/.

Developer Notes

• Modular design: Add new models or datasets by extending the relevant submodules.
• Logging is prioritized for debugging and reproducibility.
• See submodule READMEs (e.g., src/noise_separation/README.md) for more details on specific components.

Last Updated: 29.04.2026

Citation

@InProceedings{ZaiElAmri2026VibroSense,
  author = {Zai El Amri, Wadhah and {Navarro-Guerrero}, Nicol{\'a}s},
  title = {"Vibro-Sense: Robust Vibration-based Impulse Response Localization and Trajectory Tracking for Robotic Hands"},
  booktitle = {"ArXiv Preprint arXiv:2601.20555"},
  year={2026},
}