ML Project: Real-time Gesture Recognition

This project turns a Nintendo Wii Remote into a real-time gesture controller. Hold the trigger, draw a shape in the air, and a neural network recognises it and sends it to any program listening on the network.

Features

• Streams raw 6-axis IMU data (accelerometer and gyroscope) from a Wii Remote over Bluetooth.
• Two modes: Data Collection, which records and labels gestures, and Inference, which predicts them live.
• Recognises 9 gestures out of the box: bolt, circle, m, swipe up, down, left and right, trash, and z.
• Only acts on predictions above 85% confidence, and broadcasts them to other applications over a local UDP socket.

How it works

1. Connect and calibrate. The remote connects through my Wii Remote handler, flashes an LED to confirm the link, and turns on the Motion Plus gyroscope. An optional guided calibration finds the sensors’ zero offsets.
2. Capture. Holding the B trigger records a gesture, and letting go ends it. Each sample holds acceleration on three axes, roll, pitch and yaw, and the time since the last sample.
3. Collect data. In Data Collection mode, each gesture is labelled in the console and saved as JSON, which builds the training set.
4. Train. An LSTM network learns from the recorded sequences, using data augmentation and K-fold cross-validation.
5. Predict. Gestures take different amounts of time, so each sequence is timestamped relative to its start and padded to a fixed length before it goes into the model.
6. Broadcast. A recognised gesture is sent to 127.0.0.1:4242 over UDP, so a game or UI in any language can react to it.