I love playing around with the LeapMotion. It is a wonderful little piece of technology, has great documentation, and is way ahead of its time.
More specifically, I’m interested in its potential to communicate with MIDI, the protocol which allows software to translate musical data.
A quick aside: My background is in music, having played drums since I was a small fry. Naturally, my instinct was to make air-drumming a reality with the LeapMotion.
I checked out the available options for LeapMotion “drumsets” and found current apps pretty difficult to use. There is a nice air-drumming example built in Python called AirDrum , but I wanted to be able to use more than just a few sounds for my drumset. So, I sought to build my own.
Getting LeapMotion Sensor Data
To get up and running quickly, I used Artoo’s adapter for the LeapMotion . If you haven’t checked out Ruby on Robots from Hybrid Group yet, I would give it a look-see.
The LeapMotion sends data at each frame about every ‘Pointable’ it sees. A Pointable can be a finger or a tool (such as a drumstick or pen). The Pointable has a X, Y and Z position, as well as the velocity in each direction. Check it out:
Creating Drum Surfaces
First off, major props to my teacher and good friend Giles Bowkett for helping me architect this!
We built a Surface class that could take a left and right boundary, as well as a drum note. The basic setup was hi-hat, snare and bass drum:
Next, we used the wonderful UniMIDI library to play the surface sounds via MIDI.
UniMIDI lets you set the volume of each note, so we could play both soft & loud drum hits:
Next came the DrumSet subclass of the Artoo::MainRobot, which created the surfaces at initialization and allowed me to swap in any drum sound I wanted:
Recognizing a Stroke
Once I had my surfaces in place, I needed to capture a single hand stroke.
At first, I attempted to read the velocity at each point in the stroke and determine the surface as a ‘hit’ once the Y velocity changed from positive to negative:
This attempt was a good first stab, but it became very complicated to read the velocity before and after a hit, as well as timestamp each frame of the stroke.
Also, I wanted to account for the dynamics of each hit. The velocity of the stroke is the key ingredient in determining how hard the drum is hit. This could not be captured effectively if the velocity was always so close to 0.
The solution, thanks to some great help from Alex Cruikshank , was to create an imaginary threshold (say, 6 inches from the LeapMotion) and log the previous Y position of each stroke.
This way, a hit was only detected if the current Y position was less than the threshold, and the previous Y position was above it:
Next, I found the ‘hit’ surface in a
drum_for method which compares the x_position to each drum’s boundaries:
I used the velocity of the detected hit to calculate the ‘loudness’ of the stroke.
The MIDI protocol maxes out volume at 100, so the
determine_volume_from function finds a proportional value between 0 – 100 based on the velocity:
The Final Result
With this combination of techniques, I was able to track a single stroke with a pretty reasonable amount of accuracy:
The next challenge for this experiment is tracking the movements of both hands. Pointable objects have ids, but they are reset if the Pointable ever goes out of range.
Luckily, the LeapMotion has a concept of a Hand object, where I can determine the X, Y and Z position of the palms. I gave this a shot in a spike , but it is still quite a WIP
If you’d like to keep track of the project’s progress, you can view it on Github .
I had a lot of fun doing this, and would encourage you to attempt playing around with the LeapMotion. There’s a lot of room for growth in the community, and it’s always a little fun to live in the future.