These strange glyphs represent the dictionary of hand gestures Geco and LEAP can turn into music control.
Here we go again. Touchless hand gestures have been part of electronic musical performance ever since the Theremin first hummed to life almost 100 years ago. And those gestures embody the same challenges and promise. We have the ability as humans to think spatially, in three dimensions, and to have a tight sense of control via our muscles of gestures in space. We use gestures to communicate and to manipulate our world. Those same expectations can be disappointed in electronic systems, however, as they lack tangible physical feedback and may misinterpret our intentions.
It’s easier to play with these ideas and experiment with them than talk about them, though. And for everyone who’s turned off by the idea, someone else is enthused.
What the US$79.99 Leap Motion may do for gestures in music is to lower the bar for entry – and up the bar for performance. Leap is affordable hardware, there are already lots of developer units out in the world, and there’s an easy-to-program SDK. We’ve already seen Microsoft’s Kinect open up gestural control to lots of new music projects. Leap may do more: it’s cheaper, it’s faster and operates with vastly lower latency, and it’s more precise for individual hand gestures. It also offers a platform for developers to share their work, in an app store full of stuff you can use, so that the hardware theoretically won’t become a paperweight in the cubicles of the digerati.
Latency alone could make a big difference for musical applications. It’s not the only challenge in motion control, but it has been the showstopper, particularly with the hefty lag you get using something like Kinect. Leap is different, offering latencies low enough to satisfyingly control music applications.
The unobtrusive Leap Motion hardware. Courtesy the manufacturer.
That doesn’t mean you should run out and buy one. Healthy skepticism is always good practice. So, I actually agree with some of Chris Randall’s complaints about Leap, as discussed on Twitter. I think anyone experimenting with novel control schemes, though, may learn something from successes and failures alike.
If you’re ready for the adventure, though, Leap will make it immediately easy to start mucking about with music. Leading the charge is Geert Bevin and his Geco (originally Gesture Control) app. I’m testing it now, but here’s a quick look at what it does.
By making a virtual MIDI port, and using a library of gestures and mappings, Geco allows a wave of your hand to control any music tool that works with MIDI.
- Using two hands, create up to 40 different streams of control messages.
- 16 MIDI channels.
- Mappings with MIDI Control Change or (with greater data precision) pitch bend.
- Manipulate different streams using “open” and “close” gestures of your hands.
- Low-latency control, with visual feedback on both MIDI and movement analysis.
- Send MIDI on the Mac using a virtual MIDI port you can then connect to other applications – or, on either platform, to physical MIDI ports.
- Graphical UI with color/graphical customization, information on gestures and so on.
- Thin out your MIDI data to work with old gear that can’t respond to all those messages.
The intro price will be US$9.99. It should launch with the Leap Motion app store – dubbed Airspace – when the controller launches on May 13.
MIDI is useful, but it’s too bad there’s no higher-precision control implementation here. (OSC would be one option; it seems apps that do that are a likely addition.) There is a whole lot of detail and thought that has gone into how the UI works, and Geert promises that the whole engine is low on system resources and approaches “zero latency” (at least, it’s very, very fast).
Updated: Geert fills us in on that high-resolution data question and OSC. From comments:
Yes, there will be OSC in a next version and I plan to add direct hosting of AU/VST also. I’m also thinking of making an AU/VST version of Geco itself so that it can perfectly be integrated into any DAW and process the audio that’s flowing through.
It’s worth taking a look at the draft documentation for more detail:
Here’s another experiment showing VST and AU control:
Nor is Leap Motion the only game in town. On Create Digital Motion yesterday, I wrote about another project that is using crowd funding to launch an open source rival. I can imagine developer APIs that let you work across each. The advantage of open hardware would be that people can understand how the device works, and modify it for specific applications (both code and hardware form factor.)
DUO is a DIY 3D Sensor – Like Leap, But Open Source, From Gesture and Vision Veterans
I’m clarifying the details of their licensing plan. At least one of this team has come under criticism in the past for the approach to open source releases and Kinect hacking – you can read the discussion in both directions, though I’m encouraged that developer AlexP was ultimately responsive to some of those concerns. We’ll see how this project is structured.
It does seem that people will continue to develop this thread in motion control. We’ll be watching.
As I do have a Leap, let me know if there’s anything you’d like tested or developed (summer project!), or questions you may have.
https://www.leapmotion.com