Today, I’m in London doing a hands-on workshop on visual metaphors for music, and covering various topics filed under “synesthesia” at Music Tech Fest. It seems appropriate, with the subject matter on the brain, to revisit the topic of visuals and music in a series of posts.
When you make hardware, with knobs and faders, you’re constrained by physical space – the amount of room on a circuit board, the radius of a knob cap, the size of your fingers. But before you get there, the first step is to sketch an idea. Imagine if you could do that with a marker and paper and ink.
SketchSynth is the latest attempt at a drawable set of controls, letting you turn an illustration on paper into something you can actually use to make music. It’s not the first – this dream of being able to make things come alive with nothing more than a magical pen is an old one – but the execution looks superb. There are two basic approaches to the idea: one is to use some sort of conductive ink to turn the drawing itself into a sensor, and the other is to point a camera at the drawing and calculate where a user makes contact with the drawing. SketchSynth opts for the computer vision approach, by way of OpenFrameworks and the old standby of free and open source vision, OpenCV. (Kyle McDonald’s ofxCV does the heavy lifting.)
Conventional vision fits the task well: faders and knobs respond as expected, even though they’re only ink on paper. While it’s a drawn interface, and could look like anything, the behavioral metaphors all come from hardware: there are sliders, momentary buttons, and pots. Place the tool in “edit” mode, and the computer analyzes what you’re drawing; in “play” mode, the camera tracks your finger as you manipulate what you’ve drawn. The project goes one step further than many that have come before, by overlaying a projection calibrated with your drawing for interactive visual feedback as well as sonic. Sound in this case is provided by Pd, but OpenSoundControl (OSC) messages let you connect to other musical (or visual) tools. See more in the making-of vid, at bottom.
Creator Billy Keyes is working with the right mentor, too, completing this as research for Golan Levin, who has long explored the relationship of drawing and musical gesture. His Sonic Wire Sculptor was a seminal creation in connecting drawings to sound, using a tablet to produce three-dimensional “wire” structures and corresponding sound synthesis. His Messe di Voce performance piece neatly reversed the relationship, using the voice as the input to animate drawings and illustrations. More of Levin’s work at flong.com. See also: composer Xenakis’ UPIC, which lives on today as IanniX, a tool getting a lot of development attention.
Keyes’ project, already getting lots of blog buzz, is notable for its practicality and immediacy; it seems a tool many others might use and build upon rather than a single piece of art.
It’s lovely to see projects, particularly academic projects, come to some form of completion and clarity. Speaking of completion and clarity, at some point a proper survey of drawn musical interfaces seems a must, but that will have to come another day. Where can I get a nice full English breakfast?
I’m late to the party, but hat tip in particular to Creative Applications
Project Page at golancourses.net; Linux source is promised soon
I’ve covered a number of these sorts of projects over recent years.
Paper, Drawing as Musical Controller: A Round-Up (including a number of paper examples)
Drawing Sound: Crazy Touch Interface Sound Experiments with Usine, PC (using only a screen – and thus producing a very different experience of drawing
Imaginary Instruments: Marker and Paper as Controller (this 2009 project is almost a direct analog to SketchSynth, minus the projection)
iPhones, Pencils: Hand-Drawn Music Interactions, Tokyo Subway Mobile Jam (this one isn’t quite the same as the others, using pencil and paper to design an interface for a screen – but it’s nice to recall that you can do that, as well)
Pen and Paper as Graphical, Digital Music Score (from earlier this year, a more open-ended design)
Video: Sonic Wire Sculptor, 2003: