Multitouch Prototype 2 from Randy Jones on Vimeo.

2008 has been an amazing year for music technology. But I can’t bring myself to look back on it on this New Year’s Eve: not when there’s so much to look forward to in 2009. Case in point? An extraordinary, innovative new controller that in a matter of hours was already spreading among connected music technologists around the planet.

At the end of the day, it’s not hard to describe what you might want out of an expressive music controller. Most people would agree on that. The challenge is really an engineering problem. Solve the engineering problem in an artful way, and you can spend the rest of your time just practicing playing your invention. That’s what makes the above video so exciting.

Randall Jones has built a really elegant and wonderful multi-touch hardware controller, as reported by MAKE:blog (and picked up on Hack a Day). With $50 in parts and a lot of clever hardware design and software coding, Jones has built an interface that responds to both touch and pressure and, using some smart sonic mapping, can realistically reproduce instruments like the dumbek and guiro.

Intimate Control for Physical Modeling Synthesis [Project Page / Paper Abstract]

PDF, Randall Jones MSc Research Paper

Who needs a “top 10 technologies of 2008” post for CDM when this particular instrument could pretty easily top the whole list? Let’s just call it done, and uncork the champagne: major congrats, Randy! (This is a master’s thesis!)

Jones’ work does have some precedent, but just to review how much he’s accomplished here: he’s innovated in terms of the sensing, the form factor, the software interpolation, and the way in which the control data is mapped to a synthesis method. (Whew!) That has had a number of specific achievements:

  • A clever form factor: The basic design here is elegant and could be adaptable to other form factors. Most importantly, the use of polypropylene and silicone rubber means the interface deforms nicely as you press it, giving you feedback. And that opens lots of other similar design possibilities. Translation: being squishy rocks.
  • Multidimensional / force-sensitive input: “Multi-touch” usually only means something that can take multiple touch inputs at one time, as in, from multiple fingers. But as I’ve complained in the past, the problem is that most multi-touch interfaces, like the Lemur and iPhone, don’t respond to the amount of pressure or deform (providing tactile feedback) when you use them. That makes them feel a bit like an ATM screen that happens to take more than one finger at a time. Jones’ prototype responds to how hard you’re pressing or hitting it, and it’s more sensitive – more like a real-world instrument. Jones calls this “multidimensional,” but force-sensitive would be another way to look at it. Translation: it’s more organic than entering your ATM PIN code.
  • Audio-rate resolution: By choosing to use audio signal for control rather than … well, anything else, Randy gets an extremely responsive control signal. The controller itself is passive, requiring no power. Everything is done by processing audio created by the sensors. That’s not a new idea, but by returning to it in this context, Randy makes a much more responsive controller than most touch and multi-touch controllers before it, and returns to some of the analog-style control of instruments like the Theremin and acoustic instruments. Translation: it don’t mean a thing if it ain’t got that swing.
  • Smart centroid processing: To make touch points accurate, Randy wrote a new object for Jitter that does centroid calculation. Translation: Randy worked through the details.
  • Waveguide mesh modeling for deep sonic results: None of this would be meaningful without meaningful sounds coming out of it. Here, Randy builds on previous work in creating a physical model of a drum head (back to the dumbek), but both refines the model and works intelligently through how to match it with the controller. Translation: it makes good noises.

There are some other related touch devices, each with their own strengths and weaknesses, but rather than reinvent his work, I’d suggest reading through Randy’s own research paper. He does a great job talking about what’s great and not-as-great about other research and products, and makes a terrific argument for his design choices. (Basically, see also the Haken Continuum, the Korg WaveDrum, the Tactex MTC Express, the CNMAT multitouch controller, and the Audio-Input Radio Drum.)

Now, if this kind of development had come about a few years ago, the next thing we’d be talking about is how this might be commercialized. Part cost is cheap – that’s good news. I don’t know Randy’s plans for licensing and future development. But we can already compare how this might have gone had this been introduced, say, a few years ago.

The old model:

Ah, there’s nothing like the NAMM trade show. Hey, did I forget to make my travel reservations this year? Shame. Photo: Buzz Andersen.

1. Spend a couple of years more refining the prototype on your own.

2. Show off the work at an academic conference in Italy. People can see it if they’re associated with an academic institution that’s also willing to pick up their airfare and let them leave for a week. (Anyone who’s had to wrestle with their department to get a couple hundred dollars for a thousand dollars in actual expenses knows what I’m talking about here.)

3. Publish in an academic journal a handful of people get, so they can try to figure out what it’s like from diagrams and grainy black-and-white photos (and no sound). Oh yeah – no one can actually comment on the story, either, so advanced researchers lack feedback and newcomers lack context and commentary.

4. Get the prototype ready for NAMM. Blow a bunch of cash on a hotel in Anaheim and a NAMM booth.

5. Work out a complex, expensive production and distributi
on scheme that in turn jacks up the price. Hope someone finds it and can afford it.

6. To justify the cost, make the product as finished as possible. Sell it as a “Digital Dumbek” to eccentric, loaded rock stars.

Now, I don’t want to sound grumpy or naive. The truth is, some really brilliant instruments have been down this road. But I’m not sure the description above really did a whole lot for their brilliance. I’m not necessarily saying there aren’t benefits to some of the above techniques, but clearly they won’t work for everyone.

The new model:

The way we prefer to discover new technology: get together with friends and play, in the real world (as at Handmade Music, here at Etsy with the help of the Make folks) or, increasingly, I hope, virtually.

“Where’s my flying car?” That’s the question that often comes up when technology and a New Year coincide. With music, though, I think what we’ve really been waiting for is a convergence of new technology and new communities. Flip through the Computer Music Tutorial’s section on new instruments, and you’ll see some fantastic, exotic, and often familiar new interfaces. So what has 2009 got that the last few decades didn’t? Think cut-rate, faster tech, and a connected Web community to develop, evangelize, and use new technologies.

1. Get 20,000 Vimeo views the moment you post your first demo. No one has to fly anywhere.

2. Connect with people doing research and experiments worldwide and get instant feedback and ideas and potential collaboration – even if they’re not publishing research or flying to conferences or are even academics.

3. Instead of assuming you’ll build a finished, closed product (hello, “Digital Dumbek”), figure people will want to hack the result. (Randy has already posted parts lists, so it’s possible to build this thing right now if you’re so inclined. And he also says in Vimeo contents he’s thinking about doing a version that transmits OpenSoundControl data, so you could use it to control other instruments, music, or even visuals.)

You’ve already skipped a lot of the steps that were previously necessary just to find interested parties. And that network could in turn be used to figure out how to make the thing around the world at a lower cost and get it directly to people who might want to buy it. It also assumes those people might hack the tool for very different applications, instead of having to target one particular kind of person.

Nothing about this is a panacea for innovation: most of the hard engineering problems remain, and this doesn’t mean you can magically create new products. But there’s no question that even changing the hurdles means there’s new potential. If 2009 is about anything, I hope it’s about people finding new solutions to taking that potential and tapping into it. You can bet it’ll be a major focus of this site.

Congrats, Randy – we’ll be watching.

Happy New Year.

  • I only read CDM for the articles. 😉

  • Brilliant. All that's needed to make this a hotcake-like product is a stripped-down, multichannel USB or Firewire interface that handles the analog IO. An enterprising hacker could even move all the tone generation and sampling and centroid detection into a tiny, dedicated hardware module so the computer doesn't have to bother itself with that.

  • Capacitive sensor-based interfaces always get me going. Great coverage of a wonderful master's thesis and potential product. I really appreciate the outline of the "old" vs. "new" approaches to interface design. A few thoughts:

    1 The software is a huge part of this. As a percussionist/hand drummer, I'm really excited to see the physical modeling being used in combination with this system.

    2 The fact that the hardware is passive is so cool. The only problem is that people don't want to use their audio interfaces as control interfaces. Unless… there was a trend in this direction of using cheap, passive objects for control, offloading processing onto the software and audio interface. The same way some people buy computers (expensive) and then VSTs (potentially cheaper), instead of buying lots of audio hardware (mid-range).

    3 It's great he's using a capacitive force sensing matrix, because FSRs would be unreasonable if bought individually: you could hack apart four 16-pad control interfaces for about $600 (I'm thinking MPD-16s or Trigger Fingers), or FSRs bought individually at $350 (you'd still need the communication interface).

    4 Doing DSP on 8 inputs in hardware and communicating via USB shouldn't be too difficult/expensive, though it does go slightly beyond Arduino-level microcontroller + electronics knowledge. This device could be made standalone for less than $100, probably less than $50 en masse. And I doubt extending it to a 16×16 grid would do much to the price.

    I think two things would be magnificent to see happen here:

    1 A mass-produced standalone that's really durable, for $100-$150 each.

    2 An intermediate-level solution to the DSP in hardware that preserves the low latency and high resolution, but doesn't require an expensive audio interface.

    Sorry, I've been thinking about this thing a bit recently 🙂

  • poorsod

    This is really neat! The drawbacks to the design of this prototype that strike me are: your touches are quantised into only 64 areas (8*8 grid of capacitors), and it takes up a whole interface of its own. It's also not a screen, which does put it significantly behind the Lemur and iPhone in my opinion.

    But as you say that's not really the point, I'm sure these problems are by no means insurmountable and someone will definitely hack them in.

  • Wow. Thanks for the good words, Peter.

    Hans: thanks, that's the plan.

    Hi again Kyle. I'm interested to hear your comments from a hand drumming perspective. I think that it might make sense to have a drum-oriented version, round and super durable, and a more keyboard-like one, rectangular and thinner. They could share the same control board, so making two designs might not be out of the question. If you want, email me your wish list for features!

    I have a hard time imagining the passive-only version making it on its own because, as you say, not too many people have a spare 8×8 interface they can use. But it's possible that the standalone product succeeding could allow various parts to be made available to DIYers pretty inexpensively.

    The monome has definitely been a big inspiration in terms of open hardware and local manufacture. I would look to their products for some idea of what prices could feasibly be. I'm committed to as much local manufacturing as I can do, and more interested in quality and durability than mass-anything. At the end of the day, this is still a niche product.

  • I'm impressed!

    And I'm hoping this can all be done with a relatively inexpensive USB device… something to look forward to. I like hand percussion and computer music and have yet to really be able to combine them well, but this would do the trick. 🙂

  • "There are some other related touch devices, each with their own strengths and weaknesses, but rather than reinvent his work, I’d suggest reading through Randy’s own research paper."

    Do you have a link?

  • Fark me.

    Thats an impressive coming together of ideas. What's the latency on this thing?

    maybe some LEDs could be embedded for visual feedback, and some basic haptics (via a speaker) might also add some useful feedback.

  • Magnus Olofsson

    "your touches are quantised into only 64 areas (8*8 grid of capacitors)"

    Actually it is not. It interpolates between the grids so it has a much higher resolution than that.

    This is cooler than sliced bread.

  • Roald Baudoux

    You forgot to include the costs of the FireFace 800 in the price (makes it much much much higher).

    So pressure-sensitive audiorate multi-touch tracking is sure nice but no lie about the real price please!

  • WaaaaahOOOOOOOOOWWW, man.
    It's this mad scientist stuff that I love CDM for.
    Man that's awesome and so damn clever!

    What really makes it useful IMO is the sample rate. Trying to find a drum pad type interface that can really capture the subtleties of complicated percussion parts (like the hand drum example) is difficult.

    I'm no engineer, but my guess is that you're limited by the bandwidth of USB, and the resolution of MIDI. This thing bypasses both, and damn that's cool.

    I wish I had the background to be able to both design my own controller, and write the software algorithms to drive it, and play the thing … man, that dude's a genius!

  • Randy, I'll keep thinking about this and shoot you a wishlist 🙂

    Shannon, here (pdf) is his paper.

    Keith, the latency should be based on the audio interface (5-10 ms?) plus the FFT framing and processing time (another 10-20 ms?). "some basic haptics…" I think that's exactly what isn't needed here, as any interface that deforms and is "intimately" responsive will make a lot of "haptic sense" already!

  • BRAVA! *standing ovation*

    At last, a contender and feat of engineering that will bring expressive multitouch controllerism to the massess.

    Dr Lippold Haken, Will your Continuum Fingerboard get priced out of the market if this develops further? Consider a collaboration with Randall Jones? This technology needs appropriate marketing and distrobution channels but has the affordability to become mainstream. Yamaha may be gutsy enough to take this one on too..

    Anyway, spooling off the top of my head there. The best news for 2009 that I could have wished for!

  • That a really cool toy. How do you learn to make stuff like that?

  • vinayk

    I assume you could do a similar thing for this device over a USB connection instead of the audio interface?

  • matt holland

    this is so fake.

  • @Roald: the passive parts are indeed $50 … but even 8-channel audio isn't all that expensive these days.

    Anyway, yes, it should absolutely be possible to do audio over USB, at least theoretically. That's not an easy thing to do, necessarily, but it is doable. In fact, the Kore controller uses the USB audio stream to transmit control data.


  • Damon

    "Intimate Control: Multi-Touch, New Models, and What 2009 is Really About"

    Reminds me of one of those 60's style guru of sexual enlightenment "let me teach you in the way's of love and knowledge and consciousness" pick up lines.

  • Ali

    This is amazing! Bummer that it uses up 8 audio inputs, but still extremely cool. Definitely blows the socks off my MTmini project (using it + tbeta + processing to make a multitouch midi controller). Well done!

  • @Damon:
    With this crowd, I think the line is:
    "You had me at waveguide mesh."

  • bliss

    On my sound card, I have at least 10 audio inputs that are unused. So I definitely wouldn't have a problem with Randy's prototype.

  • bliss

    But, yes, that is amazing work! Where can I buy one?

    Cats are the secret to everything good.

  • bliss

    How's this for a solution to multi-touch shortcomings? Apple files for patent on winter-friendly iPhone gloves


  • Pingback: randform » Blog Archive » Multi-Touch for (and through) audio()

  • gbsr

    holy crap thats sexy.
    im a percussionist amongst other things myself, both handdrumming and otherwise and man i soo want one. mm, complete with pressure and everything. who cares if it has screens or not? im more concerned of the whole audio interface part myself. nonetheless, i so want one.

  • mitch

    this is preciely the type of innovation that keeps me coming back to this site..

    many kudos randy! i am excited to see how this further develops.

    is there any possibility in making this two sided and having the interface read the negatives?

  • cinco

    Maybe you could split the (interface) difference and have active electronics for the oscillators and free up the outputs of the interface. Probably plenty of people have the inputs free (in a live context) but couldn't sacrifice all 8 for output. At any rate, good job, glad to see more work on multitouch.

  • @Randy niche product?!? more like the freakin' future

    @Peter "You have me at…" LMAO

    I want one too.

  • Superb. I would very much like one.

  • Joseph

    Wow! Great work…..I have a feeling this is going to be as big as equal temperment; can't wait to see where this goes. BTW…..from an interface perspective, any thoughts on making multi-touch multi-dimensional? Specifically, I was wondering if there would be added interactions possible with more then one (or more) control surface. Maybe a platonic solid or something. I guess that because I'm not a hand drummer I'm not as interested in striking/slapping something as I am pressing/squeezing. Just a thought…

  • clevaclearjr

    I would very much like one of those

  • Pingback: Intimate Control and the Multidimentional Force Sensor - machine quotidien()

  • Pingback: A Glimpse of the Soundplane Controller, Innovative Tactile Multi-Touch, in the Lab; Call to Action()

  • Pingback: A Glimpse of the Soundplane Controller, Innovative Tactile Multi-Touch, in the Lab; Call to Action | It_Boy()