David Merrill, working with Jeevan Kalanithi and (for the audio engine) Josh Kopin, wowed audiences at the TED conference with his Siftables interactive blocks. These strike me as what the Audiocubes have tried, sometimes unsuccessfully, to be — physical objects that react to the proximity of other objects, allowing you to manipulate music and media by moving around tangible blocks. Siftables are gifted with multiple expressive controls (tilt helping them break the plane of the surface), and intelligent screens that make them more adaptable and provide more visual feedback.

The music sequencer is very cool, though I think it’s actually the Scrabble-like game that may be the winner among the demos. But while TED celebrates all things cool and futuristic for their easily-digestible novelty, sometimes I think the most important design achievements are as significant in their shortcomings as their successes. Siftables raises some important questions. Sure, you can now use two hands, as opposed to the single mouse pointer. But do those same tangible blocks actually limit the kinds of interactions you can have, even compared to a traditional UI? Does it sound any different/ And note that — a little bit of tilting aside — the interface is still essentially two-dimensional. I’m personally really stumped by the question of how you can make a successful three-dimensional controller. Yet three dimensions is how all of us interact with space and movement daily. Maybe it’s the fact that we do so much of this, comprehend movement so richly, and take it for granted, that makes mapping those gestures so challenging.

That’s not a criticism of the project – or a claim that I can do any better. On the contrary, I think it’s important to do this sort of work because it can raise those kinds of questions. We’re gifted as a generation to try out and test these ideas with flexibility that was never before possible — and the intelligence built into these objects shows the potential of that power.

More of Siftables after the jump. And it’s well worth checking out David’s other projects, too – when I last ran into him, he was showing off the totable, Linux-powered Audiopint sound-processing box. Oh, yeah — and he’s the face control for guitar guy!

Siftables Music Sequencer from Jeevan Kalanithi on Vimeo.

Siftables project page

More Merrill Goodness

  • Would be great to replace all my coffee table books with this. Might make me look a little smarter

  • wow this would be a great tool for teaching
    children. reading, writing, arithmetic etc.
    it def has much potential.
    mad props!!!!

  • MattH

    "I’m personally really stumped by the question of how you can make a successful three-dimensional controller."

    3 theremins at right angles? ๐Ÿ™‚

    A wiimote is a good start.

    The blocks concept could work with three dimensions, but you'd have to be careful about what you map to height, otherwise you run out hands pretty quickly. Gravity always wins…

  • rhowaldt

    i was wondering, can you program the blocks yourselve? so can you say 'i want variation 3 of my reverb-block to be deep and long'… can you make your own interactive portraits, program your own colors for the color-buckets, and code your own word-games? how are they going to market this, and sell this, pre-programmed or with accompanied software or something?

  • Nice post. I think you hit on something with this:

    "Yet three dimensions is how all of us interact with space and movement daily. Maybe it’s the fact that we do so much of this, comprehend movement so richly, and take it for granted, that makes mapping those gestures so challenging."

    I think the weakness that you pointed out in this project is also a strength. Once they get over the complexity of the mapping, we'll be able to interact with music similarly to how we would interact with anything else in the physical world. Very exciting!

  • A friend of mine and I started tackling the "3D sequencer" concept 2 weeks ago. Well, I began writing music as though I'd have access to such a thing 15 years ago, so much like anyone else who read Howard Rheingold's "Virtual Reality" book in 1991 I've given this idea waaaaaaay too much thought. I am singularly poor making physical things so thus far, aside from the p5 data glove, my attempts at multidimensional control have been concepts and not neat new toys.

    Just waitin' on the multi-gestural hologram table sequencer…

  • On the 3D thing — well, absolutely. And it raises more questions, of course:

    * no reason this very interface couldn't be 3D, so…
    * is this the kind of 3D interface you'd want?
    * is it actually better to just work in two dimensions? is a surface actually preferable?

    These are more questions than criticisms. ๐Ÿ™‚

  • Defyer

    2 XY's can control 4 parameters, whereas a 3D interface should be designed in such a way that it can be manipulated with only one hand, otherwise you get only 3 parameters with both hands.

    I think a neet (and weird!) solution would be a small ball suspended on some elastic strings (set on a cubic frame) which can detect tension changes and convert them into 0-127 values. An alternative would be using position sensors, though I'm not sure if that would be cheap.

    But an easier solution would be a custom 3D joystick (basically an XY with push/pull)

  • I've tried thinking up a wah-wah that is nested on a sphere (so it can be turned this way and that) plus with up/down pressure sensitivity… more important to me now that I can't use my p5 glove (which I use to send midi from 3D + pitch/yaw/roll and all five "finger bend" sensors so they operate much like faders). But gestural 3D, much as multitouch is gestural, was thought up by Gibson long before any of this was remotely feasible. I'm not a craftsman, though; I just want to use it. It couldn't possibly be expensive to make this stuff anymore, but there needs to be a prosumer use like iPhones and Nintendo DS to bring the price down.

    As to a 3D interface for sequencing:

    1) There's more room to draw. Imagine all automation channels spaced out with metatags to tell you what they're latched to and what they control, distinguished by color and shape.

    2) I use a lot of binaural processing and now that surround sound and high-def music storage are common I'd like to return to attempting to score in multichannel automated surround channels. 3D interfaces would certainly aid that.

    3) 3D representations of recorded 3D gestural data: it's all well and good to use a VR glove to "express" your automation, e.g. 3D mapped to filter cutoff, filter resonance and distortion wet/dry as a simple example. But in a traditional 2D view how in hell are you supposed to ever see wtf is going on between those three interrelated things? Or edit it? Whereas a knotty sphere of color in a time-based histogram that you could "scrub" through with a jog wheel would be heavenly.

    I will stop there and see if ya'll think I'm way out on a limb. It'd be invaluable to me but I can't really assess whether it'd be useful to others.

  • d. h.

    reminds me of sony blockjam.
    to me it seems that many of these "new" _tangible_ (fancy word, isn't it?) interfaces after some years are copied (or re-invented independently), recoloured, extended by an additional sensor and then presented as new again. nothing too bad about this, as long as development and evolution of man-machine-interaction continues.

  • d. h.

    sorry. didn't get the link correct. wanted to say: reminds me of sony blockjam.

  • Aaron Liven

    I am personally very impressed with the Siftables. With sample and screen customization as well as a large variety of different parameters, those could be an incredible live music tool. Of course you'd have to many more of them than the video showed, but very very cool.

  • Yep, absolutely, Blockjam is a good comparison. I think what these are is smarter and more versatile — that is, not so tied to just one application.

    It's really that customization of feedback and parameters that's cool here, as Aaron says (and I'm likewise very impressed)!

  • Having experimented a bit with 3D interfaces (basically "3 theremins at right angles"), I think one of the biggest issues is haptic feedback. Tracking with capacitive sensing, IR arrays, data gloves, or structured light 3D scanning are all fun, but have big learning curves due to the lack of haptic feedback. Free space 3D interfaces and physically manipulable 3D interfaces are useful for entirely different things.

  • Good point, Kyle. One's arm doesn't exactly like being held straight out and waved around for long periods.

    As for feedback, acousmodules' "P5 Control" VST plug-in is invaluable for data glove owners:


    A very nice 3D way of seeing what your waving hand is up to, plus finger clench.

    "This plugin allows at the same time to visualize the eight controls which are send from a P5 Glove (X, Y, Z position and the bend of the five fingers) and to modify the values. It must receive the MIDI datas that are sent through a MIDI Router by the P5GloveMIDI from Ross Bencina (in absolute coordinates).
    It can process the datas in the following manners :
    – change the controller type
    – scaling of the curve
    – invert the direction
    – shift
    – transformation of the curves by simple mathematicals functions or a graphical transfert function
    – lower and upper limit
    – inertia, to smooth the irregularities of the P5 or create very smooth movements
    – the thumb can trig a note from a threshold"

    A $40 glove and a freeware VST and you've got quite an amazing piece of futuristic gear.

  • Jerry

    Interesting article. But honestly, comparing the Siftables with the AudioCubes is just plain stupid.
    It's comparing a prototype (that maybe would cost thousands of dollars if sold) with a moderate-priced product that actually exists, is accessible, works well and has proven its suitability in the real world.

  • This is an amazing little device, i can’t wait to get a set and start using it. The possibilities are endless.. especially in a school / educational system / environment! Anyone know when the release date is for these?