Ben X Tan writes to let us know he’s working with hacks for Microsoft’s Kinect 3D camera system for Xbox to perform MIDI control. Result: depth-sensing, gestural musical manipulations! It’s just a prototype, but since today I cover the larger landscape of what’s happening with Kinect, it’s well worth teasing. From the description:

Coded in using this:

Very hacky ugly, yucky, alpha prototype, source code available here:

Next project is making a version of pmidic that uses Kinect. Then, you can control Ableton Live or any other MIDI software or hardware with you limbs. Isn’t that amazing!!!

If you are interested, you should also check out:

He tells me, “It’s just proof of concept for now, but want to make something nicer in the long run 🙂 The community on IRC for this stuff is great. Very supportive. I love all the hacking going on at the moment. Especially after watching the Social Network. Very inspiring!”

So, what’s this whole Kinect thing about, why should you care, why might it be useful to artists and musicians and designers generally, and where do you go to find the code? I’ve rounded up various hackers working on the project to answer those questions on Create Digital Motion (in this case, Capture Digital Motion):
Kinect Hacking and Art Round Table: Why it Matters, What You Need to Know

  • I've been thinking about this since I read a post on this at Of course you know what this means? To get into this, we're going to have to take dance lessons or higher choreographers ;^).

    It will be interesting to see where this all goes.


  • ^exactly, knowing some dancers and martial artists, we thought about the applications and were very excited. a kick would sound like a kick and a sweep would be a sweep.

    ps: Ben Tan X is from the martix

  • Adrian

    Not sure where else to mention this, but you need a favicon, stat.

  • Aaron

    The Dev behind Blok Modular has also been working on implementing Kinect control into his modular app.. maybe give him a buzz and see whats up?

  • Jeff

    Neat proof of concept. Thinking of all the applications for this makes it really tempting to buy one. I can't even imagine the possibilities–just having a 3D XYZ controller for filters and things would probably be incredible.

  • It is dandy alright. For me, personally, I see this as an opening up for integrating the audience into the work. Set it up in a museum or mall and let people take part in it. The person on the left controls the rhythms by his/hers movement, the second person can play melodies and shape the sound, the third can pads as well as intruducing FX to the sounds the other people are making. If someone just walks by the installation it would produce sounds of all three layers… I have spent lots of time last couple of months working on concepts for interactive music and ways to get the audience to be a participant in the creation without having to learn how to play instruments. Great times ahead due to the evolution of stuff like this!

  • Oh so my ten yrs of breakdancing is going to mean something?!

  • I intend to build something with the Kinect for my band's EP launch gig. Something that involves the audience and a projector 🙂

  • awesome!

    just played with a kinect for the first time the other night at a friends house. started getting ideas, specially after seeing some of the vision related robotics projects a few other groups did at my uni.

    could be interesting to try integrate this with a bands live set indeed.
    for sound and also for visuals..

    its like Johnny Lee's Wiimote hacks meets

  • Why is this relevant? Maybe i'm missing something, but he is not using the depth information from this kinect thing, so basically it a camera.
    These things where cool 30 years ago.

  • @Pierre Actually, I am using the depth sensor. Its a lot more accurate than processing a flat image. Check out this update and you will get a better idea of what its doing.

    As I've mentioned…its just a prototype so far, and there is still a lot to do before its something "real". To give you an idea, I intend to use skeletal tracking on multiple points of interest on multiple skeletons.

  • Fancy Guey

    Hi, with the help of Shinect.