“Gesture” is a term that gets tossed about regularly in modern interaction design. But to me, the word is most deeply associated with classical music – and the gestures that first brought me to music, the piano. In this video for TED@BCG, I got to talk about that and why I think it can inform design through today’s newest interfaces.

In rapid-fire form, obviously more could be said about this.

The software involved:
NI Mate, which I need to revisit – it’s gotten more sophisticated since what you see here

And visuals by Geso.

I am now deep into another TED project – TEDxESA at the European Space Agency – but while having a quick coffee, it seemed long overdue to share the last TED project, in front of the audience of TED’s TED@BCG (an official TED-curated event organized with Boston Consulting Group). I’ll be honest: I was a bit slow to share this partly because I would have loved to have had more time to prepare that talk and performance. Performing onstage at a conference is a unique challenge, on another order of magnitude when adding computer vision.

Anyway, thought I might share it (having put off doing so) to see what reactions were.

And I will say, it was a pleasure to work with the TED organization and see how they operate behind the scenes. It felt like having a personal trainer for talks, even in just a brief time. And now I look forward to an entirely different setting, with the folks at ESA. Stay tuned – suffice to say, in the midst of posting this, we’re exploring the world of space exploration and research through the medium of sound. And I think there will be a lot more to say very soon. So back to that – and greetings from south Holland.

From Beethoven to Kinect, linking music to our bodies [ted.com / TED Institute]

Photo: Wolfram Scheible

ted

Previous post

Now you can sync a Game Boy to a Korg volca or monotribe

Next post

Confidence, humanity, and drumming: Kiran Gandhi interview

  • Name User

    first of all, props for getting up there and twiddling invisible knobs in front of a crowd

    • foljs

      Probably scarred a lot of them for life…

  • Name User

    first of all, props for getting up there and twiddling invisible knobs in front of a crowd

    • foljs

      Probably scarred a lot of them for life…

  • chap

    I am still really, strongly against those type of ‘in the air’ performances. I think it makes absolutely no sense at all, they just make the artist look ridiculous. Instruments need to be touched, that’s why they are called instruments.
    I’m not saying that using Kinect in artistic performances is useless at all, but definitely not in actually ‘playing’ music.

    • Dubby Labby

      Yes… It is like these people using computers for letter sending!

      Ironic mode: off.

      C’mon!

      • foljs

        No, it’s like people feeding digital “pets”.

        Just because something involves computers doesn’t mean it’s a) progress, b) a good idea. There’s a whole field called HCI you know — and invisible “hold your hand in the air for long” interfaces don’t fare that well scientifically (or anatomically).

        • Dubby Labby

          Sure because dancers can’t be instrumentalized to perform. We only could do music with knobs and buttons like violins.

          Every thing has its use. If it isn’t your cup of tea (or the original rant guy) walk on but don’t try to argue against new paradigms in Music technology with subjectivity or “common sense” because ergonomically a Violin or piano have their own drawbacks too and we still use them to make “Art”.

    • I certainly wasn’t advocating this as the future of instruments. On the other hand, the Theremin is effective and expressive as an instrument and it doesn’t require touch. It requires years of practice to master, but then so do many instruments.

      Anyway, this was for an audience of non musicians, and an effective way of demonstrating one possible direction for interaction design.

      • lala

        its ergonomically terrible, its the worst interface of all (waving your arms in the air) – its an interface without any feedback except for sound.
        And you have to stop playing it after a few minutes – because everything will start to hurt.
        Its unusable, a gimmick for live shows.
        humans are not build for this interaction, sorry minority report fans.

        • lala

          you play the piano yourself, you know you can do this for hours.
          How long do you want to hold your arms above your head or do the zombie gesture (i guess you get what I mean).
          just because its possible to do it doesn’t mean it makes much sense to do so. Sorry for the rant.

          • lala

            leap motion is ok for very little and short time interaction, like wave your hand to go back in your browser or something.

  • chap

    I am still really, strongly against those type of ‘in the air’ performances. I think it makes absolutely no sense at all, they just make the artist look ridiculous. Instruments need to be touched, that’s why they are called instruments.
    I’m not saying that using Kinect in artistic performances is useless at all, but definitely not in actually ‘playing’ music.

    • Dubby Labby

      Yes… It is like these people using computers for letter sending!

      Ironic mode: off.

      C’mon!

      • foljs

        No, it’s like people feeding digital “pets”.

        Just because something involves computers doesn’t mean it’s a) progress, b) a good idea. There’s a whole field called HCI you know — and invisible “hold your hand in the air for long” interfaces don’t fare that well scientifically (or anatomically).

        • Dubby Labby

          Sure because dancers can’t be instrumentalized to perform. We only could do music with knobs and buttons like violins.

          Every thing has its use. If it isn’t your cup of tea (or the original rant guy) walk on but don’t try to argue against new paradigms in Music technology with subjectivity or “common sense” because ergonomically a Violin or piano have their own drawbacks too and we still use them to make “Art”.

          Edit: Often Art isn’t related to “common sense” or ergonomics. That’s the idea I was trying to point.

    • I certainly wasn’t advocating this as the future of instruments. On the other hand, the Theremin is effective and expressive as an instrument and it doesn’t require touch. It requires years of practice to master, but then so do many instruments.

      Anyway, this was for an audience of non musicians, and an effective way of demonstrating one possible direction for interaction design.

      • lala

        its ergonomically terrible, its the worst interface of all (waving your arms in the air) – its an interface without any feedback except for sound.
        And you have to stop playing it after a few minutes – because everything will start to hurt.
        Its unusable, a gimmick for live shows.
        humans are not build for this interaction, sorry minority report fans.

        • lala

          you play the piano yourself, you know you can do this for hours.
          How long do you want to hold your arms above your head or do the zombie gesture (i guess you get what I mean).
          just because its possible to do it doesn’t mean it makes much sense to do so. Sorry for the rant.

          • lala

            leap motion is ok for very little and short time interaction, like wave your hand to go back in your browser or something.

  • Ray Johnson

    it’s Dieter and der Sprockets( Saturday Night Live) in 2015! The more things change…
    chap- gestural interaction isn’t necessarily meant to replace playing an instrument, but
    it is a new way of playing Music. it’s more that you can dance an arrangement (or a mix, for you young’uns), or control the mix of light and sound, or other things that haven’t been thought of yet. check out Momix on youtube, and maybe you’ll get an idea

  • Ray Johnson

    it’s Dieter and der Sprockets( Saturday Night Live) in 2015! The more things change…
    chap- gestural interaction isn’t necessarily meant to replace playing an instrument, but
    it is a new way of playing Music. it’s more that you can dance an arrangement (or a mix, for you young’uns), or control the mix of light and sound, or other things that haven’t been thought of yet. check out Momix on youtube, and maybe you’ll get an idea

  • genjutsushi

    Roland D-Beam

  • genjutsushi

    Roland D-Beam

  • misksound

    I own a Leap controller with a constant migratory pattern between ‘front and center’ on my desk, and in a drawer… my real issue with the concept is the lack of resistance i get from just.. waving my hands in the air.

    Someone who has trained as a dancer would probably be more comfortable performing with these tools—as they’ve taken the time to understand the subtle nuances of their own body.

    Big up to peter for doing this, though! Having performed pieces for my composition degree, I can relate to performing in this context.

    When you’re used to reading a floor of active, moving people to determine where to take a performance, it can be *very* weird to be playing a piece for a room full of people sitting in chairs, observing you.

    also—Peter’s style? “WilliamsBerlin” 😉

  • misksound

    I own a Leap controller with a constant migratory pattern between ‘front and center’ on my desk, and in a drawer… my real issue with the concept is the lack of resistance i get from just.. waving my hands in the air.

    Someone who has trained as a dancer would probably be more comfortable performing with these tools—as they’ve taken the time to understand the subtle nuances of their own body.

    Big up to peter for doing this, though! Having performed pieces for my composition degree, I can relate to performing in this context.

    When you’re used to reading a floor of active, moving people to determine where to take a performance, it can be *very* weird to be playing a piece for a room full of people sitting in chairs, observing you.

    also—Peter’s style? “WilliamsBerlin” 😉

  • Rafael Marfil

    Congratulations for your conference at TED!! 😀

  • Rafael Marfil

    Congratulations for your conference at TED!! 😀

  • Bandwidth

    Well done Peter! Looking forward to seeing more talks from you in the future.
    As a musician I actually find this very exciting. Eventually when the technology becomes more advanced, I think we will see a lot more depth and dynamic control over the music with regards to the level of expression from an individual body, a little bit like the touch sensitivity on keyboards.
    Imagine the expressivness of someone like Bernstein for example. It will be very personal to each indvidual. 5 different people can play the same note on a violin and they can all make it sound different. Where this will go is not there yet but the idea is and that is what is important.

  • Bandwidth

    Well done Peter! Looking forward to seeing more talks from you in the future.
    As a musician I actually find this very exciting. Eventually when the technology becomes more advanced, I think we will see a lot more depth and dynamic control over the music with regards to the level of expression from an individual body, a little bit like the touch sensitivity on keyboards.
    Imagine the expressivness of someone like Bernstein for example. It will be very personal to each indvidual. 5 different people can play the same note on a violin and they can all make it sound different. Where this will go is not there yet but the idea is and that is what is important.

  • James

    I know the talk was about gesture, however your comment about motif somehow reminded me of a Matt Groening quote where he claims to create all his characters such that they can be identified by silhouette

  • James

    I know the talk was about gesture, however your comment about motif somehow reminded me of a Matt Groening quote where he claims to create all his characters such that they can be identified by silhouette

  • The Brain

    More fine tailoring from the men who brought you the wardrobe featured in “The Emperor’s New Clothes.”

  • The Brain

    More fine tailoring from the men who brought you the wardrobe featured in “The Emperor’s New Clothes.”