How Gloves and Wearable Tech Could Change Music Performance: In Depth with Imogen Heap and Team

In fits and starts, musical interface inventors have tried for decades to make manipulating digital music more expressive. But that persistence comes out of a clear goal post. They want the machine’s seemingly-endlessly possibilities to fit the human like a glove.

Imogen Heap is no stranger to pushing the boundaries of electronic musical performance, always making it seem as effortless as her songwriting and stage presence. For the Gloves Project, she assembled a super-team of wearable experts, interaction designers, and music researchers, several doctorates between them. This who’s-who have finally unveiled a project they’re ready to make public, and the whole team joined CDM in conversation about their work (first via global video chat, later via considered answers).

Gloves in music aren’t new. The challenge: make them better. Make them more expressive, able to actually improvise lines and not only control parameters (which you could do with a knob). Make them more precise, more responsive, lower latency. Make them more wearable – not only good musical instruments, but good gloves.

And then, make them public. Via a crowd-funding project on Kickstarter, the team seek backers both inexpensive (a few pounds, please) and substantial (pony up for a tailor-made set of gloves of your own, the kind that would normally require you to actually be Imogen Heap). They want to make something open source, and promise even backers at entry levels access to information; other rewards include in-person workshops and kits, some friendly to people doing their own experiments in wearable tech.

The gamble here may be not only to create one product – Imogen gloves you can bring home – but to advance the whole field. Wearable tech in music and other areas has often reinvented the same wheel over and over again. That isn’t necessarily a bad thing, but it has left the expressive output on a single plateau. This project claims to be the one that would advance past base camp.

Imogen and the Gloves Team joined CDM last year in the hacklab we hosted at CTM Festival, bringing an international group of hackers to try out the gloves themselves. Those participants included musicians, dancers, and designers with fashion backgrounds, not just the usual electronic musicians and electronic nerds. At the time, the Gloves Team pondered how they might open up this sort of experimentation to a wider audience, and how they might make the tool itself something that others might use. With the Kickstarter project, that vision has taken form.

The question is, what happens next. Right now, the whole crowd funding project hinges short of the finish line. With days to go, substantial funding still hasn’t propelled the idea to the goal. But then, that’s the perfect time to really challenge this brain trust to explain to us why they think what they’re doing matters.

(Obligatory, before someone quotes bad Nintendo-themed movies in comments:)
“I love the Mi.Mu Glove. It’s so bad.”

Got that out of my system. (Know your meme.) Let’s carry on.

Imogen, can you describe how you uses this in your performances? What are the gestures you make, and how do they navigate the music?

Imogen: One of the goals I wanted us as a team to achieve was to be able to develop the gloves system (physical and software side) to the point that I could write and perform a song entirely with the gloves.

I imagined the bass line first, before any of the rest of the song. I just knew I wanted to play it like a slow motion yo-yo or bouncing a basketball. It felt more intuitive to play it like that than linearly on a keyboard. I wanted to bypass programming in all the bends and curves of automation I would have needed to do after-the-fact to achieve this sculpted effect. I wanted to play everything all at once. To carve it directly. For this, I would map the parameters I wanted to adjust beforehand and then perform the whole ensemble of mappings. I love to program through the small hours, tweaking automation, but I loved so much to be able to have the choice to create the whole personality of the sound rather than always have to carve it after the fact.

Same goes with sculpting my voice with Adam Stark’s brilliant ‘Jellyfish’ patch. It’s a granular sampler where I can freeze or smudge my voice, choose grain length, assign panning and volume all on the move, instantly, instinctively. It feels almost as If I can stop time or extend it. To have these abilities at my fingertips and to be able to ‘jam’ with them in realtime. To be able to mould sound literally with my hands like play-doh has changed the way I write and think about music. A total game changer for me.

However.. I don’t expect to go to these extremes for every song or performance. The gloves can augment an acoustic instrument’s sound, be an instrument themselves or do more simple tasks like ‘record this audio’. My next project is a box of tricks, I will attach in and onto an acoustic grand piano. Combining the gloves and other gestural interactive hardware and software, with contact microphones and a batch of effects, I want to transform the way I play and sing at the piano. The gloves will be a part of this but not the whole.

Adrian Lausch: The gestures can technically be anything. We use certain terms for them, like “puppet hand” which refers to a gesture where you sort of have all your fingertips touching while outstretched. Other gestures of course include making a fist, or opening your hand entirely. These are 100% configurable and it is up to the user to define what they think is most intuitive for them. However, we are considering shipping the gloves with some very basic gestures predefined. Considering is the key word here. It’s a challenge, as all hands are different.

Kelly Snook: For her song “Me the Machine”, Imogen uses data primarily from the glove, but also from one other source, to map postures, gestures, and her location on stage (from the Kinect) to the musical parameters programmed inside Ableton. Here are some of the types of mappings she uses:

• Continuous Hand Orientation – Pitch, Roll, and Yaw – The continuous angles of her hands relative to horizontal and relative to either her own back or the audience are mapped to many different musical elements: Depending on which part of the song she is in, yaw (horizontal angle) controls panning, granular synthesis grain position, note interval, arpeggiator speed, drum cell, and a myriad other things. Pitch (vertical angle) can control track volume, note pitch, granular synthesis parameters, drum cell, reverb wet/dry mix, among other things. Hand roll is often mapped to change the timber of a note by controlling volumes on a pair of tracks containing different layers of a sound.
• Hand postures as triggers or modifiers – A number (usually between 4-10) of hand postures can be trained and recognized (via custom neural network machine learning), then used as switches, triggers or modifiers for preset switching, mode switching, drum kit/sound switching, starting and stopping recording, looping, muting and un-muting tracks, synthesizer voice selection, or to differentiate one orientation control from another similar control. for example, rolling the hand while making puppet hand bends the pitch of a note, while the same hand roll with an open hand changes the sound’s timbre.
• Discretized Hand Orientation Forward, Backwards, Up, Down, Left, Right – The broad directions of her arms can be used as additional triggers or modifers – something only happens if the hands are both up and making fists. Preset switching (the ability to switch between entire sets of mappings with pre-programmed gestures) is controlled by up to four simultaneous states.
• Peak Accelerations (Gyroscope peaks and drum gestures) – More custom algorithms in the software listen for drum gestures and peaks in acceleration that can be used as drum triggers or other event triggers, such as a flick of the right wrist turning on and off the arpeggiator in the introduction of Me the Machine.

The section “Why gloves” in your Kickstarter text answers the question of why you chose gestures – but why choose the hands?

Imogen: I didn’t want to ‘hold’ anything but wanted really detailed control. I didn’t want to ‘break the flow’ of the performance or being in the zone by having to walk over to something to interact with it. The gloves are like a second skin. They are part of me. An extension of me. I become hyperreal. I’m not forced into having to be in one place or stopping what I’m doing.

I play the piano and other percussion, sing, occasionally pick up a guitar and the odd random instrument here and there. I need my finger tips and sometimes I use my hands for body percussion, hence the hole in the palm, so you can still hear a full hand clap or slap. Plus … like most people, i gesticulate with my hands anyway… all the time… I do it for when words can’t express or I want to accentuate stuff. Music is one of the hardest things to describe and ‘talking with the hands’ comes into a lot of conversation when discussing sound!

Adrian: This was really motivated by the fact that a lot of musical instruments are simply very hand-focused. You would struggle finding anything that can match a trained human hand in dexterity. As you see a piano virtuoso flying over the keys with their hands, or a guitarist playing mind-bogglingly fast songs, you realise that the same can be done with the gloves. Thus, hands offer the most potential for limitless music making.

Kelly: The hands are usually a human’s most dextrous tool. Mind you, this is not always the case, but in music, hands are more often than not already being used and trained for expression. One of the goals with the gloves is seamless integration with music that’s already being made, such as augmenting the piano playing or looping an instrument while playing it, removing the conflict between expression and control (especially electronic control). To put the controls where the action is presents some challenges, but also, we feel, opens up new possibilities for expression.

What’s the significance of the objects themselves, the physical gloves? Does it mean something different to put something on rather than just use a camera? (We know cameras, etc., have latency issues – though some interfaces, like Theremin sensors, less so. But is there meaning to the act of wearing something?)

Kelly: This is a great question. It would be tempting to argue why the gloves are “better” than other types of control, but in fact, everything has its strengths and weaknesses. One of the strengths of wearing something so sensitive and sophisticated is that the artist is not limited and doesn’t have to be concerned about losing line-of-sight connection with a camera or other visual tracking device, completely liberating them from such concerns. In addition to being gorgeous, they can literally become part of the body, part of the movement, part of the expression, completely integrated into the art form. They can even become invisible, as desired by some artists who do not wish it always to be so obvious how technology is being employed. They can be illuminating or mystifying, as the artist wishes.

Imogen: I hardly know I’m wearing them after a while, but there is something about putting on your weapon of choice. Puts me into a state of mind, primed for music and movement, a freedom. I find them deeply liberating. There’s something about gloves that do that. Emancipate you, protect you. Washing up gloves, gardening gloves, safety gloves. These feel like the extreme continuation of that practical glove thread. The main reason though, is they’re the best tools for the job, literally in hand. There wasn’t anything out there that would give me the flexibility of movement on stage and the detail of control over the software.

Adrian: In my opinion, wearing something is simply a consequence of what we are trying to achieve. The gloves also look very cool and can certainly be used as a fashion statement, but I believe that wearing technology so close to your skin gets you one step further to overcoming the barrier between humans and technology. Even as I write this, I do not feel in touch with my computer. I am rather removed from it, hitting keys and staring at a screen. Making electronic music looks very similar. If I could make music as I walk through the room and listen to the computer outputting it in realtime, I would feel much more naturally connected with the technology underlying it all. This in turn helps people ‘forget’ the technology and focus on the music (or any other application they may be interested in).

Tom Mitchell: HCI [Human Computer Interaction] is moving along a continuum from the past where mechanical limitations were the main defining factors in interaction design, to modern times where humans form the basis of all interaction design and we can start to think about the most natural means of interacting with machines. Gloves sit in an important position along this continuum, as they offer uninterrupted monitoring of fine motor activity without relying on external monitoring equipment such as cameras or optical sensors, which suffer from obvious problems associated with limited work spaces, occlusion, and latency — all of which are serious problems for music interaction.

There is a clear and fascinating difference between technology existing in the form of distinct objects and wearable/camera/optical technology. The latter offers a fascinating position on the continuum above, where the technology disappears entirely and you are left with human movement. Of course these ideas are very well established and attempts have been made in the world of VR, but these ideas so far are ahead of their time. In the early days of VR, the technology was so limited that the implementation fell far short of the idea.

The obvious progression in technology is most notable than in the components of mobile phones where devices are smaller, more efficient, less noisy, faster — in every metric the progress is staggering. This is the technology that we are putting into the gloves: the motion sensors that we are working on now became available just last month. We are small enough and dynamic enough to progress with technology. There’s so much to do to keep with with technological progression.

Gloves have been commonly used in musical interfaces over the years. What projects inspired you from the past? How, specifically, is this one different?

Imogen: The one glove that got me inspired, as I wasn’t looking in the right places for ways to free me up on stage, was happening in Elly Jessop’s Vocal Augmentation and Manipulation Prosthesis (or VAMP) at MIT’s Media Lab. She was developing it as part of her research for the Opera of the Future. Elly showed me the glove and when I saw her catch a grain of her voice between her fingertips and ‘hold’ it there it was both a A-ha and a ‘duh’ moment! I saw the connection between music software and hand gestures. I knew this is what I had been missing! Ticking both boxes of transparency for the audience when playing with my ‘in the box’ music software live on stage and being mobile when interacting with it.

I’d tried all kinds of external controllers and boxes, sensors all over the stage… dreamed of stretchable, pliable material or body suits laden with controllers.. but totally missed the GLOVE. I then discovered and got introduced to all manner of incredible artists and musicians working with gloves but nothing I could get off the shelf.. so I set to finding a team to help me realise the set up that was now blossoming in my mind.

Kelly: When you reinvent the wheel, it had better be an exceptionally good wheel. 🙂 This is what we feel we have done. We have taken an tried-and-true concept then refined and refined until we felt it was beginning to be optimized for one particular purpose: music.

As it turns out, many of these refinements, such as in our extremely low-latency algorithms and wireless communications (x-OSC), our robust, stable, accurate, and precise sensing (x-OSC), our advanced textiles (thin, comfortable, durable, with design that works well with music performances, typing, programming, and long-term wear), and our advanced mapping software allowing a richness in mapping that has no limits. Now we want to generalize this for a wide number of users over a wide range of arts and other applications, which is the biggest difference. It is one thing to design a system that works with one performer for their specific needs, but another entirely to attempt to change the way music is made by everyone!

Adrian: From my perspective and from my research I noticed that most glove solutions are prohibitively expensive and do not offer quite as much functionality as what we are doing here. This is not to down-talk other specialised glove solutions. I think we just have a very specific aim of making gloves as affordable and therefore accessible as possible.

Ed.: Tom Mitchell provided us with a comprehensive research guide which is so interesting I will make it a different post. Suffice to say, the team have done their homework, they know their roots, and they know the work to whom they owe a debt – and they have some opinions on how they have advanced beyond that work. That seems fair enough. -PK

Experimental interfaces do range from very inexpensive to very expensive (and more expensive than this). What sorts of decisions went into the cost of this design, specifically? Were there particular elements of the design you can describe – textiles, sensors, etc. – that you were unwilling to compromise, which were the source of some of this cost? Were there already some optimizations that made it cheaper?

Imogen: For me, all options of cheaper were not an option as a musician, performer. I was not willing to sacrifice quality of playability over cost. These are not toys. Latency issues, or not [having] enough control is greatly stifling and creates a barrier to your instinct and flow to create. I want to be as close to the sound as possible – in time and in touch. There is undetectable latency when playing virtual drums, which is an absolute must for me! Many companies may scrimp on these details but as a musician this level of speed and control are paramount.

Rachel Freire: The process of making the textile gloves at a high enough spec has been challenging because currently there is no existing process that we can directly tap into. We are working with a manufacturer to create a totally unique design which is very exciting. This in turn will establish a process which could eventually be used by other people working on similar projects, so we are also pushing the envelope in bringing wearable tech manufacture to more people. This level of R&D makes the first edition gloves very expensive, but means people pledging on Kickstarter are also backing development of the whole new emerging world of wearable tech!

Tom: In the current market of high performance data gloves, we are ridiculously good value. But I suspect that we being viewed alongside music controllers and it’s hard for us to appear like good value next to mass-produced music controllers made by huge international corporations. We do hope to get the costs down in future – we are a non-profit company, so our aim is only to share the glove.

Kelly: Of course, our eventual goal is to make the gloves as inexpensive as possible, but we are adamant that we do not want to compromise on quality, because the most frustrating thing as an artist is to hopefully explore a new instrument, only to run up against a (literally) show-stopping limitation. We are interested in removing limitations, so we have opted for one of the highest performing IMUs available, truly state-of-the-art textile design, and meticulously designed software that will support and inspire artists or users rather than hindering them.

Relating to that, do you imagine that future development could make a similar design more affordable? What would having this crowd funding mean for that process?

Imogen: Yes yes yes! We absolutely want a glove that people hardly have to think twice about buying. Would love to make it as affordable as a mouse and a keyboard in time. I have no idea if we can achieve that, perhaps in a few years. This initial stage of external funding through Kickstarter is very much a part of that journey.

For myself, I have a pair of gloves which works for me, there’s always room for development, of course! I, however, have an amazing glove team to work with who can fix these bugs and stitch bits up. Others won’t have that luxury and the bulk of our work will be how to make these gloves robust and sustain wear and tear of other users without the need for glove team backup!

It’s an expensive thing to do… and I understand now why no one else has truly achieved a commercial music glove. It’s bloody hard! Somehow, we feel compelled to make it happen as we have done all this work and so so want others to be able to experience this joyous new found flexibility and freedom in electronic music making! We may be mad… and we may not succeed, but I believe we are going to do a mighty fine job if we are given the chance! Worst case scenario, I still have my gloves but I’ll still be really disappointed as I want to be able to give to others what I couldn’t find.

Rachel: The first round of gloves will have to be bespoke to a certain extent. Though the system is stable and refined, we are running manufacturing processes for the first time. This increases the cost, as more support is needed to ensure all our first-generation glovers get the best quality product. We are all so used to products which are made en masse by huge companies with millions in R&D budget, so seeing complex intuitive products in final development seems foreign. Kickstarter is the perfect way to communicate with future glovers during these final crucial stages and let people know that this is an ongoing project to create the best possible glove, rather than a gimmick product with limited capabilities which just pops out of a box. These gloves can and will be made cheaper in future iterations in larger numbers, but right now we want to be able to communicate with the whole Mi.Mu community and that is something which a larger, cheaper run of manufacture would not allow.

Kelly: Absolutely. Success with this Kickstarter will mean that we will be able to generalize our design while fully exploring options for manufacturing and design that will reduce the costs to their bare minimum. It might also bring to light features that aren’t all that necessary for most people, despite them being important to us. If this happens, we could envisage eventually making a less expensive, less capable glove that could be even more accessible and affordable. But we need to get these gloves into many hands to learn what we need to know about this.

I see there are already backers at the higher cost levels. What can those backers expect? Without violating their privacy, is there anything you can tell us about the sorts of people who have joined in? (Or maybe some of them are happy to share.)

Adrian: We have some high profile backers, yes. Of course we are obliged to protect their privacy. We approached them and asked whether we are allowed to mention their names and Amanda Palmer, for example, agreed. Those backers can expect very close, personal support from the team and an opportunity to really shape what the final product may look like. It’s really rather exciting to know that by August we may have a whole range of different artists working with the Gloves!

Kelly: We have backing from some very exciting individuals! Some, such as film composer Harry Gregson-Williams, have given us permission to let everyone know. Reggie Watts also can be seen in the list of backers (public), but i’m not sure he’s yet given us his blessing to shout from the rooftops, but he probably wouldn’t mind. Collaborator backers can expect a high degree of support and responsiveness from us, as well as questions from us that we’d like them to answer. They can expect a very high-functioning but not polished hardware and software system that their use will directly shape. And they can expect to have a lot of fun!

There’s now involvement for people at all levels of funding. How will you work with those people? Is there an advantage to doing this for some of our readers who are working on wearable projects of their own?

Kelly: We are still defining exactly how we will work with different groups of people, but we are serious about the collaborative nature of this process, especially with our Kickstarter backers. Even backers at the lowest reward levels will have access to the software and community forums and will be welcomed as contributors in the process. The hardware is very hackable, even if someone wanted to just start with an x-OSC and their own sensors, there is room for that in this community. All are welcome.

Adrian: The advantage is that you’re going to become part of community of makers and hackers and wearable tech enthusiasts. Your own projects could really thrive from their input and if you’ve bought one of our gloves, you may find all sorts of interesting projects happening around the gloves. The ingenuity of the online community is absolutely staggering! See the Kinect.

[When I questioned the team, no one had yet sprung yet for the kit, and I asked about that and a workshop offering – these were new levels the group added. -PK]

Adrian: Someone has [signed up for the kit], actually!

At the workshop, we want people to see the gloves. It is very important for us to get the Gloves in front of people, so that they can get a feel for what they are like. We are already doing this with some individuals here in London and giving them small demonstrations, but the workshops are there to do this on a larger scale. The exact details of the workshop may change still, based on community input and based on how many people may bring gloves they have bought already. Overall we want to offer people an in-road into using the gloves, help with setups and also listen to their feedback.

Kelly: We’ve got one kit backer! And as a matter of fact, he was a participant at your Berlin workshop [the 2013 CTM Festival workshop hosted by myself and CDM -PK], so this is an exciting development and an example of just how important that workshop has been to us in the development of our gloves (see later question).

You will eventually open source much of this project, I see. What should people expect to see out of that project?

Adrian: Hopefully the ability to take the gloves to any level they want. It is not necessarily infeasible to create, for example, Virtual Reality hands while wearing an Oculus Rift. The thing is, someone has to write the code to do this. As a team of 6 that is far outside of our reach. However, once we have a community behind us, that may change very quickly. Sign-language translation is also mentioned rather often.

Kelly: We will open source all the software and as much of the hardware as we own. We do not own the x-OSC hardware or firmware, so that will remain closed, but everything else will be open, hackable, and available freely (eventually). We are not sure yet of the schedule for opening up all the hardware and software, but it will be well after the Kickstarter gloves are delivered, to give our backers the benefit of advanced access to the software and source.

What was the impact of doing the Hacklab with us in Berlin? Was there useful feedback? Did it inform this project at all?

musicmakers hacklab short from CDM on Vimeo.

Imogen: The hacklab was really a fantastic experience for all of us. Firstly, a great bonding time for all us and for me to watch other people play with what had always felt like ‘my babies’… and to see the moment of inspiration as to what they were now free to do. To see how very differently each person chose to explore music and visuals with them. It absolutely undoubtedly set us on this journey to get them not just in my hands but out there for everyone to have a go with.

Kelly: The Berlin workshop was a turning point for us. It was our first (and only, to date) workshop interacting with a wider group of artists and engineers who were trying to use the gloves for their own interests. It taught us a lot about the universality of the gloves, the possibilities for the future, the limitations of cheaper technologies that were used in the workshop gloves, and some of the challenges of making gloves stable and robust enough for general use. We are very grateful for the opportunity to have had this workshop!

I hate to ask this, but what if this isn’t funded by the deadline? What’s the backup plan? And would you say something to people regarding the 5/10 GBP level?

Imogen: Argh! Terrifying thought but highly likely! To be honest… I’m not entirely sure. We need the cash we need to get us to the next level with a minimum amount of collaborator pledges to enable us to do that. It’s at this stage, an unknown… but we would be pretty sad to have failed so likely we will blindly continue because we love and believe in the gloves! Plus we also happen to really really love working with each other 🙂 x

Adrian: Not reaching our goal is always a possibility. One has to be realistic about that. Currently, projections are telling us we may overshoot the goal or get slightly underfunded. The best we can do is get the word out there, get people to share their excitement about the gloves and see what happens.

Should we not get funded, we would not consider that a failure. We have already raised a substantial amount of money and had a lot of extremely positive press. This is something we can work with even if we did not get funded. So, yes, we may approach people after the Kickstarter and initially work with a smaller group of those who are still wiling to buy the gloves directly from us. However, at this point I had rather not spend too much time thinking about not getting funded 😉

Kelly: We will start refining our backup plans if things really don’t pick up within the next week. Our number one goal is to make excellent gloves and get them out to as many people as possible, so we will likely do our best to provide gloves to as many of our backers as are willing to buy them directly from us, rather than waiting for another Kickstarter. We are keen to move the project forward as quickly as possible and learn from people, so this will likely be the main feature of our backup plan. Obvious, priority will be given to the 5K collaborators, but we value each and every person who wants a glove enough to spring for one (or a pair, or even a kit!). We’ll try to come up with a plan that includes everyone.

This is all listed in UK Pounds Sterling, but – can we get involved if we live in Europe, America, etc.?

Adrian: So long as you have a credit card, you can get involved anywhere in the world. Kickstarter displays price levels only in the one currency where the company running the campaign is registered. That’s simply a mechanic of their platform that is beyond our control.

Kelly: Absolutely! Due to the miracle of currency conversion, all things are possible (if not a bit painful). Welcome, Americans! 🙂

Thank you, team. Readers, endless, endless resources – from the history of gloves to lots of technical tips for making wearable tech work – at the official site for The Gloves Project. If you have any interest in this, this site is already a treasure trove – funding this project would make it that much deeper, we hope.

http://theglovesproject.com/

And, of course, the Kickstarter campaign:

Mi.Mu Glove for Music