With the Web abuzz about Apple’s latest patent, filing, it’s worth reading the actual patent, 0060026536. Like all patent filings, this research may never translate to a shipping product. But it does make for good reading, and it clears up some issues — the most important one being this is about gestures, not specific hardware. Oh, and yes, Apple is working on a touchscreen music mixer:
Gestures, not hardware: Hardware isn’t mentioned anywhere in the patent. The patent seems to suggest, for the purposes of the filing, at least, that Apple is developing a software method for processing multi-touch inputs from hardware, not proprietary hardware itself. Specifically named in the filing are data analysis and classification, gesture recognition, floating controls and interactive UIs that respond to touch, and several specific gestures: zoom, pan, rotate, scroll, and page turns. (Yes, that’s right: this could be an e-book reader and not a real tablet Mac. But let’s hope not.)
For a sense of just how many possible hardware solutions are out there, just read the patent: “The touch sensing device may be based on sensing technologies including but not limited to capacitive sensing, resistive sensing, surface acoustic wave sensing, pressure sensing, optical sensing, and/or the like. Furthermore, the touch sensing means may be based on single point sensing or multipoint sensing.”
The audio mixer: hrmpf.com gets points for the best anaysis of this. Aside from their ability to get the images on the US Patent Office website to work properly, they’ve noticed specific specifications for a mixing device. And yep, we’re talking mixer as in audio mixer. This may just be for purposes of demonstration — or, perhaps, Apple is planning a home entertainment interface. Mixers often make the least compelling application of touchscreens, because (as you can see from the image), they duplicate hardware exactly but lose the critical feature of mixers: tactile feedback. That said, it’s in there.
What this means for music UIs: The patent itself points out the limitations of conventional UIs: they’re built around the assumptions of mice and joysticks, of a single X,Y cursor point in space. I don’t think anyone would argue that exclusive model will work forever. Interestingly enough, most music software, as an on-screen replica of devices like mixers, isn’t built around this model. It already inherits the assumptions of real-world devices that recognize you can use more than one finger (cursor) at a time. Otherwise, all mixers would have only one fader.
Is this coming, and when? Well, “when” certainly anyone’s guess (confident Web speculation aside)! There are plenty of hurdles here to making this a mainstream computer device: making the hardware affordable, building new UIs from the ground up, fine-tuning gesture recognition, and then re-programming the UIs for applications to take advantage of the interface. Tablet computers are easy. Multi-touch gesture recognizing computers are harder. It took 20 years or so to get the mouse right; it could be worth the time to get this right, too. But “if”? Again, the underlying technology is there, and the basic interface concept is intuitive. I think you’ll see all sorts of applications of the idea, possibly starting with simpler products first, and it’d be hard to argue that this will never happen.
What about the Lemur? Music thing observes that if this ships, it moves in on the niche-marketed Lemur multi-touch device for music. Of course, patent filing or no, many of us have wondered if general-purpose multi-touch devices could eclipse the specialized Lemur eventually. (The “eventually” being extremely unclear, given the cost of bringing such products to market.) The Lemur has the advantage of a head-start, of course, so this argument is meaningless until someone else actually ships. But it’s also worth noting the things the Lemur isn’t: it doesn’t do gesture recognition. It doesn’t have a truly interactive interface: your fingers can’t create new interface elements, for instance. Instead, they act as a set of ten cursors in place of one: while the objects underneath your fingers move around and respond to physics, you’re still fundamentally just pointing at them. A real gesture-based interface as hinted at in the patent is potentially much more sophisticated. It would also be incredibly challenging to develop; let’s face it, even Apple is talking about more pedestrian features like “page turns.” So who knows how sophisticated the Apple R&D labs are getting. (I’d watch out for research from other large companies, too — someone with money to invest in this project and large potential markets, like a Samsung or even medical instrument suppliers.)
Bottom line? The potential of the technology, as usual, is more interesting than the Mac rumor mill. It’s great to hear Apple is playing around with gestures. But they’re not the only ones. I’ve even seen experimental music systems for gesture processing. Ultimately, we’re likely to see new hardware to play with, in low-cost, mass-produced form. So the challenge becomes the one Apple is trying to conquer: build brilliant software, brilliant interfaces, and make them work for music. That’s a challenge no one is likely to definitively solve. Trying — that’s the fun part.