With Apple’s PrimeSense Acquisition, Shifting Landscape in Depth Sensing, Motion Art

It wasn’t so long ago that point-and-shoot cameras were big, dedicated affairs. Now, camera sensors are everywhere.

What’s next? Expect depth-sensing cameras like the Kinect’s to become as ubiquitous as camera sensors are in phones. And don’t listen to the analysts: if Apple is buying PrimeSense, they’re thinking iPhone, not only their Apple TV “hobby.”

The news for the open source art hacking community using this stuff? Bad. And good. But… more on that in a bit.

With touch staked out as input method, vision and, more broadly, “perceptual computing” seem poised to reshape the way we interact with devices. Touch broke the boundary between machine and person imposed by keyboard and pointer – as Jobs famously said, using your finger as the pointing device. But, nearly a century after Leon Theremin first built musical instruments that sensed presence, the next interfaces may not require touch at all.

This week, Microsoft should have the headlines with Xbox One. But, having already seen the Xbox sensor, just as much interest may revolve around Apple’s reported acquisition of PrimeSense.

The new Kinect camera on Xbox One, superior to the original Kinect in every measurable way, isn’t based directly on PrimeSense technology. With Microsoft the key customer for PrimeSense’s unique expertise, that meant Microsoft’s move away from their technology set up an acquisition. But PrimeSense did have a strategy. The best read I’ve seen on the company comes from Engadget, who charted their early history, how those first meetings with Microsoft went, all the way up to their future plans:

Life after Kinect: PrimeSense’s plans for a post-Microsoft future [Engadget, June 2013]

Now, it seems, Apple (pending final approval) has acquired PrimeSense – and the tech Microsoft orphaned. (Microsoft may regret leaving them out on the open like that.) Excellent coverage by Charles Arthur at The Guardian:

Apple ‘has bought Israel’s PrimeSense, maker of first-generation Kinect sensor’
Deal points to functions in upcoming hardware as sensor could be used to detect gestures, faces or internal geography

Much speculation from the tech scene has suggested that Apple is focused on their Apple TV product. That may be possible – and these interfaces do work well in living room-style situations, as Microsoft so vividly demonstrates. But a look at what PrimeSense has next suggests the iPhone and iPad might be just as likely recipients of the tech.

The roadmap at PrimeSense focused on Capri, which the company calls the “world’s smallest 3D sensor.” Like Microsoft’s efforts, it improves upon depth resolution. But it also works better in daylight (cough, phone!), and shrinks in power consumption as well as size (cough, cough, cough):

“The Capri is the future of PrimeSense,” Dagan said. “Not only is it smaller, lower-power and cheaper, it also has better depth, better middleware that can actually run on a mobile processor… our end goal is to make it small enough to make it into every consumer device.”

Source: Engadget

As Forbes’ Connie Guglielmo notes, Apple has been active in acquisitions this year:

Apple CEO Tim Cook said that the company had done 15 acquisitions in fiscal 2013 – or an average of one every three to four weeks. Those deals include maps-related technology companies including WiFiSlam, Embark and HopStop, and Passiv, a low-energy chipmaker whose tech can be used for fitness and health-monitoring wearable devices.

And you can put those acquisitions alongside past acquisitions that helped Apple bring its hardware and systems-on-a-chip expertise in-house.

Whither, Hackers?

Apple and Microsoft aren’t the only players in this game. Despite lackluster efforts by PrimeSense to partner with other hardware makers, there are still other efforts. Intel, for instance, is working on their own “Perceptual Computing SDK” which they hope to cover a range of Xbox-like capabilities, from voice to facial recognition to depth sensing. Right now, there’s just one Creative camera, but that could change – and Intel has the expertise in their OpenCV toolkit, which before Kinect was the standards bearer for artists tinkering with vision.

Perceptual Computing SDK [Intel Software]

The irony of all of this is that the real case for this sort of shift to “perceptual computing” came originally from artists. Artists and hackers made the cultural case that made this popular, and by generating a surge of interest in Kinect, must have at least contributed to that hardware’s success as one of the fastest-selling hardware launches of all time. (How many readers of this site, I wonder, heard from parents and family members for the first time about this tech? The impact on the press was far greater than Microsoft alone, armed with a couple of Wii-like arcade games, could ever have accomplished on their own.)

So, how do artists fare in this?

As far as PrimeSense, it’s hard to be optimistic.

Let’s go back in time.

Remember, the first tool set used by artists was one generated by a bounty sponsored by Adafruit – one that arguably helped launch that company to its current success. (Adafruit’s Limor Fried was recently named Entrepreneur Magazine’s entrepreneur of the year.)

WE HAVE A WINNER – Open Kinect driver(s) released

The success of the bounty project and the creative projects that resulted likely changed internal politics even at Microsoft. Good reading on that:

Kinect developer claims credit for hack bounty idea

For Microsoft’s part, they have been stalwart advocates of their own development tools for Kinect and Windows, and most recently embraced interoperability with open source tools on Windows. And while not officially endorsed by Microsoft, hacked tools have continued to thrive on other platforms – and Microsoft has been generally supportive of the work that has come out of those platforms.

The open source work Microsoft is doing now, and the research they’re supporting, holds promise for both the huge installed base of Kinect cameras and the countless more that will spread with Xbox One (and next year’s Windows release). See, previously:

Microsoft Embraces Open, Creative Coding: New Kinect openFrameworks, Cinder Integration

But the use of Microsoft tools is still a subset of the work done by artists. After the original hacks and bounty, PrimeSense worked with partners to establish OpenNI. That included robust development of open source middleware, and an SDK that could work across different camera sensors, not only the Kinect-based ones:

OpenNI

OpenNI is both a set of code and an organization. The code is safe; it’s released under an Apache license. And once something is open sourced, you can’t un-open source it. If Apple decided to use their acquisition of PrimeSense to take patent action against competitors, that’s possible, and Apple has been aggressive about their patent portfolio. But that brings us to the more important point: the OpenNI software is middleware. It doesn’t work without the camera and sensors.

The issue is, a lot of the real work is presently in the sensor, not the software. It may be the system-on-a-chip expertise that Apple specifically desired. Doing this in hardware is what allows depth sensing today to work so accurately, and so quickly.

Now, it’s not that OpenNI’s middleware isn’t useful; it is. And having a standard set of APIs has created common practice in these applications, one that should allow people to substitute new cameras and sensors as that tech improves.

The problem is, that doesn’t resolve the question of where the hardware will come from. We know with Microsoft (well, mostly, as we await Windows developer tools), but Apple would be entirely hypothetical. And it would be up to Apple how much of the capabilities of the camera they expose, which becomes doubly an issue if you’re talking about mobile devices and privacy.

And if you follow the partners in OpenNI from the beginning, the really valuable technology has always come from PrimeSense. Check the organization:
http://www.openni.org/organization/

Side-Kick is a games house – that makes stuff with PrimeSense tech.

Willow Garage is a robotics house – that makes stuff with PrimeSense tech.

Asus was mainly an OEM – that needed PrimeSense to make any depth camera.

The most interesting partner, actually, is the least known – OpenPerception makes a point cloud library, but more generally, they’re an open source foundation dedicated to open software. They’re likely to be the torch-bearer from OpenNI going forward because of their software specialization. And even without OpenNI, they can continue to focus on open, cross-platform solutions, and what can be done in hardware.

And while most of the GitHub commits come from PrimeSense, OpenPerception is doing code sprints and more that are generating new code. It’s also relevant that they’re partnering with private industry, demonstrating that the open source approach can yield commercial value, too.

Also, whether or not OpenNI continues to be the primary organizational entity, it’s worth looking through the advisors of OpenNI the organization. There’s a lot of research and development expertise outside PrimeSense. Those people’s value to the industry is on the rise. But a lot of the work is in academia, too, and academia may continue to leverage that investment. It’s not hard to imagine some of these figures carrying on where PrimeSense’s contribution to OpenNI would end, if Apple chooses to stop contributing. (And it would be up to Apple what role they would or wouldn’t take going forward.)

PrimeSense does sell their own developer sensors, which you can still buy – for now. (Might want to act fast, if you really want one.) But perhaps the bigger question is whether these other initiatives will be able to keep pace, in what’s likely to be a very competitive fight to get smaller, more precise, and use less power.

My sense, though, is that the value of the code artists have contributed to tools like OpenFrameworks and Processing, in their own projects and libraries, will be greater than what’s currently available in OpenNI. Indeed, given the more secure future of the Microsoft tools, that effort we looked at this fall has just become more important.

For one glimpse of where open hardware might go, look at the Kickstarter-funded (generously-funded, over its goal by a lot!) STRUCTURE.IO.

Just be a bit suspicious about this line: “We will work closely with OpenNI so that the Structure Sensor will work with the latest version of OpenNI 2 (on Windows, Linux, OSX, and Android) by the time backers receive sensors.”

Of course, “with OpenNI” has generally meant “with PrimeSense.” It’s possible Apple could choose to support open vision the way they have WebKit and Safari, but again, because more of the magic and value is in the hardware, not the middleware, don’t hold your breath.

For the same reason, though, other open code efforts could work with hardware like this – or competing PC and mobile hardware – and we could see more development.

I’m also guessing artists are nonetheless dreaming of someday writing tools for the iPhone or a future Apple TV or the Mac, though – and while it’s too early to speculate what that looks like, it could be very nice, indeed. In fact, if Apple is open to creativity and finds a way to make this tech available to artists, and not just something proprietary like Siri, it could spawn a whole new generation of apps.

If they do make that call, the work of a community of intrepid artists who have demonstrated what’s possible are who you should thank.

So, the ground is shifting. But don’t sweat it too much: it’s still artists and developers (and, mainly, artistdevelopers who can succeed at both) who make this stuff useful. “Perceptual” and “ubiquitous” computing had been something people predicted for years, but as with touch and even photography, neither becomes valuable until people use them in creative ways, until they demonstrate the value to the culture – and make that translate to value for the consumer. The competitive, fast-changing landscape should make things a lot more interesting. And open art around these tools is here to stay.