Microsoft’s Project Natal unveiling for Xbox 360 was no question a blockbuster of technology presentations, nothing short of sheer magic in a games industry that has lately looked somewhat backward-looking. The combination of a 3D-capable camera with facial and object recognition and vocal recognition and mic interaction takes already-smart elements and puts them together into something bigger. But demos are just that – it’s the reality of what’s happening in interaction design that’s interesting.
So, some more details on Project Natal:
Latency?
Note that the video in the post yesterday carries a significant disclaimer: it’s essentially a conceptual mockup, not a real demo. In videos we’ve seen of the current prototype, there does seem to be a significant lag between an action and its representation on the screen. This may have to do with the sheer amount of data and analysis that’s being done on it. Unfortunately, as this is only in prototype stage, it’s impossible to do much more than speculate.
I’m not the only one to notice this: Keith Lang, interaction designer at Plasq, sees the same concern in his (excellent) round-up of coverage of Project Natal:
Microsoft Announces ‘Natal’ 3D System [UI&us]
Don’t underestimate how important the latency could be, either. Even tiny differences in latency can have a major impact on how someone feels about an interaction. This is also significant to music people, who generally like their interactions to use tiny latencies and approximate the rate of the audio they’re controlling.
I’ll reserve judgment until the final version, naturally! But it’s something to watch.
Johnny Chung Lee and the 3D Technology
The ingenious creator of various Wii tracking hacks, it seems, is now with Microsoft. (Nintendo, your loss. Rest of the world, he has code tools on his site, so even without hiring the guy, you can benefit from his knowledge.) Cristian Campo spots the news in our comments.
For his part, Johnny is careful to note that he’s not responsible for what you see, but is working with them on productization.
Project Natal [procrastineering]
He can’t reveal anything but what’s public, but he does have some more extensive details on the technique – essentially, information that is public but in a more technically-specific form:
The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.
But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision research. The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product. At times, working on this project has felt like a miniature “Manhattan project” with developers and researchers from around the world to coming together to make this happen.We would all love to one day have our own personal holodeck. This is a pretty measurable step in that direction.
Creepy Kids
Seaman, you’ve got nothing on this. (Sorry, Leonard Nimoy.)
Yes, it seems Peter Molyneux’s latest project uses Project Natal to simulate interactions with a kid. This does start to make me wonder if – as “realityengager” wonders in CDM comments – we should just go out into the real world and interact with that. (Daddy? Why won’t you play with me any more? Why are you only playing with Xbox 360 Milo kid?) But as a tech demo, of course, it’s mind-boggling – and it’s nice to think what it might mean for storytelling.
See the video at top. Molyneux suggesting that even science fiction hasn’t written about this sort of technology is especially absurd, as it seems science fiction spends most of its time writing about exactly this, but you get the point.
I just want Project Natal support in XNA so artists can play with this stuff. Hear us, Microsoft?