Palm VJs and visualists? Imagine having HD digital video sources you can fit in your pocket, outputing to HDMI. That’s the vision already coming to realization with a new generation of mobile graphics chips. And it’s not just on paper any more, or some time in the near future. Hardware with these capabilities is shipping now.
The coming wave is a combination of two factors: one, powerful new mobile chipsets with video acceleration and basic 3D capabilities, and two, new software platforms – particularly Google Android and other Linux-derived tools – that will give you the freedom to use them.
Tegra, Handheld Awesomeness?
Take Microsoft’s Zune HD: powered by NVIDIA’s Tegra platform, the Zune is able to drive HD video playback in 720p, straight to a TV. In the case of the Zune, that requires an A/V dock, though there’s no reason other devices couldn’t deliver HDMI onboard.
Look closer at the Tegra platform, and thinks get very interesting, indeed. NVIDIA says they’re targeting the Tegra “computer-on-a-chip” platform at mobile devices that could street for as little as a hundred bucks. And the Tegra is capable of video encoding and decoding that were unheard of not so long ago on desktop machines – think 720p H.264 / VC-1 decode, 720p H.264 encode. Now, you would expect that Tegra and chips like it are especially optimized for certain codecs and will do more poorly with other fomats, but for live visualists, JPEG encode and decode are accelerated, too. With up to 12-megapixel sensor support, that means you could see handheld Tegra devices as live visual capture or playback devices.
The Tegra has some reasonably significant 3D capabilities, too. The ULP GPU from NVIDIA suports OpenGL ES 2.0. Note that that is ES, not full OpenGL, and NVIDIA advertises a “programmable pixel shader” – guessing that means one of them? But this could still be powerful enough to, say, run a Processing sketch on a handheld device.
Really, at this point, all we need is an HDMI mixing solution, though as Anton recently pointed out for CDM, that could even be a computer with HDMI acquisition. (See his review of the Matrox MXO2-Mini, and expect more updates on that soon.)
Free as in Freedom
Now that hardware is capable of these feats, the remaining stumbling block is whether mass-market devices force you into being passive consumers of their content, or whether they give you the freedom to output your creations. Apple’s iPhone, after all, should be capable of outputing live visuals, but Apple has crippled that capability so that it works only with applications and methods they deem acceptable. I wouldn’t hold out high hopes for Zune, either. (I do want to research, though – if you use Microsoft’s quite lovely XNA game framework and build something for the Zune HD, can you output it at 720p? If not, why not?)
For purely practical reasons – without needing to subscribe to any of the philosophical ideas around free software – you may want tools that let you do whatever you damned well please with hardware you own.
Enter the Zii EGG hardware, running on the Zii Plaszma platform, from Creative Labs. In unparalleled branding, Zii (rhymes with Wii?) is “stemcell computing.” (Ewwwww.)
Terrible branding aside, the EGG looks like a potentially amazing platform. It actually runs two separate software stacks. From Zii Labs, it runs the Plaszma platform, which appears to be a bundle of proprietary media support atop a full-blown Linux install. If we’re really lucky, that could mean you could run apps like Processing, Flash, OpenFrameworks, or even VJ apps on the device, but it’s unclear to me what the actual flavor of Linux is here, and Plaszma’s future is way too murky to analyze now.
Alternatively, the Zii EGG also runs Google’s Android – only it’s Android with massively beefed-up media capabilities, thanks to the hardware and Zii’s special version of the Android platform.
And the hardware is what makes the EGG look delicious, almost too good to be true. Want cameras? The EGG has two of them, one on the front (VGA) and one on the back (HD). Like NVIDIA’s Tegra, it supports OpenGL ES 2, but it’s capable of output up to 1080p instead of 720p. It’s got an SD slot. It has X-Fi audio processing, which could make it a nice choice for OpenAL audio. The reason for all of these capabilities is Zii’s own custom chipset for media processing.
It’s the output capabilities of the EGG that make it most drool-worthy. With both composite and HDMI output, you could hook up the EGG to a video mixer or computer and have an additional visual source – one with sensors (accelerometer), multi-touch, and the ability to run, say, your Processing sketch folder.
You can buy the hardware now – see the full specs – for US$399 as part of a developer kit. Remarkably, though, Zii wants to target an even lower price point in volume, and I think the reason this is called a “platform” is that, like NVIDIA, they hope the chipset will power various other devices. The difference is, instead of running hobbled, DRM-locked Zune software, you could run Linux or Android (or both).
Now, running the Linux kernel on a mobile device with OpenGL ES instead of OpenGL is not going to be the same experience as running a desktop system. But I expect that “desktop”-class systems will also benefit from the low-power, low-cost, low-heat, small-size revolution. I think you can also expect netbooks with more usable GPUs like the NVIDIA 9400M, in place of the underpowered Intel integrated graphics they’v
e had. Again, you’ll be able to install Linux-b
ased software stacks for visual performance.
Why Go Handheld?
So, if handhelds are more work than desktop machines and less powerful, why am I so excited? I imagine that they’ll finally give digital visualists the kind of modular, plug-and-play hardware setups audio folks have had for ages. You might tote a handheld gadget with video out alongside your laptop for use as a backup or another video source. You might get a couple of them plugged into a video mixer – especially if someone can work out how to do an HD hardware mixer with HDMI inputs, perhaps powered by plentiful new mobile chipsets like these. You might run a small installation with them, or carry one and connect it in HD to a tiny projector to show off your work.
You’ll also have a new audience of people with these sorts of devices. Consumers who opt to buy Android-powered devices with video output won’t face the limitations iPhone owners do today – they may download your interactive video art, plug it into their HD TV at home, and watch your work in full 1080p.
Now, I just have to decide whether I’m ready to risk shelling out $400 on Zii’s mystery device, or if I can convince myself – and my wallet – to wait.