Using video in Processing is, sadly, really painful. You can do absolutely wonderful things once it’s working — pixel-by-pixel manipulations that are hard to do elsewhere, and easily-coded OpenGL manipulations that should help generate powerful eye candy. But the list of issues runs something like this:
- Windows doesn’t support capture without the addition of the buggy WinVDIG, which often doesn’t work properly.
- Linux doesn’t support anything.
- Capture is slow on Mac, and sometimes doesn’t work.
- QuickTime updates regularly hose the whole setup.
- Important QuickTime features aren’t supported.
- Playback is slow.
- Playback often crashes.
- Things just don’t work, and you don’t know why.
Now, granted, for simple sketches and experimentation, the library often will work. Or at least, it works except when it doesn’t — and you can read about how often it doesn’t in the reference.
Fortunately, I think the situation could be — and soon will be — very, very different.
The Reason Why
Here’s the important part many people miss: none of this is Processing’s fault, and all of it is theoretically fixable. Processing is based on Java, and for video support it uses QuickTime Java. Generally, video libraries in Java aren’t actually even Java; they’re just native libraries that have been supported via Java APIs. So, if it’s not Processing and it’s not Java, then it’s — you got it — QuickTime, or specifically QuickTime Java (qtjava). QuickTime Java has been dead for some time and while Apple’s rule seems to be that anything you might want to know is covered by an NDA, suffice to said it’s been recognized to be officially dead as of this year. It’s been decaying for a couple of years, but we finally got a (confidential) death certificate. Sun, for its part, never supported QuickTime Java, supporting instead something called Java Media Framework which it never finished, but which had the power to boldly not work on multiple platforms. Sun and Apple were never interested in a solution, Microsoft doesn’t really need you to use Java, and Linux people until recently ignored Java because it was proprietary. Result: a mess that left the Processing team using the one and only possible evil.
Thankfully, it doesn’t have to be that way. Open source video libraries haven’t gotten mainstream recognition because of gray areas in patent law, but technically, they’ve been leaping forward. (In fact, forget about Java for a moment — as you may already know, open source tools like VLC and ffmpeg are often more reliable and perform better on both Mac and Windows than native libraries from Apple and Microsoft. Viva le open source!)
In theory, we should also see improved video support from Sun’s JavaFX API. I’ll believe it when I see it, though, and the current status is that it can do playback on Windows only when it’s not itself crashing — and it appears to rely on external open source codec support anyway even to do that. We’re also waiting on it being exposed as Java classes, so until that happens, forget I mentioned it.
Testing GSVideo
You want something now. You want it to work. You want it to play back and capture reliably. You want high performance. And you want it on all operating systems. This is perfectly reasonable to want, because we have wildly powerful computers that are certainly capable.
Andrés Colubri’s library GSVideo seems to be the best answer for that moment. It’s designed to act just like the internal video library in Processing, so the syntax and operation is absurdly easy. But it relies on libraries that actually work. And best of all, you can couple it with his GLGraphics library to keep processing on the GPU for higher performance. (There are even HD video playback examples in the latter that connect to the GSVideo library.)
So, now you have an introduction to why I’m so excited to have this library to test. I’ve started playing with it now, and intend to test on Mac OS X, XP, Vista, and Linux by the time I’ve done. Building on Windows looks like a bit of a pain, but Windows does have very solid capture support native to the OS, so I think it’ll all be worth it. Linux looks easy, Mac OS X looks fine. I have done enough testing of GLGraphics on a MacBook to know that you’ll want a recent dedicated video card for OpenGL texture support, but then, regular readers have already discovered the non-Pro MacBook to be fairly useless for visual work.
Will You Join in My Crusade?
Here’s my question: who’s with me? Anyone else want to try this out? There’s not much action on the Sourceforge forums and whatnot, so I actually wonder if we’d like to start a small discussion and testing group and miniblog here on CDM?
If you’re interested, say so in comments. Leave your email — it won’t be published, but we can see it privately and keep you posted.