Technology still has the power to appear like magic. And one place we may desperately need magic: straightening out our horribly shaky, handheld video shots. Software makers like Apple have already offered up some techniques for doing this – in the case of Apple’s Final Cut Studio, optical flow analysis attempts to track the image as it shakes around the screen and compensates by adjusting the orientation of the frame. But a research team at the University of Wisconsin, partnering with Adobe, will present a new approach at the legendary graphics-geeky SIGGRAPH conference in August. They go one step further, applying a 3D mesh to the image to warp your image three-dimensionally to make the stabilization even more seamless.
Me writing about it is basically useless. Check out the mind-blowing results in the video. From the description:
In this paper, we describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method can adjust the video to appear as if it were taken from nearby viewpoints, allowing for 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.
The research, at the University of Wisconsin-Madison:
Content-Preserving Warps for 3D Video Stabilization
You can view all the techie details there, as well as many more demo videos. This is promising stuff, and we’ve seen in recent years a vast acceleration of the time between academic research and shipping commercial products — especially with cheap computational power on home computers to play around with, and increasing challenges for software vendors to differentiate what they’re doing in a mature application space.
Side note: boy, do I want to go to SIGGRAPH this year.
Also along these lines: Spacetime Fusion, tests of Final Cut’s SmootCam feature, more SmoothCam tests
For those of you purists, yes, it’s still worth considering the art of steadicam shots – at least before technology obliterates it for us clueless masses. Previously: B&H Interviews Steadicam Inventor: Shooting is Like Dancing