Ed.: Many of Snow Leopard’s improvements – new, under-the-hood enhancements for 64-bit and multithreading – don’t impact visual creation right away. But significant changes to Quartz Composer could be the most useful, most immediate reasons to look at the latest version of Apple’s OS. Resident Mac guru Anton Marini looks at those changes for CDM.
Quartz Composer, the interactive visual tool included free with the Mac development tools, has intrigued visual creators since its release in Mac OS X 10.4. (Some VJs even used its pre-Apple predecessor, Pixelshox. But it’s been the slow maturation of this tool, which uses “nodal” visual patching metaphors to assemble 2D and 3D interactive visuals, which has given it depth. Add to that the ability to call up Quartz Composer patches in the popular VDMX, and Quartz Composer is a must-follow tool on the Mac. Here’s Anton with more. -PK
There are elements in 10.6 Snow Leopard that are relevant to visualists, including Quicktime X, Open CL, Grand Central Dispatch, and general performance optimizations. But one feature to which many visualists have been looking forward is the new Quartz Composer 4.0 (included with the XCode tools). If you’re not familiar with Quartz Composer and you perform visuals with a Mac, check out:
Apple Developer Documentation
Quartz Composer tag on CDMo
Aesthetics and UI
The first thing you will notice is that Quartz Composer 4.0 (QC4) sports a newer, sleeker, more pro look. As Apple puts it: “The Quartz Composer graphic user interface has been redesigned to make you smile.” Quite a few patches in QC4 have been trimmed to make your compositions waste less space, and “snap to grid” helps to align outlets so your patch cords are more tidy, which makes for faster comprehension and easier debugging. You’ll appreciate this if you make large spaghetti-like patches; it really helps.
There is an updated Library panel so you can get to items faster, drill down to specific groups of patches easier, import and export macros, and generally get around more smoothly.
Apple has also streamlined the use of “Virtual Macros.” Think of these as compositions you re-use in other compositions, like re-usable ‘abstractions’, or perhaps classes in object-oriented programming. Many of the new objects you will see in the Library are actually just compositions, so you can open them up, edit them, and learn via example. This makes a lot of what’s new functionally transparent, which is great both for users who want to learn from example and power users who want to dig deeper and tweak existing compositions to meet their needs.
My only real complaint with the new color palette is that it loses some of the clarity of the older QC 3.0 interface in distinguishing between different types of patches. Consumer, provider and processor patches were more easily recognized in QC 3.0. This makes it easier to get lost in a sea of mostly black patches, which counteracts some of the gains in being able to organize the graph.
The new big thing: OpenCL, Meshes
If there’s one killer feature in Quartz Composer 4.0, it’s this: Quartz Composer now includes a new 3D model importer by default. The importer loads Collada (.dae files), and handles them with a new “mesh” output port type. Ed.: For those of you new to QC, that means you can route 3D geometry into your patch, a bit like patching sound signal in a synthesizer. This is in addition to Quartz Composer’s now-standard image, structure, color, string, and number data types, and their corresponding port types. (Yes, Apple, visualists are still awaiting a real audio port capability.)
What does this mean? Out of the box, you have industry-standard 3D model importing with texturing support. The Collada format is quickly catching on as a standard 3D interchange file format, so it’s safe to say you will be able to export from various 3D modeling applications to Collada for use in QC4. Ed.: As it happens, in addition to the existing OBJ importer, there’s work on a Collada importer for Processing. Just saying.
Dragging a .dae file to the canvas of an open composition creates a new Mesh Importer object, as pictured in this video:
Quartz Composer 4.0 Mesh Importing from vade on Vimeo.
Since there is a new outlet type for mesh, you have to patch the mesh output from the importer into the mesh input on the new “Mesh Renderer” patch – simple enough. This will display your rendered object with textures automatically. The only catch I have noticed so far is that any skinned, weighted, or animated Collada files seem to fail to load. If you have Collada files with built-in animation, these appear to work.
(If you say mesh aloud as many times as I have recently, it starts to sound funny. Try it.)
The great thing about this new mesh datatype is that you can dig into the geometry and texture data of your loaded mesh, or you can make your own vertex, normals, texture coordinates, indices and texture images from Quartz Composer structures. This allows for native generative geometry. With the new OpenCL support in 10.6, you can use effects (called “Mesh Filters”) on mesh objects and twist, bend, and warp meshes. Mesh filters work on meshes you make yourself or those you load from Collada files. You can also extract the loaded textures from a mesh as a Quartz Composer image data type and manipulate it as you would any still or video frame, or replace the texture on the model completely with your own. That’s huge. Ed.: This capability is certainly present in other tools, as we’ve seen in the work of people using tools like Processing, Pd/GEM, Jitter, and vvvv, but of course, it’s great to see it in Quartz Composer’s unique workflow and interface, and with its unusually strong connections to the Mac OS. -PK
Here’s a video showing how you can extract a texture from a loaded mesh, affect it, and then send it back to the mesh for rendering. Note that the model I’m using has multiple textures mapped to different areas of geometry, which is why you don’t see the effect applied to the whole model (or every texture). This can produce some great effects if you use your own models and swap out various different textures on the fly.
Mesh Texture Manipulation in Quartz Composer 4.0 from vade on Vimeo.
The lighting patch has also been updated to add the ability to render shadows on your geometry, including, of course, the new imported meshes. This works as advertised, even on my older ATI x1600, but it adds significant overhead to your framerate depending on the quality of shadow you want.
Here’s a lovely offline render (read: non-realtime) from Quartz Composer using the new Mesh and Lighting features, as produced by George Toledo:
shadow test from George Toledo on Vimeo.
Mesh Filters, like many other new processor patches in QC 4, tend to be just loaded compositions. This means that to Quartz Composer, a Mesh Filter effect is just a regular old QC composition that adheres to the “Mesh Filter protocol,” loaded into your current composition. What that means is that you can dig into the guts of those compositions; they’re viewable and editable. You can look at the OpenCL code used and edit to your heart’s content; just remember to back up the original. The Mesh Filters all live on top of the new OpenCL Kernel programmable patch, which is a new patch that lets you – drumroll – write OpenCL code. Ed.: OpenCL is a new, cross-platform standard for writing “parallel” code across CPUs and GPUs, developed by Apple in collaboration with the GPU vendors and with support across all operating systems planned from ATI and NVIDIA, including a robust implementation in Mac OS Snow Leopard. See the official documentation at the Khronos Group, but since that’s not really in English, this is definitely a subject for another article. -PK
The OpenCL Kernel patch joins the Core Image Kernel, GLSL Shader, and JavaScript patch as the only objects in Quartz Composer that allow you to use text-based programming rather than visual programming to make effects or logic.
The bad news is that OpenCL Kernels, and thus Mesh Filters, did not work on my system (Mac Book Pro 2.33Ghz with ATI x1600). That struck me as odd, as my understanding of OpenCL is that it should be able to fall back to the main processor should the GPU not be able to cope (and my x1600 is definitely not supported). I’ve confirmed with Apple that this is a bug, and that Mesh Filters should indeed render on your CPU if your GPU is not up to snuff. OpenCL and meshes are new, so expect a bumpy ride for a bit.
A quick note about 3D modeling for real-time visuals: if you are making your own models, you probably want to look at some game development tutorials, as the lessons learned there (keep vertex count down, hide details in textures not geometry, etc), will be equally helpful in any VJ app that supports 3D model importing. This will help keep your frame rate high.
Gestures, Interaction, Forces, Feedback
QC4 introduces a slew of new objects for interaction that can allow for some potentially very interesting creative use.
Quartz Composer 4.0 Interaction, Inertia patch from vade on Vimeo.
Okay, so maybe my video example wasn’t amazingly creative, but it does show the new interactive hit testing for arbitrary objects. Just attach the Interaction object to any consumer object, and you can interact with it in the render scene by clicking and dragging.
QC4 adds gesture input support for the Mouse and Interaction patch, allowing you to listen for the Swipe, Rotate, and Magnify gestures, which use two fingers on a supported trackpad. Ed.: Recent MacBooks and MacBook Pros have this functionality. Combine those with the new Feedback, Inertia, and Force patches, and you get some really nice dynamic physical interaction in your compositions.
Similar to the Mesh Filters mentioned above, there are new Spring, Clamping Springs, and Directional Spring patches. These are just Virtual Macros, so they’re user-editable and endlessly tweak-able.
Meshes and classic 3D consumer patches can now apply and interact as physical systems, responding to interactive gestures and complex hit detection. Sounds like a lot of fun.
Performance
QC4 also advertises performance optimzations. To test this, I ran some movie benchmarks as well as standard example compositions included by Apple in the developer tools on the same system in 10.5/Quartz Composer 3 and and 10.6/Quartz Composer 4. The results: Quartz Composer 4.0 does indeed deliver on average higher overall framerate for many usage scenarios.
I chose compositions for testing from the various examples and tested full screen (1440×900). I ran the same composition file from QC 3.0 / 10.5 in QC 4.0 / 10.6 to make sure that the compositions themselves were identical.
I tested Movie playback with some basic effects; the input movie was compressed with Photo JPEG, 24 fps, 854×480 resolution, using the default Movie Importer on ‘Asynchronous’ setting. In these tests I do not crop the output movie, so Quartz Composer and Core Image properly convolve the input image and make the output image size much larger than the input size. We are talking images around 3030 x 1700 pixels. Sounds like a good stress test to me. Ed.: I read this too quickly and saw what looked like low framerates before realizing these were epic resolution – so an excellent benchmark of performance which can translate into much higher framerates when you crop to a reasonable output. -PK
The results:
- Movie with Gaussian Blur; QC 3.0: 18 fps, QC 4.0 18 fps
- Movie with Zoom Blur; QC 3.0: 10 fps, QC 4.0: 12 fps
- Movie with Motion Blur; QC 3.0: 15 fps, QC 4.0 20 fps
- Movie with chain of 4 custom effects: QC 3.0 60 fps, QC 4.0: 60-65 fps
Individual developer example compositions provided by Apple:
Club Lisa (music visualizer, feedback, particles, iteration, 3D)
QC 3.0: ~30 fps, QC 4.0: 30-35 fps
Depth Computation (heavy custom core image kernels, 3D)
QC 3.0: 20-30 fps, QC 4.0: steady 30 fps
Blazed Trail (feedback)
QC 3.0: ~30 fps, QC 4.0: 45-60 fps
Psychotic (image caching, core image effects, many chained effects)
QC 3.0: 30-45 fps, QC 4.0: 35-45 fps
Blurrier: (many chained effects)
QC 3.0: 45-60 fps, QC 4.0: 60-65 fps
Noise 3D (Iteration, 3D)
QC 3.0: 30 fps, QC 4.0: 25-30 fps
Smooth Rotation (Animation, Iteration, 3D)
QC 3.0: 20 fps, QC 4.0: 18 fps
Wavlet (very heavy custom Core Image Kernels)
QC 3.0: 6 fps, QC 4.0: 10-12 fps
Julia iteration (Fractal, Core Image Kernel)
QC 3.0: 20 fps, QC 4.0: 20 fps
Now, this is really more of a benchmark of my older ATI x1600 than it is Quartz Composer, but it does look like QC4 has improved the GL pipeline in some cases, especially with feedback. Chances are that Apple paid more attention to newer machines with more modern cards than my older system, so you may see more of an improvement on newer hardware. All in all, QC4 edges out 3. Ed.: And likewise, whether this is the result of improved drivers, an improved render pipeline, Quartz Composer, or some combination this is a clear reason to upgrade to Snow Leopard if these are things you’re doing. -PK
Another note is that certain objects known to be relatively poor performers in QC 3.0 have gotten a pretty significant update. Most notable is Render In Image, which now appears to use a much more modern back-end rendering system, since it no longer needs to support non-Intel Macs with older graphics cards. This is a huge win: combined with the new mesh rendering, you can do full-scene post processing of your 3D models. Just throw your meshes into a Render In Image object and don’t be worried about potential slowdowns you used to get.
Hidden Preferences
Holding down option and selecting preferences in QC4 gives us the fairly well-documented “hidden preferences”, similar to QC3. While it’s probably not supported, one major addition to QC4 is the multisampling option in the editor preferences. This adds multisample antialiasing if enabled on Quartz Composer’s output window. Note that not all Intel systems have GPUs that support multisample antialiasing, and that this applies only to the actual Quartz Composer editor application. Enabling multisampling does not magically mean any Quartz Composition opened anywhere is antialiased. However, it’s still useful, especially to those rare and brave souls who VJ straight from Quartz Composer.
A sample frame without Multisample Antialiasing (click to view full size):
With Multisample Antialiasing:
A simple checkbox makes a world of difference: it adds visual fidelity and removes ugly aliasing artifacts.
Compatibility, Stability, and Final Thoughts
While Quartz Composer is marked as a 4.0 release, in many ways, it feels like 1.0. With a new look and major new features there may be more bugs than the obvious ones I noticed here. QC 4.0 seems a bit more willing to crash than 3, and while that’s not so bad for programming, it’s not good at all in the middle of a set.
Quartz Composer also now runs as a 64-bit application by default. This is consistent with 10.6’s move to 64-bit, as advertised by Apple, but brings some potential complications. Some third-party plugins may not load under Quartz Composer in 64-bit mode. A 64-bit application in 10.6 cannot load a plugin bundle that is compiled as 32-bit only. Fortunately, Apple’s normal plugin template for XCode compiles plugins both as 32- and 64-bit, but there are cases in which some things may just not work as a 64-bit plugin. To get around this you can choose ‘Get Info’ on the Quartz Composer app and select “Open in 32-bit mode”, and your 32 bit plugin will load as usual. This worked for some of my plugins.
This is especially an issue with any plugins that leverage lower-level QuickTime operations, as limitations in QuickTime in 10.6 do not allow certain types of code to be ported to 64-bit. My custom movie player plugin is affected by this, unfortunately, and due to some complications, may be functionally limited to 32-bit. If you have installed a plugin and it won’t run, try opening QC4 in 32-bit mode and see if it loads; you’ll probably be pleasantly surprised.
The good news, however, is that most plugins should work out of the box with QC 4.0, and few existing plugins use APIs that are not fully supported under 64-bit. Kineme, one of the main sources of amazing plugins for Quartz Composer has already released a statement regarding 10.6 compatibility. Basically, many things work, some things don’t; stay tuned in the next days for updates. I can say the same for my plugins as well. Many of them work, some don’t, expect updates; stay tuned!
All in all, Quartz Composer 4.0 is a compelling upgrade. There are tons of new features I didn’t cover here and once the bugs are worked out (I’m looking at you, OpenCL), it’s going to be a mandatory upgrade.
For a full rundown of additions and changes, see Apple’s Release Notes for Quartz Composer 4.0. [PDF]
And if you have 10.6 and Quartz Composer 4.0 installed, you can explore a ton of new compositions located in /System/Library/Graphics/Quartz Composer patches, as well as conceptual examples in /Developer/Library/Quartz Composer/Examples.
Have fun, and happy patching!