Developer Anton Marini, aka vade, is bringing back the legacy of Quartz Composer and proving Apple’s visual APIs are ready for real-time. Have a look—and watch what you can do with 3D Gaussian splats (if you know you know).
Splat goes the Metal
3D Gaussian Splats are a novel way of capturing and real-time rendering 3D environments – see 3D Gaussian Splatting for Real-Time Radiance Field Rendering. You can capture them on most recent iPhones with the superb app Polycam; those developers also have a JavaScript Web browser interface. But rendering is intensive. You want your renderer to be optimized for the display hardware.
Enter MetalSplatter, a free and open-source (MIT-licensed) tool by vade built in Swift/Metal to load any PLY file and visualize it in 3D on macOS, iOS, and visionOS (with stereo rendering support). If you’re a developer, that will lead to various other useful tools (and not only for Apple platforms):
https://github.com/scier/MetalSplatter
But this is just one thing vade is doing proving the possibilities of Metal.
And the shape of things to come
Once upon a time, there was Quartz Composer, originated by developer Pierre-Olivier Latour (under the name PixelShox Studio) before Apple came knocking and the software was reborn as an official Apple tool. This free node-based tool lived on equal footing with other Apple developer tools, both as a way of patching interactive visuals and as a standalone live visual/VJ tool. Even if you have never used Quartz Composer, it has clearly influenced the recent evolution of node-based tools and helped advance Apple’s proprietary 2D/3D/video APIs.
Quartz Composer was abandoned after its 2016 release, and Apple’s API plumbing has been totally transformed since the days of OpenGL, Core Image, and Core Video. vade (v002, Syphon) has always been passionately devoted to finding his way around Apple’s internals to squeeze the maximum performance and creativity for artists. So he’s been working on a new tool called “v” – part live coding, part node-based graphical development tool, all live visuals, for macOS, iOS/iPadOS, and visionOS.
It’s worth a look just to see how well Metal is running when someone actually targets it directly. I love that interface.
“v”is slowly getting there.
— Anton Marini (@_vade) July 18, 2024
120 FPS home made node based video mixing live coding setup using Metal via Satin (@rezaali) and @lygia_xyz (@patriciogv) as a SwiftUI playground.
THEY CANT TAKE THIS FROM ME. #quartzcomposer RIP pic.twitter.com/OGvK99GAfC
He’s also been working on enhancing the open, cross-platform Lygia shader library for Metal optimization. Oh, and that brings an important point—a lot of Metal development will still rely on open-source, cross-platform frameworks, even if they target Metal optimization specifically for delivery.
But look at this:
Added some additional metal support @lygia_xyz and now we have blurs that are so much faster than Core Image cc @patriciogv @rezaali pic.twitter.com/Bg7eqURu1T
— Anton Marini (@_vade) August 13, 2024
Good stuff. LYGIA, meanwhile, deserves a post of its own – it’s fantastic and already sports integrations with Unity, Unreal, three.js, Processing, OpenFrameworks, TouchDesigner, and many more, across all modern OSes and architectures.
Create Digital Motion, y’all.