For all the ubiquity of Unreal Engine, a small handful of people are pushing the envelope of what Unreal can do as an audiovisual tool. With years of effects and animation production work under his belt, Zack Berwick is uniquely creative and inventive with the combo – and he’s sharing his approach, using Ableton Live, Max for Live, MIDI, OSC, and UE.
What I love about this is how many unconventional creations Zack is using. Now, nothing against some of conventional particle system techniques or flashing lights – sometimes that’s exactly what you want! But being expressive and making an emotional connection requires a greater range. So here, we get full lighting rigs that act as realistic environments, all the way to moody pools of water and some floating rubber duckies. This one today is my favorite so far:
Artistic experiments
Please, book Zack (he’s in Toronto); this show sounds fun. He’s pairing his Ableton Live rig with the beloved Dirtywave M8 tracker/synth, and there’s some great music (at bottom).
A fascinating workflow in Unreal is to treat its virtual lighting like it is an actual lighting rig – you can absolutely rig up Unreal with DMX and use it to prototype IRL shows. But here is Ableton Live (with the M8) controlling lighting in a virtual warehouse:
You can see what happens when a beautifully constructed set is illuminated live:
Or here’s a moody tunnel and pool of water. Lighting effects make the music reactivity, though there would be ways to take the same concept and develop it in another direction:
Oh yeah, and here’s one for fans of Severance (and The Stanley Parable):
When you are up for some more particle effects / splash, here’s some good M8 fun:
There’s plenty more in that channel if you dig in, so I’ll let you do that. But let’s talk about what’s going on here…
How-to
Unreal Engine has native MIDI support, so you could route MIDI between Ableton Live and Unreal. But Open Sound Control (OSC) is more flexible. Since Ableton Live (cough) still lacks native OSC support, that means digging into Max for Live:
Then on the Unreal Engine side, it’s time to use the power of Blueprints to formalize how you receive OSC data on the other side. (Actually, I might try adapting this approach to native MIDI so you have a quick answer to MIDI input, even having said what I just said about OSC vs. MIDI.)
From there, Zack also added reactive audio input response (and Tom Hall’s envelope follower Max patch, which is a great tutorial dive into gen~):
By the way, on that last point, you could even imagine using RNBO and Unreal for a native MetaSounds tool that would do this inside Unreal without any add-on like Max for Live. I know at least one Unreal Engine developer taking that approach. It is more work, but if someone wants to take that on, I’d be interested.
Now Zack has been gradually building up his Unreal Engine visualizer on GitHub. There are breakdowns of each interesting new animation with downloads, but the main visualizer GitHub is still probably a place to start as you follow along:
https://github.com/ZackBerw/Unreal-Engine-Interactive-3D-Visualizer
And see also the Max for Live and Unreal Engine Blueprint downloads:
These projects, in turn, were inspired by the Sem and Tris AV Club, who did some early work in connecting Unreal and Ableton Live via Blueprints:
Once you grasp the basics, Zack has gone into detail in breakdowns. He also says he has synesthesia, which I always find fascinating because people tend to have strong sensory reactions, but they also often differ wildly from individual to individual. (I was fortunate enough to visit Scriabin’s apartment, where over a century ago, he had built a DIY color organ by wiring up colored lightbulbs to a keyboard. There’s a direct line from that invention to these creations in Unreal – even as the technology seems radically different.)
What’s great about this level of detail is that I could imagine someone pursuing a very different aesthetic by changing the objects, rendering, and design. Putting the information out there is a challenge to folks to find their own path and push this medium forward.
Here’s the light triggering approach itself:
Music
Visualizers are ultimately only as good as the music, so the great news is that Zack’s music is spectacular. Get some deep, dubby, dark ambient, a kind of lo-fi/chillout-industrial crossover, via his Caustic Clouds project. It’s just the thing for a soundtrack to working in Unreal Engine, which is welcome because if I listen to my own music, I’ll get sick of it. So he’s unintentionally given us a “soundtrack to Blueprint” and a perfect audiovisual world with his visualizer.
Here you go – it’s all pay what you like, so throw this artist some cash! (Hmm, it’s Toronto, so I want to say – Peameal Bacon dollars? Butter Tart budget? I’ll shut up and let the locals correct me.)
Me, I still have those rubber duckies on the brain. So here, as a bonus absolutely no one asked for, “Rubber Duckie” in Arabic:
And “Do de Rubber Duck/Ulica Sezam – Pačji ples,” in Hrvatski/Croatian: