Take a break from the surveillance nightmare around us for computer vision that’s local, not online, and doing creative, artistic things, instead. Resolume 7.28 added some powerful Wire nodes and new effects and blends, all powered by vision. Yep, Resolume’s removing your backgrounds, segmenting your humans, estimating depth — and it’s good, clean fun.
Underneath it all are two powerful Wire building blocks. One is Human Segmentation, as in segmenting the image (though, sadly, segmenting the humans sounds magical — also I would like to be a Decepticon)! The second is Depth Estimation.
With just those two building blocks, 7.28 gives you a bunch of higher-level effects and a new Depth Blend mode you can use out of the box. They’re all built with Wire, meaning you can pull them apart, learn from them, and mod them. Or create your own effects. Plus I imagine that with these tools in 7.28, we can expect more, given the Resolumian update cadence.
New effects:
- Background removal (maybe more accurately human segmentation-based background separation, which you can use as its own mask for any purposes you like)
- Depth estimation (based on a 2D image, so no additional camera needed … ah remember hacking Kinect?) — again, that gives you a depth map you can use as you please
- Depth blur, which you can think of as faux depth-of-field, though can also be used creatively
- Depth field — really quite beautiful, combining Ghost and Xray; this one in particular seems ripe for modding
- Depth displace, which uses depth as a displacement warp
- Depth pixelation, which uses it to pixelate stuff
- Depth shift, slicing the layers of the image based on the depth map
… and the Depth Blend Mode mixes by depth map rather than opacity.
Also new in this update:
- Group FFT
- MIDI and OSC monitoring improvements
- Right-click resets to Default in Wire, just as it does in Arena
And next:
We’re currently improving our audio engine and the first thing is FFT audio reactivity in Arena. In Wire we’re implementing a whole lot of new audio spectrum features and path drawing so you can build audio reactive visualizations in Wire. And the next release (7.29), will also include a few new slice effects so stay Tuned!
The funny thing to me about this is how many tricks we had to use to simulate the effect using basic (often CPU-bound) processing. I did a whole dance piece in Jitter years ago based on making a difference image (for movement), then using that as a displacement map, then working with the dancer so that the motion simulated depth. Now, that being said, look at the number of creative, artistic possibilities Resolume has already used — and you could easily experiment with adding other effects. All the many people with that background are likely to approach Wire and immediately come up with some new ideas.
Media art and surveillance dystopia
Media art can’t exist independently from the growing surveillance around us — but it can provide contrast (and sometimes criticism).
Looking at these examples, you can see why creative use of computer vision is fundamentally different than surveillance applications. You’re ultimately playing with human perception; the computer vision component is only there to give you some intuitive ways of working with the pixel data. And it will immediately have an aesthetic, which you can use as-is or bend into some other direction.
Crucially, it also runs locally. And if you think about even interaction design, it needs to be opt-in, not opt-out: you create a playspace where people intentionally put themselves on the camera.
I have all of this in mind as I started today’s morning coffee with Janus Rose’s piece for The Verge, “Everything is spying on you and there’s no opting out.” Benn Jordan, long-time friend of this site and the synth world, even testified in front of the Senate this week.
Media artists can’t ignore this growing reality, though. That means you’ve got a choice. If you reference these vision features directly, you had better be prepared to work in a critical context (as we’ve seen recently at Massive Attack’s shows — more on that separately). If you use them aesthetically, then it may be more about creating a separate playspace and applying the vision processing capabilities to playful, expressive outputs — more paintbrush, less seen-through-the-eyes-of-the-Terminator.
Speaking of new technologies and what happens if you’re not reading the room:
Resolume and MCP
So, Resolume recently added AI assistance to an update — that was 7.26. I missed writing about it at the time. But as I could easily have predicted what happened next: it was hugely divisive. It was divisive enough that Resolume admitted in this week’s newsletter that “it was received with mixed emotions in the Resolume Community” and “not everybody likes AI and we get that.” They immediately go on to tout its productivity boost.
That seems to miss the point. People rejecting AI are not questioning that it could boost “productivity” or make people work faster. People are questioning whether working faster is really the objective.
Let’s look at this update’s new Wire features. Going through the effects patches, node by node, and tweaking values and experimenting and making mistakes is the whole process. “Speedrunning,” live coding, skillful patching — the ability to work quickly and efficiently is something people have done for years without the assistance of an MCP.
There are other objections that have nothing to do with being “faster” at live visuals. OpenAI and Anthropic are both implicated in defense applications that have killed people (via the US military, in particular), among other serious questions about the ethics of both companies. And there are ongoing serious environmental concerns I won’t enumerate here. I’m not here to guilt anyone about using their products; I’m typing this on an Apple computer and phone that are each full of conflict minerals. But I also wouldn’t object if people raise concerns about my Apple machine, too — we should be actively talking about it. And unlike past technologies, you also don’t own these AI tools in any way; you’re dependent on a monthly subscription and credits [is something I definitely don’t need to remind any current users]! (I haven’t looked yet at how local LLMs might be used with Resolume; that’s an interesting angle.)
This may sound like I’m telling people not to use the MCP. That’s not it at all. I’m suggesting we need a discussion that goes beyond “but it makes me faster.”
Part of the beauty of Resolume is how open it is and how many different kinds of artists use it — and on different levels. That goes from folks writing their own shader code to people who just need a tool to conveniently add some video layers, people who treat it like a video synth to those using it in immersive 3D workflows.
Anyway, here are the 7.26 announcement videos on the MCP, since I missed them before. I expect there are use cases, especially on-site where you need to work quickly, where these are useful and do give you more time to get back into creative stuff. Patching environments have often had tools for scripting/generating patches, so there is absolutely a precedent.
Well, that was long, but I guess that’s why I avoided writing about this in 7.26. You’ve earned some videos.
This is really what I’m making arguments against doing, but in the pursuit of that, it’s good to see how people are using this — and necessary, indeed, for anyone learning or teaching patching, since it is part of our new normal:
I would also be remiss if I didn’t point out, each time this comes up, that the MCP is totally evil.
But all that said, I’ve talked about this with other visual developers. Developers were curious to experiment with these facilities to test them. There’s also the possibility of using this kind of AI to intentionally mess with the code or patch and create intentional or unexpected results.
For my part, I’ll shut up, update, and get into what I can do with Wire, which involves a process that for me doesn’t fit the MCP anyway.