Visual programming or “patching”, producing custom software by connecting on-screen objects with patch cords, until recently had only niche appeal. The domain largely of academic computer musicians, patching was scoffed at by computer science departments and unknown to everyone else. Lately, though, something very strange has been happening: this technique, popularized by experimental music synthesists, is being reborn in the Web age.
Patching for the Web
Patching software goes back to the 1980s, best known in its incarnation in Max (later Max/MSP, later Max/MSP/Jitter and Pure Data), software for making music and multimedia. Max is well known in these parts, not so well among the general public. But its basic patching metaphor, itself inspired by early hardware synthesizers like the Moog and Buchla, has filtered into other software.
First, Apple quietly acquired the developer of a little-known live visual/VJ app called Pixelshox, transformed it into a new app called Quartz Composer, made it part of the Mac OS X developer tools, and made it central to their UI efforts. One day, a tiny VJ app with a cult following, the next, central to Cupertino’s OS strategy? Interesting.
Quartz Composer didn’t exactly take the world by storm, but it did update Max’s taking on patching with a pretty, zoomable patching interface. Someone must have noticed, because more recently mighty Yahoo unveiled Yahoo Pipes, an online tool for creating Web remixes, with an almost pixel-perfect, color-exact clone of the QC interface. (Imitation is the sincerest form of Web 2.0, apparently.) Microsoft’s Popfly tool is a bit different, but even it uses an object and patch-cord metaphor.
Lily, JavaScript Patching
Enter Lily: Lily is a JavaScript-powered patching environment. Lily-created software can run standalone, as a Firefox plug-in, or even in a browser. Much of the functionality is Web-focused, as you’d expect: modules for mashing-up data from Amazon, Flickr, Wikipedia, Yahoo, and the like, widget support for popular JavaScript library elements, SQLite database storage and file system access — all good stuff, but primarily Web-based.
Where things get especially interesting is that Lily has some multimedia support:
- Graphics: SVG, canvas elements.
- Multimedia: Audio and video file support.
- Connectivity: OpenSoundControl support, which in turn could be used to connect to tools like Max/MSP, Flash, Processing, Reaktor, Traktor DJ, and others.
- Custom modules: Code your own modules in JavaScript.
There’s even a demo of Lily being used as a Beatbox.
So, will there be a Web-based Max killer? Quite the opposite. As patching interfaces spread, they only serve to validate the idea of patching in the first place. Web developers get introduced to the concept, and meanwhile, musicians schooled in Max have new, more accessible Web tools. Lily could be useful for Web-based, simple patches that complement software like Max and Reaktor: speedy beat calculators, or, via OSC, perhaps even control apps. (Hmm… your Flickr stream, controlling a Max/MSP patch?) And meanwhile, this also validates the Max approach of blending patching with coding in languages like JavaScript and Java, both of which are possible in the current release. We also have a new version of Max to look forward to, as confirmed by Cycling ’74, which should continue to expand on these ideas.
Lily looks really interesting, but I’d like to see more. The feature set in JavaScript is woefully limited compared even to Flash, let alone what’s possible in Java. It’s too bad these other environments haven’t developed toolkits that allow patching for modular combinations, but still let you easily code when you like. Even Max falls short here: much of Max’s capabilities require messy coding in C, so once you get to that point, you probably don’t want to be using Max unless you’ve got a Cycling ’74 paycheck coming. (On the Pure Data side, even, where coding is open source, efforts to integrate Ruby and Python have proven less fruitful for multimedia than simpling coding in something like Java or C#. And all of these tend to separate coding from patching; we haven’t seen a good combination.)
Lily is still in development, with a pre-release beta coming soon, so we have plenty of time to think about what it’s for. But one thing’s for certain: patching is here to stay.
Thanks to Ronnie at the wonderful Rekkerd.org and to Marco Raaphorst, who has a write-up in Dutch.