The ultimate compliment to a visualization of music would be giving the unseen structure, changing the perception of the music not only by granting it visual embellishment, but producing a new experience of the composition that fuses sight and hearing.
Abstract Birds, working with French new media organization Arcadi, legendary Paris digital research center IRCAM, and Paris’ talented Ensemble Intercontemporain, realized visuals for a challenging work by Russian composer Dmitri Kourliandski. Kourliandski’s score is primarily timbral, with a densely-clamorous first movement and quiet, percussive second movement. The challenge for Abstract Birds was providing a visualization that brought out the syntax of his timbral language.
To do so, the visualization isolates each instrumentalist. 24 independent audio signals, one for each player, are analyzed separately. Developer Thomas Goepfer worked on the audio analysis; visual renderings are then produced in real-time using the software vvvv. Nothing is pre-rendered; the work is genuinely real-time, pulsing and vibrating as the 24-person ensemble plays.
Those musicians were absent when I saw Les Objets Impossibles at Montreal’s Elektra Festival; only the visualists and their software were able to cross the Atlantic to North America. But Montreal’s Usine C, a bunker-like reclaimed industrial space, was a fitting environment for the work, cold cavernous concrete extending into real space the virtual architectural spaces on the screen. It doesn’t do justice to the fidelity of the visuals to see it on a small computer screen in video form; the live 3D, spanning a proscenium projection surface, feels immersive. Incidentally, it did so in the way that cinema does, dizzying enough without any 3D glasses. (See my previous story pointing to Walter Murch’s rant on stereoscopy; here again was a case where the eye and brain were able to understand depth without having to literally project that depth in the viewing apparatus.)
Les Objets Impossibles produces a hyperreal imagined architecture. On a big screen, the viewer feels confined in some interior space, the camera shifting about in the way the mind’s eye might navigate mental architectures. Objet 1 is reminiscent of theatrical flyspace, exploded in extra dimensions, animated with the vibrations of sound. Music that otherwise would have seemed harsh and chaotic is lent space, pattern, and depth by the visuals, providing the sonic impression of a rattled cage. Recursive grids, caught in flashes of a spotlight, create an experience of inner logic.
The second movement accompanies a sparser, softer soundtrack with, a taut, tangled nest, of visuals. It appears as though an alien insect hive has evolved from wood. Tight as an upper-register piano wire, the subject of the second movement is more of an object, a kind of play on tensegrity. A triangle mesh bubbles out like resin. The subdued score squeaks and thumps, and from those naked timbres, the visuals actually enhance the ability to hear, making what might have otherwise been inscrutable experimental sounds seem strangely inevitable.
I talked to the Abstract Birds team about the behind-the-scenes process. The visuals themselves found their way through a customized pipeline of renderers – normal maps, specular reflections, dimpled surface textures, crisp light sources, and multiple shader render passes including glow and depth of field lend the results a polished look. Many viewers with whom I spoke were surprised that this was all real-time.
The audio reactive techniques, though spread across those 24 channels, were fairly simple; it seems mostly amplitude response was enough without any fancy spectral analysis.
Here’s what the creators say in their own documentation:
We created a realtime sound-reactive software to visualize the two pieces composed by Dmitri Kourliandski and performed by the Ensemble Intercontemporain.
IRCAM used 24 microphones to get a separated audio signal from each instrument of the ensemble.
Thomas Goepfer created a software which analyzes each signal and extract the sound information; the resulting data are sent to our software and used to animate the elements of the visual composition.We interact in realtime controlling several parameters of the visuals and directing the camera movements.
We emphasized the strong dynamic and timbric contrast between the two musical compositions by finding different kinetic and material properties for the two visual counterparts.
The monolithic structure of the pieces lends itself to an architectural / sculptural representation, in which the elements are animated and recombined changing the general order but still maintaining the global identity of the work.
The observer is forced / led to change the point of view, explore the object discovering its different aspects and find out that in breaking the perceptual scheme lies the possibility of a new vision.
http://www.abstractbirds.com/les-objets-impossibles/
Previous coverage of Abstract Birds:
Partitura: Spectacular Real-time Visualization of Music, and Thinking in 1D