Troels Folmann is one of our favorite composers at CDM. The fact that he’s a game composer both incidental and essential — it’s not that he’s scoring a Tomb Raider title that matters, it’s that game composition requires a new, fluid way of thinking about form, and Dr. Folmann (he did a dissertation topic on the subject) is up to the challenge.

Digging through recent entries on Troels’ blog is definitely a source of weekend inspiration. I’m fond of found samples, but I tend to record sound making things around the house up close with a mobile recorder for a more intimate sound. Troels drags them over to a concert hall and uses the natural reverb to turn a candle light holder and Coke bottle into something that sounds like massive, tribal percussion. To keep himself disciplined, he limited himself to objects in a random photo. Here’s what it sounds like:

To add to the ambience, he uses the Timefreezer plug-in ($99 for Mac, Windows, Mac Intel, the lot). As the name implies, it “freezes” samples of sound as an effect or instrument. I’ve done some similar things as DIY patches, but it sounds like they’ve done a nice job of implementation.

This approach to sampling percussion with natural reverb, and making an art of the samples, is part of why they pay Troels the big bucks. Be sure to hear his percussion demo for more of the sounds. Little wonder that he blogs the meditation on autism that’s been making the YouTube rounds: sampling sounds requires an almost extrasensory focus on the world around us that we spend most of our time shutting out.

So there you have some fiddling with household objects. What about this “future of adaptive music” business?

Our own W. Brent Latta covered this in some detail in a 2006 interview:

CDM Interview: Tomb Raider: Legend Composer Troels Brun Folmann on Adaptive “Micro-Scoring”

Troels follows up on that topic in a new interview with Game Soundtracks, as reproduced in total on his blog. He has this to say about “micro-scoring” — and notes the use of sliced samples, manipulated live:

So a part of my Ph.D. studies in game music related to developing new methodologies for advanced types of application of music in the game. One of my main focal points was – and still is – the development of something I call: “Micro-scoring”. Micro-scoring is essentially about breaking the score into a variety of small components that are assembled in real-time according to player action and/or interaction. The micro-scores are made in such a way that they adapt to player action or interaction. You have to imagine that there are thousands of things going on in the game environment — the idea behind micro-scoring is to support the major elements in the environment. An example can be a 3-second score for breaking pillars or falling stones, which is scored in the same key as the main ambient background score. We also have more detailed types of micro-scores which are based on slices samples like REX and other sliced sample formats. This allows us to fully adjust pitch- and timing based on player interaction with the game. An example of this is adjusting beat to footsteps and increasing tempo when she starts running. A good example of micro-scoring application relates to chopping up a score in multiple components. So essentially composing a score in 15 different steps and cutting each step up, so it can seemingly integrate into any of the other 15 steps. The system then blends the steps in real-time and you have a much more varied and versatile score – made from micro-scores. This allows you to adjust mood in music with using basic cross-fades, but actually have adaptive types of compositions. Needless to say it’s a fairly complicated effort.

Game scores had at one time been more interactive than they are now, because for at least a period around the early 1990s, PC games used MIDI scores instead of more-rigid audio. Scores are making progress back towards interactivity at those levels and beyond, aided by more powerful game systems that start to resemble the computers we use for live music production. Troels also speaks to where this future is going (and there have been some interesting developments since October 2006 when we spoke to him last):

It depends on whether studios are willing to commit to investing properly into game music. The commitment involves a variety of factors, including prioritizing audio in the production planning and a willingness to invest properly in the scoring. I doubt we will see a huge leap within the next ten years, but we will see more adaptive types of music based on principles similar to the micro-scoring methodology I described. We will also see some real-time DX/VST-based FX plugins like the integration of Waves plugins in Halo 3.

We will not see true adaptive music, since the next-next generation consoles won’t have the processing power to play a 50 GB orchestral sample library playing in real-time with 5 high-end convolution reverbs and an advanced AI that translate player action into music.

We will see more ties between motion picture, television and games – and most likely a larger degree of score usage between the media. But we also see a billion mediocre game scores and they will retain game music in a space it doesn’t need to be. Bleep, Bleep. Blob.

So, there you go — to any of you who are skeptical as far as quality, most game composers are, too! (They’re the ones, after all, most invested in seeing their field progress, while having to wrangle with mediocrity in their area, tight budgets, and tight deadlines.)

I am curious, though, about those 50 GB orchestral samples. Given that adaptive music in games could extend to all kinds of game play that doesn’t need massive orchestras in the background, let alone other venues for adaptive music beyond gaming (from live performance to installation), I’m more optimistic. This year, we’ll see Spore from EA powered by a sound engine built in Pure Data, and I expect other games from developers big to indie. The sound of 8-bit is making a comeback, as well, which will hardly tax game consoles.

With composers like Troels around, that may be a 20 MB sample of a Mexican Coke bottle that sounds better than the 50 GB orchestra, anyway. Troels, give us a ring if you decide to release your Things On My Coffee Table Sample Collection.

  • flip

    Very cool stuff. 50 GB orchestral samples? Damn…I'm probably running close to 900 GB. I know other composers who have libraries that dwarf mine as well. I'm just praying for quicker advances in solid state drives and larger capacities. Either way, that score was really nice. I've put a few metal and glass candle holders into my scores as well. No object should be off limits! 🙂

  • The next step for game audio probably is procedural audio. I'm writing a couple of papers on the subject. I guess you can include what he calls "micro-scoring" in that too.

    – "5 high-end convolution reverbs" ? why would you need 5 convolution reverbs for adaptive music unless the instruments are in 5 different rooms?

    – "he next-next generation consoles won’t have the processing power to play (…) advanced AI that translates player action into music". The current gen AI does things way more complicated than that. That's definitely not the issue.

    – "We will also see some real-time DX/VST-based FX plugins like the integration of Waves plugins in Halo 3". Well they are not VSTs but you can already develop your DSP effects if you want (it was already possible on the original XBOX for example).

  • Hey Nick — want to describe what you mean by procedural audio? Or got anything on the subject already?

    As for the 5 high-end convolutions, well, that is how some of us are used to working in our home studios because we're spoiled by current-gen PC hardware. 🙂 I agree with you on that and AI — but I do see his point as far as sampled instruments, just because the sonic content necessary for a truly generative score, versus a pre-recorded one, is at least potentially bigger. I think it really comes down to whether composers want to put their energies into things being interactive, making some tradeoffs so that can happen, or they care more about having a traditional sound, in which they'll trade off interactivity. So what he's really describing here is that not everyone will *choose* real interactivity — not that it won't be technically possible, because it's *already* very technically possible.

    As for DSP fx… well, I expect no one has time to code their own DSP for a game, so that'll come down to the features of the sound engine. You'll do it once and reuse it.

  • In the case of sfx, procedural audio is simply in-game real-time sound synthesis, with parameters coming from other subsystems of the game such as the AI, the physics and collision engine, etc… The goal here is to avoid using tons of samples which take a lot of memory and often still end up sounding repetitive. More and more studios are starting to implement it, at least for a few selected things (I did in a couple of games). The main issues are the current level of education of the sound designers and programmers in that domain, the CPU hit, the lack of models and tools as it was not a priotity for the game industry, etc…

    Basically, any audio produced from algorithms whith parameters coming from the game context is procedural audio, so adaptive music can fit in too. Although adaptive music in the game industry is very often limited to mixing a couple of streams together.

  • Hey guyz,

    First of all thanks for reading my blog. I am glad it can provide some insights and all your thoughts are of great interest to me.

    Allow me to address a few of the topics discussed in this thread.

    1. Regarding sample library size. The reference to 50GB of samples was mainly to suggest that any future platform will need to have realtime access to 50GB of samples. My total library is 16TB, but I have a constant stream of aprox. 50GB of samples on my renderfarm, so its not the size of the library – its the realtime access number. So I guess we are faced with two problems on this one. One being the sheer size of great orchestral libraries (TB) and another being the real-time streaming access to a pool of aprox. 50GB of sample files.

    2. Regarding convolution reverbs. Convolution reverbs doesn't really work for me anymore. The more I work in real halls – the more I realize the limitations of the convolutions. The first issue being that a convolution impulse is a single recording in a frozen matter of space and time, which is far from the real deal – where there is constant variation in the sound. The best example I can give is repeating the same sound over and over in a hall. Everytime you play something the reflection will sound different due to variance in the resonance of the room like air moving for example.

    The temporary solution for me is going back to traditional type of verbs and mixing different types of convolutions to allow more complex types of reflection. Ideally somebody should create a convolution engine that allowed multiple convolutions at the same time and constant replacement of new impulse files to create more variance in the resonance.

    Regarding DX/VST plugins in games. Its true that we have seen a few being used in games like Halo 3, however I doubt we will see a lot more during this generation of consoles. The challenge is a matter of resources and dedicated audio resources is generally down the chain of events. Don't get me wrong. I would LOVE to see more VST based technologies in games, but I doubt we will see it in a major way before 2010/2011 where the next generation of consoles are introduced. Another serious issue also relates to RAM and streaming capabilities. We used to have 2MB of dedicated sound ram on the PS2 – and now we have something like might 30MB. How far does that take you in terms of samples? The good streaming engines in games allows 4-5 stereo multi-streams at maximum, so certainly limitations there too.

    Regarding procedural music. We are seeing many interesting things happening in the landscape of algorithmic/procedural audio and music. I am personally not so interested in it, since there is a significant loss of quality with this type of music. The general argument is that procedural music will adapt to player preference, which is true in theory. The only issue is that there is a significant loss in fidelity and compositional quality when you take that route. So what would you rather have? An adaptive midi sound score – or a great beautiful soundtrack that got more limitations in terms of its adaptive capabilities.

  • Greg

    What's up with that TimeFreezer? Is it just granular or something more complicated?

  • Why "just" granular ? From the description, it could also be a FFT analysis followed by a resynthesis with either inverse FFT or oscillator bank. That would also explain why there is a free denoiser in it (it is easy in that case to not resynthetize the bands with lower amplitudes).

  • Troels, I hear what you're saying about technological limitations or audio fidelity, but I'm curious about *compositional* quality. Adding interactions without a musical reason behind them would clearly lead to an unhappy end. But is a more interactive score inherently less musical than one that is pre-composed? That's hard for me to believe. For one thing, speaking as a composer, it seems like we just have less experience in creating music that's generative or procedural or whatever you want to call it. I think any composer should be free choose whatever gives the most satisfying results. But isn't that a choice? I just don't see the medium itself limiting the quality of the composition — maybe there are technological constraints to hurdle, maybe you make tradeoffs in audio fidelity, but not compositional tradeoffs. Assuming you make the choice that's right for you, assuming you build your compositional ideas into whatever algorithm you write, and assuming the technology is cooperating, I see no reason why a more interactive score can't be just as good as the conventionally-composed score.

    Admittedly, I'm saying this without staring a deadline for a game in the face. I think it probably (ironically) takes more time to write a score that's generative. But, perhaps separate from the grueling challenges of game dev pipelines, this is something composers will begin to tackle on their own.

  • Thanks for the feedback Peter. I fully agree with your points from a theoretical point of view – however in the practical world with its current sets of limitations – we are pretty far from seeing any strong compositions made with algorithmic technologies.

    Regarding compositional quality it is a subjective thing, however there are some known quantities that we can define in this discussion.

    A truly "adaptive/interactive" score can only be done using traditional short "midi" samples, which naturally sets a limitation to the possibilities and quality of any given score. The reason is that any real-time generation of music will need actual samples to trigger and knowing the sparse limitations of RAM budgets on consoles like 360, PS3 and Wii – there is no way we can introduce quality samples to that pipeline yet. So if you want to make a truly adaptive soundtrack – you are gonna need to work it out with a small sub-set of low quality samples.

    I personally don't enjoy music created at that fidelity level regardless of the actual quality of the composition. The main reason being that the quality of the samples is too low and sets a variety of limitations to what the composer can do. I could never write orchestrations using the traditional set of 128 midi sounds. I normally have about 100 channels in my template for strings alone, so imagine the difference between 3-4 string sounds in your midi-box and a 1100 track template for orchestral composition.

    So what is the alternative?

    The alternative is pre-generated music generated in a studio environment allowing the composer to obtain an acceptable level in both sound and composition. Naturally the pre-generated music has its limitations, since its not real-time generated and therefor not adapt to the same degree as the "midi-based" scoring mentioned above.

    If we look at the current game market – I have yet to see one truly real-time generated score thats got both a great sound and high level compositional quality. So I think we are far, far away from seeing any algorithmic type of compositions that provide deeper value to the listener.

    Please feel free to send me any references of good algorithmic music, if you have any. Good being great sound and soulful composition.

  • Thanks for the dose of reality — and I'll agree, too, we keep hearing this stuff is coming, but you have to be realistic about the challenges ahead. I don't disagree with anything you say here. There's a reason scores aren't moving to fully interactive music, and there's some great "pre-generated" music for the games in the meantime that is effective and has emotional impact.

    But then I don't think, fundamentally, the medium of interactive music is at fault — for all the reasons you cite here. Even outside of games, there are necessary bits missing. To be honest, as a younger composer I didn't have any interest in algorithmic music. I wanted to say, what's the point of that? Why don't I just write down this score one way and give it to my musicians? Now that I see new venues for music, not just games but even going onstage with a laptop, I feel differently. I just feel like there are musical ideas that could be expressed by music that is generated on the spot in a way that something that's pre-composed might not be — maybe not for everything, but as an added tool for composers.

    I have to think, too — imagine it's the year 1750, and you asked if anybody had heard a good string quartet. Haydn hasn't written the classics of the genre. Mozart hasn't been born yet. No one really knows what a great string quartet is even supposed to sound like. The best players are all yet to come. There was some amazing music you could have heard then, so I don't think string quartets are necessarily *better*. They're just different, and it took time to work out what to do with them. It's easy to see that in hindsight … less so in foresight. (and sure enough, before Haydn you had people dismissing these new-fangled gamba replacements that only the gypsies played…)

    That's just a different instrumentation — not even a particularly radical one at that. So, I guess the bottom line is, maybe these scores aren't written yet. And maybe it could happen tomorrow — even if it means low-quality samples or (as in Spore) real-time synthesis. Maybe it won't be in 2008, for the reasons you illustrate here, but at this point I'm even interested to hear some of the attempts that *aren't* fully successful, because I think they could be a sign of what's to come.

  • Hi Troels, I agree with you in the context of orchestral music. But who said game music had to be orchestral music? For epic adventures, it may be an obvious choice but there are many games out there which do not require an orchestral soundtrack but still could benefit from adaptive music.

    Let's take a futuristic racer a la WipeOut for example (of course this particular one is licensing tracks, but that's not the point). In this case, the current gen (well, actually the so-called next-gen) certainly allows for realtime generation of a few electronic instruments and playback of drum samples. It would be possible to make the soundtrack evolve depending on the stage of race, if you drove very well for a given time, if you were close to a competitor, or just got hit and were starting again, if you were almost destroyed etc…

    In other games, it's also possible to compose an adaptive score with a very few selected "acoustic" instruments, especially with a clever loading scheme. There is enough memory for that.

    To be honest, I think people are the weakest link here. What it requires is the right producer, audio programmer and musican on the game, who are all pushing in the right direction. I also agree that the good examples are lacking, which is a shame really.

    By the way, great percussion demo on your web site !

  • Thanks for all the good feedback guys. Its great to read some other viewpoints on this matter and I have little doubt I am biased by the limitations of our current generations of technology.

    Allow me to follow-up on a few of your points.

    Regarding interactive music. I fully support the concept and do find it useful in certain applications like the new rhythm games on DS and/or some of the procedural elements in Spore. But as a composer I also evaluate the sheer quality (production/sound/composition) of the compositions and the results from interactive music just ain't there yet.

    Don't get me wrong. I am probably deeper into adaptive music then most people and was fascinated by it for many years both in an academic and professional context – and I do believe it is the one type of contextual application that can truly elevate our art-form, but it must be followed by the quality and thats where its currently failing.

    The reason I am hesitant about it is that its one of the most frequently mentioned highlights in the games industry. I am sure you have all read about the new and amazing adaptive scores, but how many times have you encountered one?

    My point is put attention on the quality of the game music – not its technical implementation. I still believe game music is caught in the days of "bleeps and blobs" and the industry is truly missing a talent mass compared to features.

    I am excited to see people like HGW, Debney, Giacchino and others in the games industry and they set the quality bar we should be at IMO.

    Anyway … I am prolly rambling here …

    To Nicks point I totally agree that its NOT all about orchestral music – and the possibilities for adaptive scoring are probably higher outside the orchestral realm and I do believe we will see VSTi based technologies on the PS4 and MS1080 for more electronica based types of scores. I would imagine that a .REX based format will be introduced and used in a variety of new rhythm games – where people can shape rhythms as a part of the gameplay feature set.

    I am currently experimenting with a variety of .REX based technologies to test pipelines for how this might work out – and I know others are doing the same, so this will happen for sure.

    But it still doesn't solve the question of quality and the spirit of any composition. I totally agree with Nick that it is about getting the right people together – like getting Eno on Spore for example.

    On an entirely other note I was totally surprised by the success of Guitar Hero. I think it is a brilliant game, but the fact it made over a billion $ for Activision was a big surprise to me, which I guess speaks to the interest of involving players with music creation – adaptive or not.

  • flip

    @Troels Folmann: Quick question… What storage method or hard drive configuration do you use? My library is expanding pretty fast!

  • Pingback: Create Digital Motion » Digital Tools Interviews Paris Graphics on Homebrewed Mobile Game VJ Tools()