Following mobile music making means keeping up with technical details that are complex and changing. And because the Internet is open, when you post a story on iOS DJ apps, odds are it may be read by some of the Apple audio engineers, third-party developers, and a casual DJ with little understanding of what’s behind the scenes. But I’ll say this: the behind the scenes stuff matters, and it’s a great gift that we’re all able to compare notes.
Last week, I noted that, via the developers of a scratch app called Flare, there was evidence that Apple had broadened developers’ ability to work with music from the iTunes library in iOS 4. That led to an explosion of comments, including many from developers. The issue is still one we’re exploring, but I’d like to clear up some questions.
Also, I want to invite developers interested in comparing notes and sharing code to join a new group on Noisepages, the community site we’re finishing developing this summer:
Next-gen Mobile Music + Visual Dev Hack Group
The group has forum-style discussion features, the ability to connect with other users, and a group blog which could be useful for posting code snippets and the like. That means if you want to share something you’ve discovered with other developers or ask a question, you can. We’re not trying to replace other resources, but to make it easier to find them – so you can point to a stackoverflow thread, or your Git repository, or your commercial developer blog. Part of the reason to invite you is that this July and August is when we’re establishing other features for launch, so if there’s something you wish the site did, just suggest it. You only need to sign up for a Noisepages account, which is free and now open to anyone. I kept this platform agnostic partly because I hope we can all learn more about different operating systems, and because there’s plenty of overlap.
But first, let’s clear up some of those questions users and developers might have about Friday’s story.
Why all this fuss over having access to the iTunes library? Aren’t there DJ apps already?
For a lot of DJ end users, the ability to manage music with iTunes but DJ with a mobile app is important. In the past, developers have had to keep a separate set of files for use in the DJ app, synchronized separately, which for many end users isn’t good enough.
Wasn’t this added before? This isn’t news.
Actually, this is news. iOS 3 added very primitive access to the library — that is, you could see what was in the library on the device and play and stop, but nothing else. DJ apps will naturally want to be able to do things like change pitch or scratch – or even play the volume – which weren’t available in iOS 3. That means access to the buffer data for the audio in order to process it directly. (Those are the numbers filling your head Prince was recently complaining about.)
So, what’s the story? You can access audio buffers in iOS 4?
Here’s where things got a bit confusing. Because there’s a new set of APIs, music developers on iOS are still working out what’s possible and how to implement it. As a developer, you really don’t know a platform feature until you’ve successfully used it in your own code. Even with a documented feature, you may discover subtle issues – and new features often aren’t fully documented, if at all.
Async Games, makers of Flare, did manage to pull audio files from a device library and scratch them. So far, so good. But to get there, they required an intermediate step; what developers really want is direct, full-blown access to the Library. (And yes, I got a few messages in my inbox from developers who thought I was overstating what Apple had provided.)
To see the discussion of what’s there, what isn’t there, and how to use it, complete with code samples, see comments on Friday’s post.
Async Games and some of our regular iOS developers readers have been helpful to me in untangling this. But while Flare was the first to officially scratch an iTunes track, here’s some good news – it does sound out like the makers of DJ Mixer have figured out how to build a whole DJ app. There’s still that intermediate step, however. DJ Mixer developer Aviv Eyal tells me:
It works by copying the audio file into the App’s sandbox. MP3, M4A, AIF and WAV are supported.
Once there, the App has full access to the audio data and it calculates the BPM, beats, bars, extracts cover image and other meta-data that helps in mixing.
After this one-time import, the music is mixable and can be played without additional importing.
And as I said on Friday, even for non-DJ apps, this could be significant. You could make a sample-based performance app, for instance, that allowed your user to manage samples via iTunes on the desktop instead of a separate conduit. So, for instance, an artist might take all of her albums, load it as an iTunes playlist, and then remix her back catalog in a mobile app.
It could also mean creative visualizer applications for the iTunes library. Stay tuned; hopefully we can document how to actually do this. (Join the group above.)
(Yes, some readers say they prefer a separate file store for their DJ tracks; if that’s you, you don’t have to worry about any of this! As a developer, though, you have to cater to the users who want to use iTunes.)
Here’s a new video of the upcoming 2.1 version of DJ Mixer with the iTunes Library functionality; it’s currently awaiting approval from Apple as I write this.
What about Android? You said the situation was different on Android. This is all about different philosophies of openness, Apple’s lockdown, blah, blah…
Well, yes and no. I brought up Android on Friday to provide some context – and not because I wished to make a black-and-white comparison about which was “better,” but on the contrary, as a way of highlighting the unique challenges all mobile developers face.
Here’s the deal. First, Android does allow developers freer access to files loaded on the device. That is a partly philosophical difference; since before the iPhone, Apple has maintained a fair degree of control over media files synced to their iPod. For instance, when you mount an Android device on your desktop, you can freely drag and drop files into the music database. That’s actually been a cause for complaint among some reviewers, but I will say, even a lot of less-sophisticated / less-techie Android owners I talk to like it.
In this case, though, these issues are less relevant to the task at hand. The issue with a DJ app is that you want to get buffer access not just to any file, but very possibly a compressed file. Part of the reason neither Android nor iOS has made this very obvious may even be that both platforms are using hardware acceleration for some decoding tasks, and they’d need to get the data back from the decoder.
Anyway, this isn’t a platform discussion, and it isn’t philosophy – it’s good, old-fashioned, nuts-and-bolts, hands-dirty development.
For the record, when I said it was possible to directly access files on Android, I wasn’t incorrect, but I was simplifying matters in regards to decoded files. There isn’t a software-side decoder that comes out-of-box with the platform. It is possible to decode MP3s, however; you just need to add your own library and dig into the NDK. The MAD MP3 library would be a way to do that, and it’s free. I’m going to actually look into this, because I need for a project. But yes, assuming we do work out how to use them, it looks like the iOS platform will be a path of less resistance thanks to these recent changes. (In-box decoding could be an issue on new platforms, too, like Chrome and MeeGo.)
The bottom line? If you’re building a DJ app right now on mobile platforms, be prepared to roll up your sleeves, because some things you take for granted on desktops simply aren’t there. On the other hand, there’s such interest in the problem, you won’t be alone – and I suspect we’ll be getting code examples going in relatively short order.
So, definitely do join up the Noisepages group if you’d like to learn from fellow coders; otherwise, as always, stay tuned here. And if you’re an end user, of course, we’ll explain what’s available to you as these apps ship. Having a window into development simply means you’ll know what’s coming a little earlier.