Those of us in music more or less noticed today, “hey, my SoundCloud audio isn’t loading” — while the rest of the Internet went crazy because lots of things were broken. But the reason was Amazon S3, the “cloud” storage service provided by the retail giant.
Early indications appear, though, that SoundCloud’s audio playback and buffering difficulties are the result of degraded performance of Amazon S3 storage.
See, on the outage in general (still ongoing as I write this, amidst some really confusing messages from Amazon):
Amazon’s AWS S3 cloud storage evaporates: Top websites, Docker stung [The Register]
Now, I use “cloud” storage heavily in my personal work. It’s a backup service, a failsafe data source when on the road. It’s a way of sharing music – as a journalist, as an artist. I use it to save and send critical jobs. Most importantly, I use servers to run CDM, which is a huge part of my livelihood. And it mostly does its job. But then, it’s also important to understand that what “cloud” means. These are physical servers at a physical location with physical connectivity, and they’re operated by humans. That’s a long string of vulnerabilities there, from human error to external attack to forces of nature, even apart from technical problems.
Nerdy web comic xkcd has been there – of course.
If you understand that, you understand that failure is indeed an option. Furthermore, I think you’ll agree there’s vulnerability in centralized, monolithic solutions.
And today, many pundits are reaching the same conclusion.
So, I’m not looking to criticize SoundCloud in particular here. Indeed, a service like that is likely to be more able to recover from trouble than you would on your own, and I don’t know enough about the specific interaction with S3 and SoundCloud today to comment on how they’ve set up their connectivity.
But that said, there is a larger concern about over-centralization and monocultures, even just from the standpoint of ensuring you’ll have access to your own music.
And I’m not the only one adding a red flag here.
In fact, not coincidentally, I quickly am seeing pundits making the same comparison that popped into my head – to Dyn, a DNS provider that took down a ton of the Internet during an attack last year. (That included CDM.)
Here’s Wired:
The “winner takes all” dynamic of the tech industry concentrates more and more power into fewer and fewer companies. That consolidation has implications for competition but also affects the resilience of the internet itself. So many people rely on Gmail that when the service goes down, it’s as if email itself has gone offline, even though countless other email providers exist. Facebook is practically synonymous with the internet for many people all over the world.
The Amazon S3 Outage Is What Happens When One Site Hosts Too Much of the Internet [Wired.com]
(It’s also worth reading Wired’s interview disproving the Internet was built to withstand nuclear strikes. I think I’ve repeated that urban legend. It’s not true.)
That’s a pretty profound statement, though. It suggests that there’s a fundamental problem, but that the dynamics of the industry itself are making the problem worse. (Dyn was a pretty clear-cut case of that, as I can attest. Like a lot of Dyn customers, I had originally used a DNS provider as a way of adding resilience. I didn’t even pick Dyn, though: my provider was bought by Dyn and I was given no choice but to switch to their service – and their pricing, I might add.)
For their part, SoundCloud has been a featured Amazon case study – even though SoundCloud didn’t mention Amazon Web Services (AWS) by name today. (No need; the discussion dominated Twitter if you followed anyone doing Web work.)
AWS Case Study: SoundCloud [Amazon AWS Solutions]
Here’s Alexander Grosse, VP of Engineering at SoundCloud, on that solution:
SoundCloud uses a combination of Amazon Simple Storage Service (Amazon S3) and Amazon Glacier as its storage solution. The audio files are placed in Amazon S3 and distributed from there via the SoundCloud website. All files are also copied to Amazon Glacier, to ensure that the data is available at all times, even in the event of a disaster. The company currently stores 2.5 PB of data on Amazon Glacier.
Through the combined use of Amazon S3 and Amazon Glacier, SoundCloud is able to securely store data volumes without requiring additional operational overhead. “We don’t need to worry about storage. AWS lets us sleep well at night,” Grosse says.
Now, again, I don’t want to criticize SoundCloud here, or maybe even Amazon. The simple fact of the matter is, Amazon may provide more performance and reliability at the costs SoundCloud requires than would another solution.
But ask yourself this: what about your uptime? Were you impacted by this outage today, and could you recover? (I was – one article was held up, and one work on a release, because each involved music only on SoundCloud that I then couldn’t access.)
This should also make clear that offline storage has got to be a permanent fixture of live performance and DJing. That may sound an obvious statement, but software makers are already trying to imagine a world where music in a DJ set was streamed from the Internet instead of locally.
Ironically, I was working today about an article talking about the dangers of centralization on Facebook. This, of course, interrupted that.
There are cultural and technical issues at work any time our music online is in the hands of just one vendor. You can’t point exclusively at Amazon, because its major rivals in this very business are Google and Microsoft – more big centralized operations by enormous transnational American companies. You can’t just look at SoundCloud, because odds are the SoundCloud alternative you like may also use the same storage provider. You can’t even look at centralization, because centralization for musicians might be what allows their music to be found easily and for people to encounter maximum familiarity and minimum resistance in playing your tracks.
But you can begin to say, we have a potential problem here – and that at the very least, we can’t treat the cloud as something magical that will always be there for us.
So, what do we do?
Well, for one thing, we certainly want the Internet to be less … vulnerable than it is in that IT Crowd sketch. To some system administrators and developers, today, I suspect it felt almost exactly like that.