Amazon for Infrastructure-on-Demand
The buzz is building around Amazon Web Services as an application platform. Don MacAskill has been using AWS’ S3 storage service for SmugMug, and according to Jeremy Zawodny, will be talking about it at this year’s ETech conference. Jeremy and others have been experimenting with S3 as storage backup for desktop and laptop systems. Even Dave Winer is experimenting with S3.
When designing GigaVox Audio Lite (currently in by-invitation-only alpha test) I decided to use not just S3 (storage) but essentially all of AWS’ services for our infrastructure. This graphic should give you some idea of what we’re doing:
In addition to S3 (storage) I’m using the Elastic Compute Cloud (EC2) which allows us to instantiate servers on-demand. If we need a dozen extra servers for two hours of transcoding, we just fire them up for as long as we need them (perhaps a few hours) then shut them down. And that can be automated based on the load on the current servers. Dynamic computing power on demand. The dream has been realized.
To communicate among our servers and with the external systems we also make heavy use of the Simple Queue Service (SQS). There are actually many more queues than shown in the diagram which is somewhat out of date.
In asking around, it appears we have one of the most complex and sophisticated systems built to date using all three AWS services. It is infinitely scalable, extremely reliable and costs very little until we need it. And our system administrator doesn’t have to carry a pager. All those servers are someone else’s problem. No hardware or software to buy, maintain, upgrade, etc. It has been a lot of fun to actually make use of some of the asynchronous message-based architecture ideas in my book, Loosely Coupled. Hey…this stuff actually works! Sweet. Congratulations to Amazon for brilliant vision and execution.
If you want to see it in action for the production and publishing of your podcast, send me an email, tell me about your podcast, and I’ll send you back an invitation to the alpha test.
18 Comments »
RSS feed for comments on this post. TrackBack URI
Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
“all three AWS services…”
Amazon offers seven services, not three. You left out Amazon E-commerce Service, Amazon Mechanical Turk, Alexa, and Amazon Historical Pricing service..
Comment by Larry — 2/5/2007 @ 1:35 pm
I’m myself using S3 for quite some time — other than a stray slow response time once or twice, the service has been amazing. And even if the resp time is slow, Amazon folks are very quick to rectify it.
the amazon WS is just brilliant and a life-saver when it comes to startups who want to focus on execution and business rather than the headache of dealing with storage, backups and stuff like that.
slideshare.net also uses amazon s3 to host the flash based ppts.
Comment by Pranav — 2/5/2007 @ 3:40 pm
GNC-2007-02-05 #240
Mixed bag of tech talk tonight along with some links to some really cool sites. The amount of listener has been amazing thanks for your continued support. Sponsors: [Save 10% off on any order at GoDaddy.com!] Use Code Todd [Try…
Trackback by Geek News Central Podcast — 2/6/2007 @ 1:53 am
GNC-2007-02-05 #240
Mixed bag of tech talk tonight along with some links to some really cool sites. The amount of listener has been amazing thanks for your continued support. Sponsors: [Save 10% off on any order at GoDaddy.com!] Use Code Todd [Try…
Trackback by Geek News Central — 2/6/2007 @ 1:55 am
Awesome setup, Doug. You’re one of the few out there really taking full advantage of Amazon’s offerings. For the curious, Basejumpr.com is another startup using S3, SQS, and EC2 in conjuction. I can’t wait to see what else gets built on top of Amazon’s infrastructure.
Comment by Robert — 2/6/2007 @ 7:28 am
A while back I had built this: http://www.francisshanahan.com/robocast
with thoughts of perhaps developing it into a service. It looks like there are some parallels there but you have taken it to the next level with Gigavox.
Nice job.
Comment by Francis Shanahan — 2/15/2007 @ 11:55 am
GigaVox Audio Lite: Web Services in the Wild
Technometria is fast becoming one of my favorite podcasts every week. I like to say it’s “like The Gillmor Gang, but actually good”. This week’s episode is one of the best yet. It’s a conversation with Doug Kaye of IT…
Trackback by berbs.us — 2/20/2007 @ 3:40 pm
I use S3 too for streaming videos over HTTP. you can check out the servcie at KONKAN TV. Apart from the ocassional failed SOAP / REST file push requests and ocassional CPAINT errors and/or streaming speeds, it’s been an awesome ride so far…
Comment by Rohan Pinto — 2/23/2007 @ 10:00 am
mduaoffrosmaofgoerosmonvosmogosnolvoslovnms
ncvosnrpos
Comment by Name — 2/23/2007 @ 10:39 am
Do you know where can I find the Amazon architecture? I mean, servers, and Databases.
Comment by Ioanna — 3/1/2007 @ 5:28 pm
Understanding Amazon EC2
Amazon’s new web services have been gathering a lot of momentum recently, in particular the Simple Storage Service (S3). I can wrap my head around S3 pretty easily and there’s no shortage of the canonical example of “remote backups…
Trackback by Cantoni.org — 3/12/2007 @ 9:25 pm
Hi Doug, i was listening to your video on Amazon Web Services. Thanks for sharing the info.
I wanted to follow up on your decision to use the external Database server. Can you provide some detail on why you did not choose to use one of the images as a database server.
regards
Comment by Madhan — 3/14/2007 @ 10:03 pm
The way that you break down the structure makes it much easier to understand. Great job.
Rick Marnon, Howell
http://www.oaklandlivingston.com/howell.html
Comment by Rick Marnon, Howell — 9/28/2007 @ 8:44 am
what standards or lack of, were there when you were beginning to use Amazon’s web services?
Comment by Darlene — 1/29/2008 @ 6:25 pm
Doug – where’s the graphic of the architecture?
Comment by martin — 2/12/2008 @ 7:43 am
Hi, I have a question about the real world usabitity of the Amazon Web Services. Could I run windows servers in windows vm clusters on the Amazon Elastic Compute Cloud (Amazon EC2) Windows Server running SQL, Office Middle Ware, VOIP… like a real hosted utility grid?
Can’t find any data on people using windows server os’s with it?
Thanks
Comment by Mike — 2/14/2008 @ 1:19 am
Hi there, the graphic doesn’t show up any more
Any way I can have a glance at it? I’m really interested in how it works with Amazon s3!
Comment by Tim — 3/26/2008 @ 3:49 am
amazons architecture is totally different from others there way of building in entirely different and i coulnt follow the block diagram above
Comment by poker game practice — 3/6/2010 @ 1:39 am