digg

Facebook Connect Join Digg About

Amazon S3 - World's most reliable web service is DOWN

blog.linkdiagnosis.com — The most reliable storage on the net been down for over 30 minutes and counting. This might not seem much, but think how many sites now rely 100% on Amazon S3 to keep their sites running - such as SmugMug and PageFlakes. These sites are losing revenue right now...

Who dugg this? Made popular Feb 16, 2008

Digg DudeWhat is Digg?

Digg is a place for people to discover and share content from anywhere on the web. Digg surfaces the best stuff as submitted and voted on by the community. Take the Tour to learn more.

151 Comments

show profanity settings
expand all
  • markwilcox, on 02/16/2008, -6/+212Wtf is Pageflakes and SmugMug.
  • LANjackal, on 02/16/2008, -5/+165EVERY site goes down every once in a while, people. Chill out.
  • inactive, on 02/16/2008, -0/+140Even with a 99.99% uptime guarentee, that leaves roughly 1 hour of downtime per year...
  • uptown, on 02/16/2008, -2/+140....and somehow the world continues to turn. Life goes on without Pageflakes, and SmugMug.
  • CamperBob, on 02/16/2008, -1/+75They're like ZorgBlatt and FrotzOzmoo but with 13% more XML.
  • till, on 02/16/2008, -10/+81You guys shouldn't forget, this service has been around a pretty long time (in Internet time) and personally I have never seen anything like that. So the chances of it happening were larger and it just had to happen some time. This incident puts Amazon back in the pool with the rest of us - the "anyone can be down"-pool. At least for me that's good to know! :-)

    This means two things - one, there is a reason why companies like Akamai charge you an arm and a leg for their CDN, and two, people should think about a just in case backup strategy.
  • Konrad9, on 02/16/2008, -3/+70OH MY GOD A WEBSITE IS DOWN.
    OH MY GOD PEOPLE ARE LOSING REVENUE.
    Just like every other website and every other company.
    Ever.
  • amirman, on 02/16/2008, -0/+25they're like Pageflakes and SmugMug but with less XML.
  • geckosite, on 02/16/2008, -0/+23this was just a plot to get us on smugmug.com
  • tomarocco, on 02/16/2008, -11/+34Not making revenue is NOT the same as losing revenue. American Capitalism has spoiled everybody.
  • AgentMull, on 02/16/2008, -5/+26So the more I play the lottery the better the chances of winning I have? I think I see what you're saying in that it has to happen sometime, but the chances don't increase as time goes on.
  • TheNerd25, on 02/16/2008, -11/+31Maybe people shouldn't rely so much on this one service. Spread your stuff around a bit maybe?
  • trghpy, on 02/16/2008, -1/+19Something went down?
  • nonnald, on 02/16/2008, -0/+17It's like HTML with 13,000% more tags
  • Fxer, on 02/16/2008, -3/+18For pure chance, the possibility of it happening are the same for each interval of time, but taking all intervals of time together, the possibility of it happening increases at each interval.
    I'm not sure if I make myself clear, and I'm too lazy to make analogies now.
  • MrTito, on 02/16/2008, -1/+15Hey! Don't you dare bring reason to Digg!
  • MrTito, on 02/16/2008, -0/+13http://www.techspot.com/news/17557-saturdays-googl ...
  • Calculon, on 02/16/2008, -0/+1299.99% uptime = 52.56 minutes of downtime a year
  • arunforce, on 02/16/2008, -0/+12Wtf is ZorgBlatt and FrotzOzmoo.
  • ross., on 02/16/2008, -0/+11They have the Internets overseas now.
  • Hockey13, on 02/16/2008, -1/+12There is a cost involved...opportunity cost. The cost that is incurred by doing nothing. It's got nothing to do with "American Capitalism" (whatever that means), it's just a basic notion of economics and logic. If you do not do something in a specific period of time, you lose the opportunity to use that time productively. If they actually are losing the opportunity for revenue (like if the Google ads stopped working), I can understand being pissed about that if your main goal is to make money. The "chance" of making money is precisely the point. You were supposed to make that money, but you didn't because your tech guys didn't manage to keep the site up for whatever reason. Monkeyvoodoo, the execs were screaming about it because they're good execs who don't like to tolerate things like downtime. It's not really so absurd. Their demand for internet uptime makes web services compete to provide the service. I have to admit, it's pretty good right now when we're talking about just 15 minutes of downtime, so capitalism seems to be working very well.

    There's also the issue of appearing to be online forever. Uptime is the most important draw for major web services...they lose potential clients by appearing to be unreliable.
  • eggnet, on 02/16/2008, -0/+10No, betting on which 5 minute interval it will happen is the gambler's fallacy. Betting that the chance of failure is higher in a 3 year period vs 2 year period is common sense.
  • lehdan, on 02/16/2008, -0/+10It's like XML with less tags.
  • bioskope, on 02/16/2008, -0/+9From their site

    "We’ve resolved this issue, and performance is returning to normal levels for all Amazon Web Services that were impacted. We apologize for the inconvenience. Please stay tuned to this thread for more information about this issue.

    Thanks,
    Kathrin "

    http://developer.amazonwebservices.com/connect/thr ...
  • SantaClauz, on 02/16/2008, -0/+9Wtf is HTML?
  • Jandels, on 02/16/2008, -0/+9http://developer.amazonwebservices.com/connect/thr ...
    Post by Kathrin@AWS (Page 6)

    Here’s some additional detail about the problem we experienced earlier today.

    Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.

    Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.

    As we said earlier today, though we're proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.
  • execute85, on 02/16/2008, -0/+9That means that S3 was down. No authentication means no one can access the system. Think about it like "My car's not broken, it just won't start because the ignition is broken." You still can't drive anywhere.
  • r36283, on 02/16/2008, -1/+10Ive never even heard of either of these sides, but they much be pretty kickass with names like those.
  • JAG731, on 02/16/2008, -1/+9Wait. First it was overseas, now this. How many abandoned anchors are there?
  • mannymix03, on 02/16/2008, -0/+8wow the people in the amazon thread piss me off.
    "This just shows that I cannot trust amazon with my mission critical data."
    WTF, if its mission critical it should be hosted elsewhere and should be able to be switched to a backup server at any moment, otherwise its probably not that important anyway.
  • Shogi, on 02/16/2008, -1/+8Oh the humanity!!
  • inactive, on 02/16/2008, -0/+7pageflakes and smugmug seem to be working fine.
  • amirman, on 02/16/2008, -0/+7have you heard of the sides, potatoes au gratin or mac and cheese?
  • ryanonfire, on 02/16/2008, -1/+8Wtf is XML?
  • schotty, on 02/16/2008, -0/+6Average, mind you :D So if they went 5 years straight, they could still afford 4:25 or so of downtime and still have 99.99% uptime.
    Just rubbing salt on the wounds I guess ... Sorry :(
  • ross., on 02/16/2008, -0/+6Umm, yeah we are.
  • dickeywayne, on 02/16/2008, -0/+5No. 1 in 4 children born in this world is Chinese. Your fourth child will be Chinese. And probably a girl! ;-)
  • X18999, on 02/16/2008, -0/+5well considering that they are down at the moment...
  • tybris, on 02/16/2008, -0/+5The idea is that they do that. They have vast redundant storage.
  • pikpikcarrotmon, on 02/16/2008, -0/+4WTF are tags?
  • 37prime, on 02/16/2008, -0/+4"WAS"
  • inactive, on 02/16/2008, -0/+4revenue is not a car.
  • jmgarrison, on 02/16/2008, -0/+4How is this just becoming popular?
  • morganimal, on 02/16/2008, -4/+8These container ships MUST be stopped! how long before one of them drops it's anchor on Digg?

    Think of the children.
  • violentvinyl, on 02/16/2008, -0/+4Well, technically, as the computer hardware gets older, cooling systems need maintenance and staff attention atrophies, the chances of seeing some down time do get better (and YES, I did look at this thread because I was thinking the same thing about "So the chances of it happening were larger and it just had to happen some time").
  • shagg187, on 02/16/2008, -5/+9Guess amazon accidentally burned their server in the fire created for HD-DVDs...
  • amirman, on 02/16/2008, -1/+5HEYYYYY those sites aren't REAL! ..... i know because i checked :(
  • TheFiestyFaun, on 02/16/2008, -2/+6maybe...
  • scabbers, on 02/16/2008, -0/+3Hahahahaha. OMG MY CONSTANT SUPPLY OF MONEY FOR OLD ROPE HAS BEEN TEMPORARILY HITCHED!!! Some of those people are going to hit the ground hard when the second internet bubble bursts.
  • Fetching more discussions...
    Show 51 - 100 of 155 discussions

  • Add a Comment

    Login or register to comment!
    Or, sign-in super fast with your Facebook account ...


© Digg Inc. 2009 — Content created and posted by Digg users is dedicated to the public domain | Terms of Use Updated | Privacy Policy
DIGG, DIGG IT, DUGG, DIGG THIS, Digg graphics, logos, designs, page headers, button icons, scripts, and other service names are the trademarks of Digg Inc.