What is Digg?50 Comments
- rgeide, on 10/12/2007, -0/+55Another tip to reduce your website's bandwidth would be to not submit it to digg
- markdr123, on 10/12/2007, -2/+33This blog just keeps getting better and better.
- Xanium4332, on 10/12/2007, -0/+15Digg = the friendly DDoS
- Yez70, on 10/12/2007, -0/+14You can get a wholesale 1GB up/down pipe for $15k-$30k a month, compared to a 1.5MB up/down T1 pipe for the average joe business/site at $500-$800 a month.
As, an example, the larger sites then tell everyone that pays $20/mo they get access to 10MB each - maybe 100,000 separate people - making them $2mil a month. Only 10% of them ever use 10MB at a time tho - that's still only 100MB of their 1GB pipe. If they had twice as many customers their profit doubles - yet they probably still aren't utilizing all their bandwith. This is how the discount dedicated server companies work.
It's an economy of scale, just like your cable company sells 7MB connections to 500 people in a neighborhood node, yet it's only a 100MB connection from the node to their NOC which connects to the actual net. If all 500 people tried to use all 7MB of their 'paid for' bandwith at full speed at the same time - the connections would slow to dial-up speeds. - rowlodge, on 10/12/2007, -0/+13must work, this site is still up and it's on digg.
- jackmaninov, on 10/12/2007, -2/+149 GB in a day? Awwww, muffin. The site I admin is pushing out 120GB a day across three servers and growing (three months ago it was about 50GB a day!)
mod_gzip and php zlib-output are your friend. :) Didn't see php zlib-output on this guy's list. - jrusso9, on 10/12/2007, -1/+13I am in awe of Jeff Atwood. I am posting a shout out to JEFF, who provides us with useful, accurate, documented, crystal clear information. Thank you so much for sharing, and not charging for this. I learned SO much in just a few minutes. Wow, the slippery teeter-toter of Digg vs. Website crashes, and all the decisions... are very interesting to watch.
- whistlerpro, on 10/12/2007, -2/+14Oh the irony, submitting a "reduce your bandwidth usage" article to digg.
- jongalloway, on 10/12/2007, -0/+12Jeff (Mr. CodingHorror) isn't submitting his links to digg. Other people are.
- listrophy, on 10/12/2007, -5/+16A better tip would be to not hit the digg front page twice in 24 hours.
With the exact same article. - spyrochaete, on 10/12/2007, -3/+13This is the first time I've read it and it won't be the last. Very well written for its intended audience.
- markh1967, on 10/12/2007, -0/+10From the site: "Still, I'm hoping these changes will let my site weather the next Digg storm with a little more dignity than it did the last one-- and avoid taking out the network in the process."
Well, it's hit the front page and it's still loading quickly so 'Mission Accomplished' I think. - fogster, on 10/12/2007, -0/+10That's actually the whole point, though. Last time, he was basically taken offline when one of his stories got Dugg. His goal is how to handle lots of hits without burning through bandwidth unnecessarily.
- fiveoldroad, on 10/12/2007, -3/+11How come flickr, feedburner etc are able to offer bandwidth so much cheaper than the guy can buy it himself? What's the big economy of scale that comes into play?
- crosenblum, on 10/12/2007, -3/+11I read this blog every day, just high quality information.
- inactive, on 10/12/2007, -1/+8On the same note, you can get fiber in most large downtown areas at about $700/month starting, which is starting at 10/10 and can ramp up to 1000. We have this at work, unlimited transfer at 10, and it is wonderful. T1 is old tech. if your business is centered around the net, you should seriously consider abandoning it.
- skyhighrockets, on 10/12/2007, -0/+6Doesn't swoosh require a browser plugin or some such before you can actually use it?
Correct me if I'm wrong.. - psyon, on 10/12/2007, -0/+5I admin a site that is pushing out 5TB a day across 26 servers.
*EDIT* Oh, and to save on some CPU and bandwidth, we cache the output of our PHP scripts in gzipped files, and just send them out directly. Takes alot of load off the servers when they arent compressing everything on the fly. - NetJoe, on 10/12/2007, -0/+4The phone company will charge you at least $300/month for the T1 (depending on millage charges it can be MUCH more) , then you still have to do something with it. If you host someplace you eliminate most of the transport charges.
- beejay54, on 10/12/2007, -0/+3@ mjm01010101: Where exactly is this 10/10 fiber you speak of? What city exactly? I've never heard of data/price rates like that before and unlimited transfer too?
- badnewsblair, on 10/12/2007, -0/+3Great blog. One of the quality ones still out there not trying to spam my digg with worthless redirects and copy/pasted information from other sources. Keep up the great work.
- samsite, on 10/12/2007, -2/+5very useful info, Ill be sure to put some of this in place
- Step1Mark, on 10/12/2007, -0/+3Dear God,
Please give me Japans Internet.
Amen,
Mark - badnewsblair, on 10/12/2007, -0/+3RedSwoosh is an interesting idea, but requiring end users to download a client will be the death of it. Now, if they could get a deal with an Open Source browser to bundle it, then you might have something.
- nOOBert, on 10/12/2007, -0/+2Having a 100mb line and a 1TB of bandwidth also helps :) I spread my stuff over a few differ sites just to keep it up if needed.
A constant 10mb line if it were running at its max for a month is some where around 3TB total bandwidth used... Some of choose to pay for a constant pipe of a set max speed and other choose to by bulk bandwidth with out a max speed. Both have their pros and cons.
Good tips. Good site. - crosenblum, on 10/12/2007, -0/+2I am here, to learn, and improve the performance and scalability of my companies website, learning and sharing is what makes the whole industry improve. Jeff Atwood's blog is a very worthy blog, quality content and information, should be recognized and rewareded, if not with respect, but with financial recognition over time.
Keep up the good work jeff... - Linguino, on 10/12/2007, -0/+2"A site might get shut down in a digging or a farking or a slashdotting because Apache can't handle enough simultaneous requests (a compile-time setting / config file setting in Apache)."
Using the right MPM and MPM settings are critical to this point. It makes the difference between a server that will utilize all available resources and keep going, and one that will over utilize and become unresponsive...
http://www.devside.net/articles/apache-performance-tuning - looksliketrent, on 10/12/2007, -0/+2I didn't see it mentioned in the article, but you can precompress with GZip content such as CSS files, as long as your web server recognizes that it's GZip encoded and the appropriate headers are sent. This way, you can save cycles and memory normally wasted on re-compressing the same content.
- MrScience, on 10/12/2007, -0/+2I'm constantly amazed whenever I surf to Digg on my smartphone. 1MB+ for a front page is just *painful*.
- tommyboy180, on 10/12/2007, -2/+4Here in Okinawa, Japan I can get a 1.5 T1 for $60 a MONTH!!!!!!! Japan Rules!
- Blandyman, on 10/12/2007, -1/+3boboth:
Actually, 1&1 (don't know their bandwidth limits) probably has that $20 for shared hosting on an enterprise package of some sort. Dedicated hosting (your own server) is more expensive, and even dedicated hosting is on a shared line. So, to get dedicated hosting and a dedicated line, you either have to find a REALLY good host (not 1&1) or do it yourself. - geekwithsoul, on 10/12/2007, -0/+2All good advice, but in my experience using valid XHTML and CSS, as well as semantic markup, can do more to reduce bandwidth than all of these combined. For example on the page in question, there are empty paragraph tags, excessive use of in-line styles, unnecessary white-space in the page and they use an odd mix of HTML 4 and XHTML 1.
It may not sound like much, but when I redesigned a horrible table-based site layout last year, and instituted valid code and proper use of CSS, I cut the average page size by 50%. Couple that with compression, and the site was much speedier. - MattH, on 10/12/2007, -1/+2Swoosh the content on your website
http://www.redswoosh.net/ - inactive, on 10/12/2007, -1/+2use www.diggriver.com then
- strings28, on 10/12/2007, -0/+1There is also the added value that having your images (and frankly any other file like CSS or js) hosted on another server, most clients will only request two files per server so having them on another server means your client will more than likely be able to load the site up faster as well.
Good article over all. - stephbu, on 10/12/2007, -0/+1They don't buy bandwidth in blocks of bytes egressed - they buy throughput - generally billed monthly as $/mbit/s based on 95%ile of peak egress. e.g. a month of $80/mbit/s @ 15Gbit/s 95th %ile peak egress == $1.228m.
The economy is that they get provisioned with multiple redundant peered networks, e.g. multiple high-capacity optical carriers, and a network border well beyond their physical data centers. - BladeMelbourne, on 10/12/2007, -1/+2Well written, however turning on HTTP compression for dynamic (server generated) pages can add a significant CPU overhead and should be evaluated first.
Many people do not know that when compression is enabled in Win 2003's IIS, the following static types are not compressed:
js, gif, jpg, png, css, and the following dynamic types are not compressed: aspx, asmx
You can do it by editing the metabase XML file, or using the IIS admin kit. - Linguino, on 10/12/2007, -0/+1And mod_deflate if you are using the 2.0 branch of Apache.
Personally, I would stay away from PHP's zlib.output_compression, and just use the Apache zlib module with PHP output_buffering turned to On. - DigDugDigger, on 10/12/2007, -0/+1Submissions like this is why I come to Digg :)
- BigSlacker, on 10/12/2007, -0/+1It would be more interesting to see those methods in a cost analysis rather than just with the singular goal of reducing bandwidth. Simply buying more bandwidth may be cheaper. We're not talking about a huge site at only 9GB a day.
- inactive, on 10/12/2007, -1/+2On a blog post with one image, why not? I've never uploaded to flickr before, but it can't take more than a few minutes.
- crossers, on 07/16/2008, -0/+0Very useful information. this blog just keeps getting better.
http://www.shpe-sac.org
http://www.ocflex.com/
http://www.trgovinca.org
http://www.chasr.org/ - kd1s, on 10/12/2007, -2/+2Before we moved to the new facility, we were using a 1.1mbps DSL line. It couldn't keep up with anything we wanted to do and public could hit our site all that well at key times, like election time.
Now we've got a 10mbps pipe that we provision 5/5 - 5 to our elections app, the other to our net access. It slides too, if the web site is getting hammered it'll shift usage to 7-8mbps. We survived the 2006 election cycle without a hickup, well other than tuning or Apache server a bit.
There's also load balancing. You can distribute your site among several servers because your real bottleneck might not necessarily be bandwidth, but the speed at which your web server can service requests. - LiberalsSuckAss, on 10/12/2007, -1/+1This whole article is useless. For anyone who wants reliability in a web site and plans on doing anything more than just "playing internet", you need to be hosted in a dependable network operations center - plain and simple.
We push 5 to 10 TERABYTES (yes, that's 10,000 gigs) per month through our servers hosted at a reliable network operations center.
A site might get shut down in a digging or a farking or a slashdotting because Apache can't handle enough simultaneous requests (a compile-time setting / config file setting in Apache). But there's absolutely no excuse for bandwidth being a bottleneck anymore with well-connected shared hosting environments being so affordable these days. - quick5pnt0cobra, on 10/12/2007, -4/+2LOL, Nobody who is serious about websites would even consider linking all of their images from a flickr, photobucket, or imageshack account. The time alone to do such a task isn't worth it.
- inactive, on 10/12/2007, -8/+2This article is very useful in special for those who write for digg homepage.
- Alex.w, on 10/12/2007, -7/+2Thats what I was thinking when I read that..
"When Why Can't Programmers.. Program? went viral last week, outgoing bandwidth usage spiked to nearly 9 gigabytes in a single day" For reals, 9 whole GB is a day? no wai.
Get an account on a real server already, there's no way to know how many page views you missed because the two T1s from 1995 you're still using were maxed out. - ShadowSkill, on 10/12/2007, -11/+2"Reducing your websites bandwidth usage"
Don't post a link of it on digg? - bobothn, on 10/12/2007, -11/+2why not just pay the $20 a month to 1&1 and have all his data stored in with a 20 GBit shared connection? lot cheaper then 2 t1's and he would probably get a lot better speeds.
The Digg Toolbar for Firefox lets you Digg, submit content, and keep track of Digg even when you're not on the Digg site. Download the official 
© Digg Inc. 2009
— Content created and posted by Digg users is