What is Digg?Sponsored by Threadless
55 Comments
- mikelieman, on 10/12/2007, -0/+52Interesting idea, which brings to mind two questions:
What's the cost in additional CPU usage on the server side?
Is it greater or less than the expected savings on bandwidth? - inactive, on 10/12/2007, -0/+20I gzipped the rest of this reply to save bandwidth.
qÕjƒJ×0'„%$¸#p"ê"‘!T! !¨" ""z"ñ%'R&Á$c#Í#’#¡#¦#ž#")""¨"¬"”"]!¡!’!!£"]$H$Ù); - geodescent, on 10/12/2007, -3/+22If it could speed Digg up a bit, I'm all for it
- Mootabolife, on 10/12/2007, -3/+17It would speed up the digg effect, thereby negatively impacting the rest of the net.
- nwily, on 10/12/2007, -1/+132 Major reasons: CPU intensive and response time.
mod_gzip can be very CPU intensive. On most heavier sights, they can "pre-gzip" a heavily loaded page. This way, every time they get hit for a page load, they aren't re-zipping a page. (I know slashdot works like this).
If EVERY page digg serves is dynamic, you can't pre-gzip the pages. This means you have to eat a TON of CPU utilization.
Also, the one technology that suffers from mod_gzip (afaik) is AJAX. Because it's compressed, it inherently adds some lag to the page load. Ajax is used to make it seem more fluid, but when all of your page loads take an extra half a second, it stops feeling so asynchronous... - ashnil, on 08/09/2009, -0/+8The general idea is ... Less bandwidth utilized = more savings = partial investment in hardware (to cope up with server processing for Gzip)
A precise picture can only be ascertained by Digg technical staff (who might know the actual figures and economics of this implementation). But considering the vast amount of traffic Digg drives, page compression is a great proposition (IMHO). - Sox2, on 10/12/2007, -1/+8some popular sites are gzipped according to my thorough investigation (n=4): slashdot and amazon are gzipped and enjoy 76% and 85% saving, respectively;
looks like quite a few popular sites (bbc news, gizmodo) are not and could save substantial amounts. I know zip about gzip, so anyone want to explain why site might not be doing this? is it as simple as the strain placed on the server having to do the compression? - grendelboogie, on 10/12/2007, -0/+7I think there is something wrong with their script. If you run a packet capture while browsing digg, you'll see html and css being compressed. Javascript is not currently being compressed. Digg.com doesn't use mod_gzip though.
- inactive, on 10/12/2007, -3/+9yeah, im pretty sure the people at the top of one of the leading tech sites don't know about this miraculous gzip.
- hrtattx, on 10/12/2007, -0/+6did you click the link?
- inactive, on 10/12/2007, -3/+9Where does the page size of 41kb come frm?
Opera's telling me:
This page is 7415 bytes
Inline crapola 293427 bytes
If digg was written properly instead of importing entire, huge libraries to do a handful of basic things digg could save ~280kb. Then again, if it was written properly it wouldn't need 75 servers each handling a whopping pageview a second.
http://plentyoffish.wordpress.com/2006/10/08/digg-is-doomed-unless-they-fire-their-tech-staff/
Libraries are all good and fine but generally it's more efficient to write it yourself. - canewediggit, on 10/12/2007, -0/+5never heard of this before, could someone shed some light? if the entire site gzipped does that save bandwidth for just the user or the user and the site host's server?
- ryouko, on 10/12/2007, -0/+4the only reason I would digg this is to see it on diggnation to get a real answer to why it isn't compressed.
- Silencer7, on 10/12/2007, -1/+5*itchy grammar neurons* can't resist, so sorry.
"If Digg were Gzipped, everyone would save 78% bandwidth" - JasonHilton, on 10/12/2007, -0/+3It would save bandwidth, but the site will be slower.
- vdxc, on 09/29/2008, -0/+3A much better analysis can be found at webpageanalyzer.com
http://www.websiteoptimization.com/services/analyze/wso.php?url=http://digg.com/
Intetesting things from the report linked above:
- "Total Size: 163204 bytes"
- "# Congratulations. This site is using HTTP compression, otherwise called content encoding using gzip. The sizes reported here are for compressed content sent from the server to the client." - ebob9, on 10/12/2007, -0/+2There are Network Hardware vendors that make devices offload devices specifically to Gzip dynamic content, if people don't want to bother doing it themselves (or don't want to deal with the server CPU utilization).
http://www.f5demo.com/compression/
http://www.radware.com/content/products/appxcel/compression/
Just to name a couple. - vdxc, on 09/29/2008, -0/+2@yoda133113:
Servers aren't a one time cost. eg/. Electricity and maintenance (replacing old tech) are an on-going thing. If servers were a one-payment thing there would be many more websites. Servers aren't a one time cost. However I still think it would be cheaper than more bandwidth usuage. - MrDo, on 10/12/2007, -0/+1Both Apache and IIS can be configured to compress content with gzip. A bit of tuning might be needed to ensure that the server does not compress content like graphics.
With static pages it is/was(havent tried it for some time) possible, at least with the Apache module, to place a gzipped file with the original in the webserver directory and have Apache serve that instead if the browser reguesting the page, accepts it.
A downside to gzipping, besides from the extra CPU usage, can be that some proxies will not cache gzipped pages but that isn't much of an issue if the sites is mostly dynamic content.
There are commercial solutions available where you place dedicated boxes in front of you site which will compress the content and offload the webserver. But those are expensive.
If you run a server on your DSL line at home, gzipping can really speed up your site if it is mostly serving text pages. Since home servers usually has few viewers and page servings(compared to fx. Digg) the CPU load penalty is worth it.
I am not big on IIS, but last time I looked at one, it was a checkbox in the configuration. I have no practical experience with it.
On Apache 2, check out mod_deflate http://httpd.apache.org/docs/2.0/mod/mod_deflate.html
On Apache 1, it is called mod_gzip (http://sourceforge.net/projects/mod-gzip/) I haven't tried it for a few years so I don't know the status of it.
A few older browsers did report that they supported gzip in their request even though they didn't work.
I have a mental note about some problem with mod_gzip, the http header vary:* and proxies/browser caching. I can't remember the issue(except that it obviously are about caching) but I would google it if I was to configure a site. - Bamborzled, on 10/12/2007, -0/+1@hufman
It's called the subjunctive mood.
http://en.wikipedia.org/wiki/Subjunctive_mood#The_subjunctive_in_English - MrDo, on 10/12/2007, -0/+1No, you make a browser match directive that will make it server regular pages to those browsers.
- inactive, on 10/12/2007, -0/+1A team of JS people? The stuff it does is could be rewritten in an afternoon or a day at most by a single person good at JS. For a project I've got on I emulated certain aspects of digg's functionality (sliding in / out, voting, adjusting the score + adding new comments via JS) for a total of 8kb.
Did you factor in the chicklets? They would add millions (maybe 10's of) of downloads to those files. A quarter meg x millions is a hell of a lot of gigabytes a day.
Th JS I wrote, with bloated commenting (it may not make much sense out of context):
http://pastebin.com/884875 - hufman, on 10/12/2007, -0/+1Digg is singular, so I do believe that "was" is correct.
An apple was eaten. Two apples were eaten. - groverblue, on 10/12/2007, -0/+1http://www.websiteoptimization.com/speed/tweak/compress/
- gabacho2, on 10/12/2007, -2/+3I think one of the problems might be is that you cant slowly send page output to the client, so the page wouldn't slowly start showing comments as they were loaded; the client would have to wait until it received the whole gzipped page.
- i4mt3hwin, on 10/12/2007, -0/+1You make a valid point, with all the comments and updates going on with this site the CPU would be raped.
- zlintux, on 10/12/2007, -0/+1bah, was beaten to it...
- inactive, on 10/12/2007, -0/+1@nwily
So they never need to be downloaded? Is that what you're saying .... like if I use a different browser or computer the files are pre-cached magically?
Did I mention that the "digg chicklet" embedded on third party sites also embeds the 290kb of crap? Suddenly you maybe have a few million people downloading those files a day and you're sharing the slowdown with sites silly enough to embed a ridiculously architectured feature.
Efficient css/javascript + gzip could make digg comparible to google in speed. Instead it's a trip down dialup memory lane. - ChrisJP, on 10/12/2007, -0/+1Well, most forum software (VBulletin and IPB for example) have GZipping turned on by default, and don't forget they're dynamic too, with all the posts etc. Not too dissimilar to Digg. I see many comments about the CPU usage gzipping requires - so what the CPU usage of large forums?
Grab any URL listed on big-boards and try it. I tried forums.offtopic.com - a massive forum with gzipped pages. I would assume their CPU's cope fine with this considering their size and amount of page requests they must get everyday.
I've just added it to the main pages of my site so I guess I'll found out for myself over the next few days/weeks. - grendelboogie, on 10/12/2007, -0/+1If you run a packet capture while browsing digg, you'll see html and css are compressed. Javascript is not currently being compressed. Digg.com doesn't use mod_gzip though.
Or, http://www.websiteoptimization.com/services/analyze/wso.php?url=http://digg.com can give you some more detail. Thanks @ mporcheron! - abid786, on 10/12/2007, -0/+1Have you read any of the 36 comments on this page before posting?
- Loonacy, on 10/12/2007, -0/+1@GMorgan
"It wouldn't cost the servers much for compression since they only have to do that once and serve many times."
This would be true except for the fact that most content on Digg is dynamic. On the front page, it has your username in a link, so the front page is different for every user. Not to mention not every user sees the same front page (story filters, blocked users, friend highlighting, etc.).
Even aside from that, you'd still have to recompress the front page every time it changed, and recompress comment pages every time a new comment was added. - grendelboogie, on 10/12/2007, -0/+1Good comments! You're right about the CPU load gzip compression puts on the Web servers. In Digg's case, we prefer not to bother the Web servers with that task. We let our load-balancers handle the compression.
- yoda133113, on 10/12/2007, -0/+1Well, the CPU usage thing isn't a good argument against it, the one-time cost of additional servers is less than the cost of additional bandwidth for months.
The AJAX argument might be a VERY good one though, IDK enough about AJAX. - l008com, on 10/12/2007, -0/+1Hi Guys
I am the creator of www.whatsmyip.org and all of its tools. Heres some more information for you. This site (digg.com) does appear to be compressed already. My tool doesn't say it is compressed because for some very odd reason, this server doesn't seem to be sending out the "Content-Encoding:" header letting me know what kind of data it is sending me. I'm not sure how the browsers figure this out.
Also as far as GZIPing in general, it is a great way to save bandwidth. However it does not run at the expense of CPU. Yes the CPU does have to compress each page, but because each connection is so much shorter now, most servers are actually going to see a significant DROP in CPU usage in addition to the drop in bandwidth. GZIP really is a great thing and really every site should use it.
So if anyone knows a little more about this than I do, feel free to let me know how this particular server is letting browsers know it's content is GZip'ed. - KillerJ59J, on 10/12/2007, -1/+1Apparently they don't. I wouldn't really consider Digg a 'tech' site anymore.
- GMorgan, on 10/12/2007, -2/+2Both. They save on uploads, user saves on downloads. The only loss would be CPU cycles for decompression (and I think it would add some dependencies to the site). It wouldn't cost the servers much for compression since they only have to do that once and serve many times.
- friedcalamari, on 10/12/2007, -3/+3Well it won’t benefit the bandwidth vendors.
- audiophiliac, on 10/12/2007, -3/+3What's a "dial-up" modem lol
- ashnil, on 08/09/2009, -0/+0Off-topic:
I wonder why this post doesn't show up in the "top in 24 hours" list: http://www.digg.com/tech_news/popular/24hours , and even elsewhere in the popular or upcoming stories sections? - nwily, on 10/12/2007, -1/+0@fkr2 I'm not sure one person could do it in an afternoon. I'm sure you or I could, and make it work on our browser, but thats not how it is. Most places I've worked, JS has to be designed to work with 3 or 4 major browsers, and 2 or 3 versions of each browser. The compatibility testing alone can take you most of an afternoon. And that's not even getting into the JS/CSS interplay issues.
Again, I agree, the bandwidth savings are significant, and for most people it would be worth it. But you have limited time/people, you have to pick your battles. I'd be surprised if it wasn't on their "todo" list (or more likely, figure out if this is a good use of time), but I'm not at all surprised they haven't done it. - chingy1788, on 10/12/2007, -9/+8The Tubes will get clogged Quick get the internet Plumbers, they come in a big red truck
- nwily, on 10/12/2007, -1/+0@fk2 Obviously you are correct, they still need to be downloaded. But if you're a regular digg user, and lets say you have 10-20 page view a day, digg is still only serving those .js files once. So if you want to figure digg's real saving's for writing their own JS library, here's the formula: Size of Javascript included/(number of page views it's cached for).
So let's say it's cached for 10 page views. That means they are only serving 27k of JS per page load. Again, more than you'd like, but not the worst I've seen. The question is this: is it still financially a good move to dedicate a team of JS people to handle your personal library, or save XX dollars a year by using existing libraries and spending the extra money on bandwidth. I'd be surprised if someone their hasn't don the math. - bigal90, on 10/12/2007, -3/+1sounds like a good idea to me, as long as theres no catch like a massive amount of CPU usage
- bwzippy, on 10/12/2007, -3/+1Hi, I sell network routers and switches. Please don't GZIP this site because I WANT people to use more bandwidth!
Besides, if you gzip this page, you'll probably kill my cellphone when it tries to gunzip the information. - jonathono2000, on 10/12/2007, -4/+2How do I know if a site is sending g-zipped content?
- KillerJ59J, on 10/12/2007, -3/+1Hehe, myspace is a gzipper, also.
- rhymer, on 10/12/2007, -2/+0Users can always use their own compression software!
I use 'Traffic Compressor' which is in beta and free at the moment.
It saves oodles of bandwidth! - ndiderrich, on 10/12/2007, -3/+1Well if it's going to save 75% more bandwidth then there has to be a catch, I say go ahead and do it if there is no loss in anything:
And those questions are what Mike is asking on his post. - SuperCUBE, on 10/12/2007, -4/+1This is a great idea. I'd do it.
-
Show 51 - 56 of 56 discussions
© Digg Inc. 2009
— Content created and posted by Digg users is