digg

Facebook Connect Join Digg About

Google's Distrubuted computing watch!

norfolk.cs.washington.edu — This is a talk that was given in Norfolk Washington by an engineer that works inside a datacenter for google. He describes how their datacenters have progressed and how they maintain the machines in the datacenters.

Who dugg this? Made popular Dec 30, 2007

Digg DudeWhat is Digg?

Digg is a place for people to discover and share content from anywhere on the web. Digg surfaces the best stuff as submitted and voted on by the community. Take the Tour to learn more.

76 Comments

show profanity settings
expand all
  • NihilFist, on 12/30/2007, -2/+35distrubuted?
  • zeromancer, on 12/30/2007, -0/+30I wouldn't go so far as to say that "a lot of us have a few TB in our desktops." I have 800gb and i'm a nerd. i know lots of nerds that would envy my storage capacity. But I think it safe to say that most computer users still have ~120gb in their machines, If you think about it, the type of data matters a lot. 10s of TBs of text is unfathomable, while 10s of TBs of video is still a huge amount, but much easier to grasp conceptually.
  • meatmcguffin, on 12/30/2007, -0/+28Now you're thinking with portals
  • Lazrath, on 12/30/2007, -1/+27media link:
    mms://videosrv6.cs.washington.edu/talks/Colloquia/JDean_041021_OnDemand_100_256K_320x240.wmv

    original link didnt work for me, asked me to download the .cgi file
  • cenarta, on 12/30/2007, -0/+26I absolutely love videos like this where you get a peek into great engineering minds and projects. Me being a network engineer/programmer for a small company, these videos help me immensely in charting out where we should go datacenter/programming wise. Google's clusters are simply amazing and the approach of using commodity hardware on a massive scale is very impressive. I know this video is older, but thank you for posting it as I had never seen it before.
  • ionut, on 12/30/2007, -0/+22The same presentation, but in a different location:
    http://www.youtube.com/watch?v=TQKs27Yqg-8
  • isunktheship, on 12/30/2007, -4/+20Oh thanks for interrupting my music, that wasn't obnoxious or anything.
  • isunktheship, on 12/30/2007, -2/+17he actually meant "we have tens of terabytes OF terabytes"
  • danconia, on 12/30/2007, -2/+14I'm not sure that makes sense, that's like me saying "tens of miles of miles".
  • richbradshaw, on 12/30/2007, -0/+9Exactly, 10TB of compressed text is an unimaginably large amount of info.
  • Soave, on 12/30/2007, -2/+10Heh, I read it as "disturbed" the first time.
  • WMGoBuffs, on 12/30/2007, -0/+6I find that a lot of scientists start sentences with "So..." as a pedagogical technique, and as a stalling technique to parse complex things into english.
  • indiekiduk, on 12/30/2007, -3/+9um um um um um um @£$%^&*
  • theboywonder, on 12/30/2007, -0/+6Video file name has "041021" in it.
    21st Oct 2004?
  • isunktheship, on 12/30/2007, -2/+7haha YES, someone caught on!
  • clearzen, on 12/30/2007, -4/+9whoops
  • dvd1972, on 12/30/2007, -4/+8He uses Valley Girl Speak; he says the following phrase 53 times, "OK, so".
    Sometimes he mixes it up by, "OK, sooooo (clear throat). Then, "OK, so, (giggle).
  • fugazi, on 12/30/2007, -0/+4When was this filmed?
  • mikecicc, on 12/30/2007, -1/+5god dammit I am so incredibly mad i clicked this link..
  • NeoPlatonist, on 12/30/2007, -0/+4How about a youtube link?
  • mjPayne, on 12/30/2007, -0/+4Well, they don't, but not because of what you think. There are only few thousands words in English, they store pointers to these words in huge indices. Of course this is very simplified model, but it shows how a search engine can deal with so many pages.
  • Ksg89, on 12/30/2007, -0/+4My old teacher at high school said OK 120+ times in an hour lesson... (yes i counted)
  • spargo, on 12/30/2007, -0/+3From about 22:30 -- "Petabyte-sized file systems" (talking about GFS, the Google File System)
  • Lazrath, on 12/31/2007, -0/+3http://img267.imageshack.us/img267/1866/vlcsnap234 ...
  • nooreazy, on 12/31/2007, -0/+3041021 = Inside Job
  • reginaldino, on 12/31/2007, -0/+2buried for such an awful presentation. wish somebody didn't point out the "ummm, so" combinations. Bloody annoying
  • Dohko_Xar, on 12/31/2007, -0/+2His hands are so distracting..
  • inactive, on 12/30/2007, -1/+3Has anyone else noticed that grammar and spelling have gotten better on the front page recently? Typos such as this one are getting rarer. Are they tossing the poorly spelled headlines out?
  • OneZeroZeroOne, on 12/30/2007, -0/+2I should have gone to school there. Goddam.
  • T440, on 12/30/2007, -0/+2So that was kind-of cool...
  • goobit, on 12/30/2007, -0/+2In terms of just webpages, the sizes are in KB. That's hardly anything. A PC with TBs worth of storage can hold a lot of pages. Think of it that way
  • fugazi, on 12/30/2007, -1/+3Looks like around 2002... I would want to see more of the more modern google.
  • BlackStrain, on 12/30/2007, -1/+3Well, more browsers have built in spell checking now. I think that was the nicest addition in Firefox 2.
  • linagee, on 01/30/2008, -0/+1True... Googlecache and all. Which really makes me wonder how large of a document Googlecache will store? Hrm...
  • JoeRW, on 12/31/2007, -0/+1Bloody computer scientists have been trying to use the word 'shard' for some important data fragment for ever. It's assinine. Google will not be able to download the whole Internet for very long, it's a primitive idea.
  • surfing, on 12/31/2007, -0/+1see below
  • shinderpaljandu, on 02/03/2008, -0/+1the date can be figured out as 2004 - there is a reference to an article that "we published last year 2003"
  • newwatch51, on 01/03/2008, -0/+1I don't believe TB HD's are very common yet.
  • macwac, on 12/30/2007, -0/+1I believe in distributed networks when it comes to information, energy etc... rather than centralized systems that break down and the entire network of information, energy etc.. is lost. Thumbs up!
  • cohman2001, on 12/30/2007, -0/+1He said that the average size of a web page is 10KB. So if they were to cache the images on the web page, the average size might go up to about 750KB or even 1.5MB.
  • Trevahaha, on 12/31/2007, -0/+1He's referring to what is in their index. Their index is just text (or meta data about images).. that is in the terrabytes. Obviously their cache, gmail, apps, etc. is a lot more than that.
  • antdude, on 12/31/2007, -0/+1http://norfolk.cs.washington.edu/htbin-post/unrest ... was the link.
  • STARTSOMETHING, on 01/01/2008, -0/+1You took the words right out of my mouth.
  • brettalton, on 12/31/2007, -0/+1wget link?
  • Mononuclear, on 12/30/2007, -0/+1Norfolk? This is at the University of Washington... in Seattle..
  • Trevahaha, on 12/31/2007, -0/+1I agree... I recognize classmates in the video... so it was definitely a few years ago (I graduated from UW in 2005)
  • gummih, on 12/31/2007, -0/+1Termites, they have a huge problem with termites/bugs in their storage facilities
  • Lazrath, on 12/31/2007, -1/+1that link still asks to download the 'details.cgi' file, in firefox
  • WMGoBuffs, on 12/30/2007, -0/+0I work in experimental particle physics, and our data is stored in less than 7TB. That's a lot! You have to figure, though, that that figure doesn't include all the images on those pages...
  • linagee, on 12/30/2007, -2/+2Google doesn't have to store the whole page either. They could chop off their crawler after the first 20kB or so.
  • Fetching more discussions...
    Show 51 - 72 of 72 discussions

  • Add a Comment

    Login or register to comment!
    Or, sign-in super fast with your Facebook account ...

© Digg Inc. 2009 — Content created and posted by Digg users is dedicated to the public domain | Terms of Use Updated | Privacy Policy
DIGG, DIGG IT, DUGG, DIGG THIS, Digg graphics, logos, designs, page headers, button icons, scripts, and other service names are the trademarks of Digg Inc.