digg

Facebook Connect Join Digg About

Facebook Rolls Out New Storage System - Haystack

computerworld.com — Engineers at Facebook are installing a new photo storage system they say is 50% faster than traditional systems.

Who dugg this? Made popular Apr 4, 2009

Digg DudeWhat is Digg?

Digg is a place for people to discover and share content from anywhere on the web. Digg surfaces the best stuff as submitted and voted on by the community. Take the Tour to learn more.

84 Comments

show profanity settings
expand all
  • iguanapunk, on 04/04/2009, -1/+95but if it's secure how am I supposed to get pics of totally hawt teens?
  • Iwantawii, on 04/04/2009, -1/+86I would rather they work on higher resolution photo albums.
  • inactive, on 04/04/2009, -3/+75'Johnson noted that Facebook deals with 15 billion photos - not including all of the replications. User data grows by 500GB per day.' try deleting stuff after the user has opted to remove the image from their profile..
  • inactive, on 04/04/2009, -0/+54try deleting a photo on facebook, you cannot get it back. so why do facebook need to retain the image file..?
  • wastedthelight, on 04/04/2009, -0/+42I'm sorry, i thought "delete" meant to save it over "there" for a while until i wanted it back. My bad.
  • Carsonauto, on 04/04/2009, -1/+35How does that work.

    They're different sites for different purposes, chum.
  • strangewill, on 04/04/2009, -5/+34And then a user bitches that they didn't mean to do it and want it reverted.

    You have to program for idiots.
  • braveryonions, on 04/04/2009, -0/+27IPHONEZ IS BETTAH THAN XBOX
  • ajmuir, on 04/04/2009, -2/+28..as in "like searching for a needle in a..." ?
  • fattony89, on 04/04/2009, -0/+21whether you like or hate facebook, the tech to keep should a huge amount of data and the volume of requests is amazing.
  • richid, on 04/04/2009, -0/+18Facebook uses NetApp Filers on the backend but that is not the point. Their problem wasn't how to store all this data, it was how to retrieve it EXTREMELY quickly. They were hitting a ceiling when their NetApp's were doing about 15 IO's per photo request. They rearranged the file and directory structure and got it down to 3 IO's per photo request. Haystack let's them write a ton of photos into a series of 10GB files. They then store metadata about the photos in memory. When a request comes in they retrieve the inode and offset (the metadata) from memory and can retrieve the photo using only 1 disk IO. In my opinion, that's pretty awesome stuff.

    Have you ever followed their developer blog or looked at their projects? They develop a lot of stuff "in house" and some of it is open sourced, like their big memcache contributions. They've also done some awesome work modifying MySQL to automatically update memcache keys. I suggest you check it out.

    I'm curious, what do your IT guys manage that is 4PB?
  • Myonosken, on 04/04/2009, -0/+17They're totally different sites, how are they ahead?

    If you mean technology wise, you might want to check Digg a bit more. The comment system is still pretty bad (although not as bad as it was) and the front page algorithm/ friends system is still terrible.
  • zeptobyte, on 04/04/2009, -0/+15Ahead in.. what way?
  • Archaic1, on 04/04/2009, -2/+16How fast they haemorrhage money compared to the other?
  • jjones20, on 04/04/2009, -0/+14i think they reverted the TOS to not include that line after everyone got all uppity about it.
  • HurricaneDC, on 04/04/2009, -2/+15...you're saying it's not secure? Teach me.
  • Majora26, on 04/04/2009, -0/+13Yea. they offer high quality video, but no high-res photos?
  • sonofabiscuit, on 04/04/2009, -0/+13If only the government was that responsive to criticism.
  • Thoku, on 04/04/2009, -2/+14OMG this system just lost my photo of a needle. How will I ever find it?!
  • jibbist, on 04/03/2009, -5/+17I hope this is more secure than the last system!
  • mynameistux, on 04/04/2009, -4/+14do they still own anything I put on facebook?
  • DLit, on 04/04/2009, -1/+10sweet now we can stalk 50% faster!
  • MVPhurricane, on 04/04/2009, -0/+8clever. i'll bet they hadn't thought of that one.
  • electrofreak2k7, on 04/04/2009, -0/+7actually, it was horribly slow. Sometimes photos would come up instantly, sometimes they'd take 5-60 seconds. Sometimes photos simply never came up.
  • purseonality, on 04/04/2009, -1/+7you betcha
  • jemmrich, on 04/04/2009, -0/+5Thanks for the great idea on file structure. I been using file hashes in my current project, but never thought about using the first 3 digits in the path as well. I was going to use dates, but didnt think that would work to well.
  • dunker, on 04/04/2009, -0/+5And your string before applying the md5 hash was ... "digg". :) L33t H4x indeed! :)
  • Jerryrad, on 04/04/2009, -6/+10Facebook owns the internet now... well, them and Google. Actually, I think Google owns and/or created the universe.
  • Chakat, on 04/04/2009, -2/+6I wonder how this compares to Livejournal's MogileFS. Seems very similar at least from the descriptions of a flat namespace and everything being just a reference.

    http://www.danga.com/mogilefs/
  • Daniel591992, on 04/04/2009, -0/+4So don't. I doubt they care.
  • homercles337, on 04/04/2009, -1/+5Thanks for the info. Very interesting.

    Im a scientist at a joint harvard/mit institute for bio research. Much of what we do is considered "data rich." High content, throughput, sequencing, etc. If you did a quick google im sure you could find it.
  • sipsyrup, on 04/04/2009, -0/+3If you're not on it, what harm does it cause you?
  • shnuffy, on 04/04/2009, -1/+4Stop yelling at me.. :(
  • tpmidd, on 04/04/2009, -0/+3Mavericky
  • Bakorork, on 04/04/2009, -1/+4What they reverted it back to doesn't look any more appealing to me...
  • poutch, on 04/04/2009, -0/+2man, that was rude!
  • mGARANDEUR1, on 04/04/2009, -1/+3It's probably a wise move to keep the photos at a medium resolution. Let's face it, most people do not own a 30" monitor and do not really care about the photo size or resolution. I have some friends on there that have over a thousand photos each.
  • darlingt, on 04/04/2009, -0/+2Wait for the kid in the ball pit to get poked by it and have his parents sue you?
  • aserer511, on 04/04/2009, -0/+2the name is such because finding a good picture of someone on facebook, is like, well, y'know
  • strangewill, on 04/04/2009, -0/+2@algo: Have you contacted their technical support when you deleted it?

    @wastedthelight: people are DUMB, welcome to software development for the masses.
  • richid, on 04/05/2009, -0/+2@bilbus, chakat: See my post below. The majority of their photos are served through Akamai, their CDN. However, they have an array of NetApp Filers at the origin where the photos ultimately reside. Second, MogileFS is not "some crappy system." Mogile is a great solution...as long you don't need to handle 200k req/s for photos.
  • DavidTurnbull, on 04/05/2009, -0/+2Indeed. I drew a picture of a turtle and uploaded it to Facebook and I'd be very embarrased if it got leaked all over the internet.
  • mynameistux, on 04/04/2009, -4/+6no, it was a serious question, do the TOS mean they own all my info, if so, ***** it, I will link to my photos on imageshack or somthing. and ***** YOU FACEBOOK
    If not,I will put all my photos there, also I love facebook
  • SuperMoses, on 04/04/2009, -0/+2Solution: NoseRub - http://www.noserub.com

    If you really hate the fact that one company owns your data then support the decentralize social networks initiative. Own your own data, store it on your own server if you want or sign up with a noserub service (ex: Identoo.com). It's the only way people can have full control over their information.
  • richid, on 04/04/2009, -0/+2@bilbus

    Developer blog: http://developers.facebook.com/news.php?tab=blog

    Presentation on Haystack from about a year ago: http://www.flowgram.com/p/2qi3k8eicrfgkv/
  • Chakat, on 04/05/2009, -0/+2If they were using something like a SAN, they'd be saying such. The rest of Facebook is built around a lot of small servers -- thousands of app servers, memcached servers, etc. -- it only makes sense to have their storage servers be of a similar decentralized architecture.
  • bbliss17, on 04/04/2009, -0/+2umm no dude no
  • richid, on 04/04/2009, -0/+2Their main problem was the time it took to retrieve a photo due to the metadata associated with each file, mtime, atime, etc.

    Check out this presentation: http://www.flowgram.com/p/2qi3k8eicrfgkv/ I believe it's about a year old now so the numbers have increased somewhat. I think you'll agree that what they are doing is pretty bad-ass.
  • poyboy, on 06/24/2009, -0/+1Wrong! It is "SLOW DOWN COWBOY!"
  • bilbus, on 04/04/2009, -0/+1ummm thats the same as MS DFS .. no big deal

    I am sure they are using a real solution like a san, not some crappy system.
  • Fetching more discussions...
    Show 51 - 88 of 88 discussions

  • Add a Comment

    Login or register to comment!
    Or, sign-in super fast with your Facebook account ...


© Digg Inc. 2009 — Content created and posted by Digg users is dedicated to the public domain | Terms of Use Updated | Privacy Policy
DIGG, DIGG IT, DUGG, DIGG THIS, Digg graphics, logos, designs, page headers, button icons, scripts, and other service names are the trademarks of Digg Inc.