digg

Facebook Connect Join Digg About

Build Your Own Supercomputer!

clusterfudge.wordpress.com — Use the cheap computers you have lying around, or pick up some cheap economy PCs and turn it into a thread-ravaging monster.

Who dugg this? Made popular Jan 10, 2007

Digg DudeWhat is Digg?

Digg is a place for people to discover and share content from anywhere on the web. Digg surfaces the best stuff as submitted and voted on by the community. Take the Tour to learn more.

70 Comments

show profanity settings
expand all
  • vertinox, on 10/12/2007, -1/+38No, but it will Fold proteins.
  • Sakumi, on 10/12/2007, -3/+29It suggests purchasing an eMachine? Better buy a backup power-supply and motherboard while your at it.

    I work at a PC shop and when I see a eMachine walk through the doors, it's usually for 'it won't boot'. After investigation it shows the power supply blew out the motherboard. This is happening in their PCs from 3 years ago through to the newer ones today. I think its the shoddy 'Bestec' PSUs.
  • bobcrotch, on 10/12/2007, -0/+20No he's right don't burry his comment. I also worked at a PC shop for about 5 years and I could easily count (in about a year and a half) 40 eMachines that had their power supplies blow up and kill the motherboard as well.

    Beware of the Bestec power supplies in there!
  • Dpack1, on 10/12/2007, -0/+19Is it??? It is... It's an article about a beowulf cluster and not a single joke about it!
  • ThunderIT, on 10/12/2007, -0/+14@jacks0n:

    Currently, most games cannot even make use for a second processor or core in the computer they are running on, so it makes sense that they are not able to work via remote cpus in a cluster
  • NinjaBoy, on 10/12/2007, -1/+15Apparently it can make a word press site handle the digg effect.
  • thesteampunk, on 10/12/2007, -2/+15This was incredibly noninformative. The resources are better at Oreilly.com. No Digg.
  • seuaniu, on 10/12/2007, -1/+14@hiney

    Step 1. buy a bunch of pcs and network them

    Step 2. Burn a bunch of clusterknoppix cds (http://clusterknoppix.sw.be/)

    Step 3. Relax, its not as hard as it looks.
  • 98acura, on 10/12/2007, -3/+16What is the purpose of your comment? Just to prove to everyone that you are a tool?
  • netferret, on 10/12/2007, -1/+13can it run solitaire?
  • Kale, on 10/12/2007, -1/+13Two things: 1) Don't buy the computer listed, since you are paying for an OS you won't use!! Find one without any OS.
    2) Factor in power requirements. It is cheaper, both hardware-wise and power-usage wise, to buy the fastest class of chip at the lowest clock speed (usually). Two 1 GHz P4s will have more cost of ownership than one 2 GHz machine. A 2 GHz C2D will probably have even lower cost per flop. However, investing in the fastest C2D will probably cost more per flop than buying 1 more 2 GHz C2D. I calculated up how much I would have to spend to get a cluster of XBoxes vs. a single computer for a GMP algorithm, and it was less than half the initial cost to build a single computer with the same speed (for my algorithm) over buying 7 or 8 XBoxes, not even considering hassles of multithreading and power costs.
  • Humptydank, on 10/12/2007, -0/+12It always gets my confidence when someone recommends eMachines as step number one.

    I wouldn't keep my cereal in an eMachines box.
  • MannaPC, on 10/12/2007, -2/+11Agree'd. He lost me after "for a total of 5.32ghz per machine"
  • superKduper, on 10/12/2007, -3/+12You're all wrong! Just buy an Apple G4. Steve Jobs told me it was a Supercomputer! 64-bits of power! And all the mac users skipped around saying "thuper-duper computer".
  • cdlavalle, on 10/12/2007, -1/+10A: Anything that can be broken up into discrete units and easily spread across many threads of execution without too many dependencies between threads (which defeats the purpose). Now, I don't know about OCR but there are so many programs I'm sure you can find one that supports it. Or go to sourcesourge/freshmeat, find one that lends itself to parallel computation and get to work. I deal with 3dS Max and it has built in support for parallel rendering so if you can find something like that it is ideal.
  • ROFLance, on 10/12/2007, -3/+12This article sucked=
  • alexmipego, on 10/12/2007, -0/+7Many things can be run on this clusters but besides the fact that what you're processing needs to be split in small pieces that could be tied together later there is one major thing to keep in mind. This is specially important to understand why you can't, for example, run games on a cluster, that is, modern-graphic games.

    The problem, and altough I'm not an expert I know there is a theory/calculations about this, is that not only you need to be able to split the work in pieces but both the to-be-worked data and the result-data size counts. For instance, if you are providing formulas and expect as return the first 1 million computed results from each node that wouldn't work so well because the time to transfer those numbers over the network could be greater than simply letting the master machine to process that data. Games for instance, would need to generate around 50 frames/images at a resolution of 1024x768x32 lossless, thats a pretty big file to be transfered 50 times a second over a network.

    I hope you got the point.
  • dmoore764, on 10/12/2007, -0/+6From the article: "How else could I play Halo while factoring 100 digit prime numbers?"

    I can factor 100 digit prime numbers in my head: 1 and the number
  • farrellj, on 10/12/2007, -3/+9I am fairly sure you can Blend, as I think that Blender can make good use of threads...but you might get better throughput using all those computers as a Renderfarm rather than an OpenMosix cluster.

    You can check the main Blender site: http://www.blender.org/cms/Home.2.0.html

    ttyl
  • Jugalator, on 10/12/2007, -0/+5Am I supposed to treat those blender/tube comedians as trolls or actually funny? :p
    Or maybe it's just some form of obsessive-compulsive disorder.
  • igorparsadanov, on 10/12/2007, -0/+5What do you run on something like this? Say I have a massive OCR job how would I run it on it. Or a huge batch image conversion?
  • KWhat, on 10/12/2007, -0/+4If you think linux is hard to cluster with knoppix try getting windows to work =P
  • vertinox, on 10/12/2007, -0/+4eMachines were the new Packard Bells of their time.

    But hey... They were cheap.
  • LiquidPenguin, on 10/12/2007, -1/+5@cdlavalle

    Contrary to popular belief, many (most?) games lend themselves very well to parallel processing, just not the way most of them are currently written. It's also not that much harder to create such an application, it's all really a matter of perspective and experience. A very basic (though not ideal) method would be to offload all of the A.I. to one CPU while the other CPU does the rest of the work.

    For instance, A well known developer for the SEGA Saturn (and arcade) lamented at the difficulty at porting his games to single CPU platforms (ie Dreamcast). Trying to force all the processing onto one CPU was hurting the headroom he had with a dual CPU design. I began to realize then that the reason why multi-CPU platforms haven't really caught on (at the time) wasn't because of cost or performance issues, it was because so many developers wrote software for single CPU architecture, that they couldn't think in terms of mult-CPU architectures. Look at some of the multi-CPU consoles like the PC Engine, Saturn and Jaguar and compare their games to their competition at the time. Even to this day, AvP for the Jaguar is still regarded as one of the best examples of the series ever.

    If all a person ever developed games for was one CPU and one GPU, then yes, multiple CPUs or even multiple GPUs are going to look very difficult to write code for. But once you start diving into it, then you begin to realize that you have a lot more wiggle room in terms of performance, you just have to have more care with how you manage your processes and divide the work.
  • Ramble, on 10/12/2007, -2/+6Is that a Beowulf cluster or are you just happy to see me?
  • KWhat, on 10/12/2007, -1/+4@cdlavalle

    I would love to see some of those articles because I was under the impression thats why we had THREADS for parallel processing! Threads are very easy to implement and if the OS (Kernel) handles threads correctly it should balance the load across CPU's. Clustering on the other had is for very specific tasks. I cant have lets say a cluster to just balance my Threads for everything. It has to be something of very specific calculation like graphics processing (Rendering) or number theory (Calculate Pi). I cant tell the SETI cluster to render my new trailer for Cars nor can I ask Pixar's cluster to Look for Aliens. Specific!

    Bahh 2 Cents
  • MrFlesh, on 10/17/2007, -0/+3I've got a better way. If your gonnna run a cluster use a machine that is designed for clustering from the CPU up. Use 20gig PS3s. They're cheaper than the emachines and the cell proccessor is designed for distributed computing. So what if your basement looks like a george forman cook-off.
  • Visk, on 10/12/2007, -2/+5With this, you can run Oblivion at medium settings!
  • thaibox1, on 10/12/2007, -1/+4@hiney

    I got Linux running on my laptop in a couple of hours, full hardware support, no issues. Not sure what you're complaining about but keep banging your head against the wall, clear out the gene pool a little for the rest of us that can read instructions.
  • seuaniu, on 10/12/2007, -0/+33d rendering, scientific calculations, failover servers, video compositing, video transcoding, you name it. There isn't much benefit to having a 10 cpu setup at home unless you're working as a videographer or animator out of your house. I do a lot of video work and would love to have the budget for 10 or 20 extra CPUs at my disposal when I have to encode a bunch of dvds, etc.

    EDIT: as an example, Cinelerra has a renderfarm mode that could be useful, but doesn't have anything to do with this article.
  • cdlavalle, on 10/12/2007, -1/+4@alixmepego

    Good points. Also, most games and programs do not lend themselves to parallel processing (barring the special case of graphics rendering) because compilers are not yet smart enough to compile to take advantage of that capability. And who wants to hand code it? UGH! Many a PHD thesis right now deals with that particular problem, god bless 'em.
  • merreborn, on 10/12/2007, -1/+3"1) Don't buy the computer listed, since you are paying for an OS you won't use!! Find one without any OS."

    The additional cost of the OS is generally made up for by the fact that you're getting all the other components for next to nothing on a low end box. Who cares if you're "paying for an OS" when the box is cheaper than any other equivalent PC on the market? It's kinda like value meals at fast food chains -- a drink, burger and fries sometimes costs *less* than just a drink and a burger.

    Show me somewhere you can buy a dual core processor, motherboard, case w/power supply, gig of ram, 200 gig hd, and optical drive without an OS for less than $500, and then you've got a leg to stand on. Keep in mind that for the price, you're getting a warranty, assembly, and a guaranty that the parts are compatible as well.

    The manufacturers I've seen that offer OS-less or linux pcs only give you maybe a $20 discount, at best, for opting not to use windows.
  • TonyCubed, on 10/12/2007, -1/+3Wonderful story... Anyways...
  • Negyxo, on 10/12/2007, -0/+2@ cdlavalle
    Ohhh the joys of backburner (3ds max distributed network rendering). Brings me back to the days in which i would get a couple friends together to commandeer computer labs in high school. Hearing 64 computers spin up to render your animation is a religious experience. You watch what would take days to render happen in a few hours.
  • cmdrNacho, on 10/12/2007, -0/+2http://www.beowulf.org/overview/faq.html#2

    but yes for most people setting up a beowulf cluster at home is unnecessary.
  • texnofobix, on 10/12/2007, -0/+2It is when you have his knowledge of clusters.
  • Karmalary, on 10/12/2007, -0/+2First of all, this is only one way (and not the best in my opinion) to set up a beowulf cluster. Before deciding to do something like it, decide if you even need it. Few people ever do. A lot of powerful, parallel number crunching can be done (even with Windows boxes) with just a server and some clients. Want distributed raytracing, free? Check out http://www.gammaburst.net/xrs/ A real world use for raytracing software like this running on a Linux-based cluster? Check out the Biowulf cluster running at the National Institutes of Health. http://biowulf.nih.gov/ They use it to design and model organic molecules.
  • inactive, on 10/12/2007, -0/+2I've used some clustering using the apple Xgrid software... I set up 2 computer labs to have Xgrid turned on, and I found a video conversion program called VisualHub that will distribute tasks for batch processing a whole bunch of video files at the same time.,.. Basically, each machine is given 1 of the video files to work on, that is sent back to the master machine when it's done. The total processing power of the cluster is 59.51GHZ!
  • UNL1M1T3D, on 10/12/2007, -0/+1Yup we have a tub full of dead power supplies at my work, and I would say 3/4 of them are Bestec power supplies.
  • merreborn, on 10/12/2007, -0/+1"failover servers"

    For failover, most people *don't* cluster. Generally speaking, it's just better all around to set up a pool of systems with a load balancer. Less overhead. Just as reliable, if not more.

  • JesterOne, on 10/12/2007, -0/+1Log time reader, first time poster... =)
    I've seen a number of these kinds of write-ups before but no one can seem to come up with a good answer as to what I would really do with something like this. I mean, I'm all for doing a project like this as a learning experience but when the dust settles, I can't come up with something to use this for. I'm willing to bet that most people that are tech-savy enough to try something like this don't have a use for one either.

    Prove me wrong.
  • bobmagoo, on 10/12/2007, -0/+1it could be useful,
    i'm just getting into linux so i'm not entirely sure how this would all work, but i do some video editing using Sony Vegas 6(stop laughing, i'm too broke to afford an adequate mac) it has a network rendering component, that i'm pretty sure you could get going with some kind of wine or other emulator on linux.

    but if you're going to do it, i'm not sure i would recommend this guys advice,
    i mean come on, recommending an emachine and then saying that a dual core processor is the equivalent of twice it's clockspeed,
  • vietvet, on 10/12/2007, -1/+2To use this thing as a parallel computer [and not just a pile of PCs mostly generating heat] you have to use "distributed programming" [check google for "message passing interface"]. Which leads to how if it's to get any performance, "some kind of switch" just won't cut it. The interconnecting network is roughly half of the initial cost of a supercomputer these days.

    You won't fold proteins with a pile of junk before the sun goes supernova. Sorry about that.

  • Genghis1, on 10/12/2007, -0/+1Exactly what I was thinking. A cluster of PCs burns a lot of power and that get expensive. PCs convert AC power to DC inefficiently. Where I work, everything runs on 48 volts dc so we don't need individual power supplies in each machine.
  • vipinsagar, on 10/12/2007, -0/+1what was your clusterKnoppix Version. 3.6 ???

    I had tried it with CHAOS on Kernel-2.4.x. It has an option for 2.6.x. and ran prettY well.
    CHAOS does the same and ya onlY need one 6MB CD and it runs on memory.

    its fine for off the shelf, but when people needs computer work as both compute-node and desktop I chose ROCKS

    Have ya heard of ROCKS, check mine

    http://vipinsagar.net/2006/07/19/rocks-clustering-a-review/

  • CompIsMyRx, on 10/12/2007, -0/+1I don't need a big cluster for Folding@Home (the only project I would actually use on a beowulf cluster). All I need is my old 466mhz Celeron box to run day and night, dedicated to only Folding@Home and the occasional backups as a NAS.
  • jaylaprade, on 10/12/2007, -0/+0I've thought about this in depth but can't seem to find any resources.

    I want to set up a Cluster and then run Qemu or VMWare on top of that. My goal is to have a number of machines dedicated to a cluster that would be able to run any environment for software testing. Basically so I can test any environment. Windows to Ubuntu to FreeBSD. I am aware that RedHat has clustering software, and I have seen Debian ports that are called: redhat-cluster-modules-2.6-qemu

    My thinking is that I should set up up this cluster, then install these modules. Then set up Qemu with the accelorator and see what happens.

    Does anyone know of anything like this? Or even someone who might have tried it?
  • BoontDustie, on 10/12/2007, -1/+1I work for a company that has worked with Beowulf's Clusters before and found them to be extremely over-complicated. Things may have gotten better over the years, but I still feel like there are major hurdles that need to be addressed before a widespread commercial application becomes available. Our company is working on a solution, and I wrote about our design choices a while ago on my company blog:

    http://brainmurmurs.com/blog/jiva/index.php/2006/06/27/jiva-and-the-trough-of-disillusionment/

    "Despite the substantial hype and promise surrounding grid computing, discussions are still scarce on the remaining barriers to mainstream adoption. We knew that buried beneath all the buzz, the core concept of volunteer computing really resonates with people and makes good business sense. So... what is preventing people from adopting the new technology?

    Current grid computing systems present too many risks to customers: existing systems are complicated and expensive, and most volunteer systems are too insecure for people to turn loose on their regular desktop computing infrastructure.

    We understand and sympathize with the notion that most existing grid systems present considerable security risks. These systems lack checks on code deployed to the grid, making entire networks susceptible to substantial damage from simple programmer errors. Companies are justifiably paranoid about potential damage from poorly implemented grid schemes.

    For mainstream businesses to embrace grid computing it needs to be dead simple to use, easily administered by the few people in the company that directly use it, and powerful."


  • chess007, on 10/12/2007, -1/+1 Could this (or any other type of cluster) be used with dvd ripping software to speed it up?

  • Fetching more discussions...
    Show 51 - 71 of 71 discussions

  • Add a Comment

    Login or register to comment!
    Or, sign-in super fast with your Facebook account ...


© Digg Inc. 2009 — Content created and posted by Digg users is dedicated to the public domain | Terms of Use Updated | Privacy Policy
DIGG, DIGG IT, DUGG, DIGG THIS, Digg graphics, logos, designs, page headers, button icons, scripts, and other service names are the trademarks of Digg Inc.