What is Digg?Sponsored by Dragon Age: Origins
Can't get enough Dragon Age: Origins? Check out new footage. view!
DragonAge.BioWare.com - EA presents BioWare's new dark fantasy epic Dragon Age: Origins. '9/10' from Game Informer.
113 Comments
- ieure, on 01/28/2009, -10/+73Ben,
We’re flattered that you think the features are so simple. There’s really a lot going on behind the scenes to make Digg work as well as it does. While it may only take a few minutes to whack out a solution that works on a small dataset, the problem becomes much harder when you scale it to Digg’s size. Just looking at the Top in All Topics list right now, there are over 10k diggs on just those ten stories. Now multiply that by orders of magnitude, and you have an idea of the amount of data we have to process. Digesting that data and coming up with something useful while performing well enough to serve a million unique visitors every day is not easy.
But if you’d like to show us how to do it better, please drop us a line at jobs@digg.com, or visit the jobs page: http://digg.com/jobs
Thanks,
Ian - visionaryIX, on 01/28/2009, -0/+35How about fixing the fundamental problems?
Make it so your average user actually has some chance of making the front-page?
A URL filter so we can opt not to see crap like the huffington post or xkcd? Hell, even BenRT could code this. - winry, on 01/28/2009, -4/+35it's easy to copy/paste it, but that's a very HEAVY SQL query for a site like digg. optimize that query takes some time
- Biscuitz, on 01/28/2009, -2/+24They weren't offering a job, they were welcoming you to apply. Lol
- naner, on 01/28/2009, -1/+20I think that "Even BenRT could code this" might become a new meme.
- serif69, on 01/28/2009, -1/+19Welcome to the Special Olympics.
- trentrezn0r, on 01/28/2009, -4/+21Oh you could just fix the broken algorithm so that the real most interesting/popular stories make the front page instead of ones submitted by power users. You know...like a Digg/bury function that actually works. Hmm, what a wild idea...
- nunofgs, on 01/28/2009, -0/+16I've said it before, and I'll say it again:
http://img89.imageshack.us/img89/2913/newkh8.png - BenRT, on 01/28/2009, -5/+21Thanks, although I wasn't mistaking my terminology, we do get just under 1m unique users each month. My initial post was saying that we meet what you make in a day each month. So we're at about a 1:30 ratio.
And anyway, this is all beside the point. The hits our company site gets isn't helping our discussion. I've been a user of Digg a long time, the lack of front-facing progression over a long period of time is simply curious to me. I know about your rewriting of the backend and all that, I do understand the "invisible" changes have been your focus, and rightly so. Even still, there should be at least some public feature enhancements. Over the last year the only changes I can name have been Recommendations and a reshuffling of the story page formatting.
I would like to see aggressive innovation on Digg, the podcasts section was new and interesting but that got nixed after a while. It seems though that there's a hesitance to 'change a winning formula'. If I may point out, a lack of profit does not make the site a winning formula. It means it's slowly dying, and the VC is keeping it on life support, which won't be around forever. New and engaging ways to monetise Digg (as Jay Adelson announced recently on the blog) is the way to go, and I am excited for the fruition of that initiative. - BorsKaegel, on 01/28/2009, -6/+22I thought I saw a PWDTAD!
- phatduckk, on 01/28/2009, -0/+16Ben's my hero...
While you're at it could you please come up with a business model for twitter, fix yahoo's financial problems, map out a strategy to increase PS3 adoption, save the zune, purge the world of drm, get the movie studios to allow digital rentals for new releases & shut down classmates.com?
Oh... and save the whales.
thanks
xoxo
Arin - covertbadger, on 01/28/2009, -1/+16@winry
If you're doing it in sql, you're doing it wrong. This is a simple offline algorithm that can be run against denormalised data pulled from a slave DB and periodically injected into memcached. The core algorithm as presented in chapter 21 of PCI is about 50 lines of python. Doing it all in sql would be catastrophically stupid. - trammell, on 01/28/2009, -1/+15Hiya, Burento. When rolling out features, we often only show the feature to a small group of users and then slowly turn up the throttle. I didn't want to post to the blog until we had 100% of the Digg community able to see PWDTAD.
But, point taken. :-)
Slainte,
Trammell - MarcanX, on 01/28/2009, -3/+17Oh yeah ! So now I can see all the rest of the Poweruser dugg stories on one neat convenient page !
- popsickle, on 01/28/2009, -3/+17Really? You run a site that gets 30 M+ uniques a month, which is what digg.com gets according to compete.com? What site is this, pray tell?
And have you implemented Facebook Connect or OpenID on this alleged high-volume site you run?
We are waiting to hear... - ieure, on 01/28/2009, -3/+16Who do you work for that gets 30m uniques a month? I’m just curious, since the only affiliation I could find was Rooster Teeth:
http://siteanalytics.compete.com/digg.com+roostert ...
Which isn’t quite there yet. - ieure, on 01/28/2009, -3/+16Perhaps there was a terminology misunderstanding. Digg handles one million unique visitors every day. Those individuals load many pages, which request hundreds of objects. I don’t even know how many hits Digg handles in a day; tens of millions would be a very conservative estimate.
It seems like you were confusing hits with uniques. They’re very different metrics. - headzoo, on 01/28/2009, -0/+12Ah.. I fondly remember being 16, and thinking I knew it all.
- BenRT, on 01/28/2009, -25/+37I'm a PHP developer and I have to say, features like "People Who Dugg This Also Dugg" are incredibly, incredibly simplistic. I mean you could code this thing in five minutes.
Why does it take Digg loads of "engineers" and months to roll out really easy, really small changes? Why does everything on Digg happen at such a slow crawl?
I'm not trying to troll or be mean. I just want an answer. Facebook Connect integration was announced to be released for Connect's LAUNCH. Then nothing. - sickthoughts, on 01/28/2009, -4/+16@krustie: grammar shouldn't be hard for an adult
- headzoo, on 01/28/2009, -0/+12Things are not that simple for a website with many, many servers, but database servers, plus a million+ members. These kinds of features:
A) Have to be integrated into the existing framework
B) It has to be tested
C) The existing framework may need to be changed after realizing new additions can't be added easily
D) It has to be tested
E) It has to be build to scale to 10, 20, 30 million members
D) It has to be tested
E) It has to perform incredibly fast
F) It has to be tested
G) Etc.
H) It has to be tested
Digg isn't the kind of company that thinks of an idea on Monday, and implements it by Wednesday. Any changes, even to fix existing (And well known) bugs have to go through rounds of testing, before the changes can go live.
"Why does everything on Digg happen at such a slow crawl?"
It's not just Digg, it's any large website, or large corporate website. In fact, you should be asking yourself, "Why am I rolling out changes so fast when no one else is?" - Joll, on 01/28/2009, -3/+14Nice work turning some harsh criticism into a pissing contest everyone.
- BenRT, on 01/28/2009, -4/+14$filter = $db->select('* FROM `userDomainFilters` WHERE `uid` = \'' . $me->info['id'] . '\'', 2, 'domain'); // Indexes in the array are set to the `domain` value from the db
// Include additional memcache necessities on the filter check
$story['domain'] = tools::parseDomain($story['url']);
if ($filter[$story['domain']]) {
continue;
// Or do whatever else you want to do like reduce the story to a low opacity, in case of pagination quirks
}
This obviously isn't based on Digg's syntax since I don't know it, but it's an example of how you could achieve it, albeit in a rough form. Throw in some memcache in the domain filter settings and there you go. - chrysrobyn, on 01/28/2009, -1/+11I'd really like to be able to exclude sites from my page views. I won't name them because I expect my list disagrees with other people's, but I think there are a lot of people out here who would really appreciate a blacklist. There is a small handful of sites which constantly disappoint me but keep making the front page.
I won't pretend to be very good at databases, let alone high profile interactive large scale ones, but I hope something like this is possible. - robotfriendly, on 01/28/2009, -2/+12He's also only 16 according to his profile. Not to say there aren't some smart, 16 y/o PHP devs working for large companies, but that doesn't exactly scream experience.
- BenRT, on 01/28/2009, -10/+20Thanks for the reply, Ian. We get a similar number in uniques per month, so scalability is something that is always on the mind. Working with the actual digging data obviously is a massive undertaking, I do appreciate the point at which that much data becomes unwieldy to work with, but I don't see how Facebook Connect and OpenID can take months to implement. They're not performance intensive at all. It just seems curious that something like that would be promised and then never spoken of again.
Also, while it's nice that Digg is open to hiring out of the community, I don't think I would want a job at Digg. Hiring staff just based on VC and then cutting 10% of the workforce later on is too reckless and unprofessional for my tastes. - MrJagil, on 01/28/2009, -2/+12Mhmhm... it's cool and all, but there are changes that are much more sought after:
- NSFW tag
- Removal of friends/shout system (it's bull)
- Better comments system (italic? bold? quotes?)
- Though what you're doing is the right path, i'd like for the older articles in Digg's archive to be a more prominent part of Digg. No tangible suggestion here.
- FIX THE JUMPY COMMENT BOX! ATM I AM GETTING UTTERLY BRAIN *****!
thank you - DeathRay2K, on 01/28/2009, -1/+10I love how they both say "curious" whilst delivering an underhanded insult. :P
- Biscuitz, on 01/28/2009, -0/+9Is there a fix for the jumping of the comment box, perhaps? Lol, it's a bit aggravating.
- ileftfark, on 01/28/2009, -1/+9Please let us turn it off if we don't want it.
I personally don't need recommendations or a view of what others are digging. I'm sure there are some people who do. I do not.
EDIT: Or at least move it somewhere else. When we click to view comments, we generally want to see the comments on the article in question, not completely different stories that may or may not be related in any way to what we were originally looking for. - LargeStack, on 01/28/2009, -3/+11People Who Dugg This Also Dugg
Orphaned hedgehogs adopt cleaning brush as their mother
I think it needs work. - rebrad, on 01/28/2009, -0/+7Could they put an opt out on it. To me it just gets in the way. At least put it on the side so I can ignore it with the rest of the side content.
- misterparry, on 01/28/2009, -0/+7that would make too much sense
- DoktorRob, on 01/28/2009, -1/+8Heh. This is the Digg equivalent of a stand-up comic replying to a heckler with, "Oh, yea. Well let's see you get up here on stage and do better!"
XD - PatrickX, on 01/29/2009, -0/+6Please work on reducing the digg weights of power users...
- headzoo, on 01/28/2009, -0/+6Dugg for jumpy comment box, which has been a bug for months now.
- demitio, on 01/28/2009, -1/+6I don't like it. The results are too random and don't really relate to the storry you're looking at. They are often just recent popular frontpage stories. It could be because I'm only looking at the frontpage stories to begin with, but I still found Related by Keyword a lot more useful.
- Veni_Vidi_Vici, on 01/28/2009, -0/+5I love digg too, but I really hope you were making a general statement and didn't say that in all caps just because of some lame new feature.
- Burento, on 01/28/2009, -1/+6Hah.. I was just ***** around dude.. no worries :)
- ieure, on 01/28/2009, -1/+6Hey, arcane81. I truly wasn’t trying to belittle BenRT, and I apologize to everyone if I gave that impression. I wanted to know where someone so young got experience working on a high-traffic site; that’s all.
- Ian - inactive, on 01/28/2009, -1/+5THAT'S OVER 9000!
- misterparry, on 01/28/2009, -0/+4don't say that....it might hurt their feelings
- headzoo, on 01/28/2009, -0/+4How? Some of the people that dugg this story also dugg that story. That's how it's supposed to work. It's not a recommendation engine.
- inajeep, on 01/29/2009, -0/+4Hell yeah. Keep saying it because I didn't see it before.
- Thomaschaaf, on 01/28/2009, -2/+6LOL BenRT is sixteen.. I would like a site with 30m views per month a year ago..
- Smedley42, on 01/29/2009, -0/+4It's an annoying intrusion between the articles and the comments.
I frankly don't care what others read... When I follow up I find most links tenuous at best and generally uninteresting.
Move it somewhere else, like at the end, if you must have it. - robbob, on 01/28/2009, -0/+4Like Tweety Bird, with a lisp, says Pu-D-tat
- slightlyoffbeat, on 01/28/2009, -2/+6@ sickthoughts oh....snap :)
- fotos, on 01/29/2009, -1/+5@covertbadger
Can you give a reference for this algorithm? Searching for PCI (+relevant keywords) doesn't yield any meaningful results...
Thanks in advance! - covertbadger, on 01/28/2009, -1/+4"The number of pageviews is virtually all that matters here."
No, the number of pageviews is not relevant, since a simple pageview does not require recommendations to be regenerated (or, assuming the recommendation dataset is cached, a pageview would not invalidate that cache).
The dataset only becomes stale when a story is dugg. Diggs are the trigger, not pageviews (unique visitor or otherwise), and there are nowhere near as many diggs per day as there are pageviews. We're talking orders of magnitude fewer than 30M.
Also, given that no-one would notice if the dataset was a few seconds out-of-date, it doesn't even need to be regenerated on every digg. An offline process running every couple of minutes would be more than enough. -
Show 51 - 100 of 113 discussions
© Digg Inc. 2009
— Content created and posted by Digg users is 