Digging His Way to Web Success - Systems Management News On The Web

Sort By :DateRelevance

DEPARTMENTS

HOME

TOP STORIES

ON THE WEB

BZ MEDIA

Printable version

Digging His Way to Web Success
By Alex Handy

July 7, 2008 — When TV talk show host and eventual Digg founder Kevin Rose asked systems admin and TechTV intern Ron Gorodetzky if he'd help to build a news site, the pair weren't expecting to be more popular than baseball. But shortly after Digg.com first launched in late 2004, the site quickly became a bona fide hit. Today, Digg is the 33rd most popular site in the U.S., ahead of CNET.com, the Washington Post and even MLB.com.

With over 26 million unique visitors a month, what started as a lark for Gorodetzky has matured into a full-time, enterprise-style systems management challenge. Gorodetzky, who turns 27 in August, said that Digg has evolved organically over the past four years, something the Digg team has worked both for and against over the years as the hacks turned into solid solutions.

While the current architecture does scale, Gorodetzky said that the goal is to move the Digg systems into a much more architected design.

“Ultimately what you want to do is go to a sharded architecture,” said Gorodetzky. “The first 1,000 users go on this set of servers, the next thousand on this set. You'll find that's an appropriate way to go, but right now we haven't done that yet. The way we've achieved some of the benefits of that is by doing the normal master/slave replications, and sending some types of traffic to only certain slaves and others to other slaves. We'll have queries related to comments going to some specific slaves, even though any slave could answer the query.”

Wage Slave
That slavery is also a heavy focus of optimization at Digg. The site is based on a LAMP stack, with MySQL and PHP doing most of the heavy lifting, said Gorodetzky. Since he's been there right from the start, Gorodetzky had to work through the typical growing pains, particularly those related to MySQL replication.

“The first pain point we hit was just database stuff. The first thing you'll notice is when you start to grow these queries, the database can't commit as much time to committing a certain query as it used to,” said Gorodetzky. “You'll find the normal things that work, suddenly don't. You'll find that, one day, you'll see a spike in your graphs telling you that something's going slower. Once you do that, you get to the point where the database part is as fast as it can be, you cache things. You scale out your Web server so you have more resources there, generally caching and doing less work per request.”

Another pain point along the road to success came when Digg decided to host images and video links accompanied by thumbnails of the linked visuals. Hosting all those thumbnails was actually a difficult problem to solve, said Gorodetzky.

“How do you deal with images? You can't just use NFS. We use MogileFS for that,” said Gorodetzky. The MogileFS was originally created to run the file systems behind LiveJournal, but the Digg team has found it to be capable of scaling horizontally, something they had trouble achieving with NFS.

Gorodetzky said that Digg is always looking into new technologies. One of those is currently Puppet, which the Digg team is working to implement behind the scenes for server allocation. Gorodetzky also said the team uses common tools like Nagios and a souped-up version of the Multi-Router Traffic Grapher.

Digg is not an ITIL shop. “We try to be lean. We want a lot of order and organization, but we want to copy people's ideas where it makes sense. Too much overhead is something I really don't like,” said Gorodetzky of ITIL. “The job's already hard enough, let's not add a lot of red tape.”

Cutting Teeth on Silicon
Gorodetzky had been consulting for small and medium businesses before he signed on at Digg. He cut his teeth on SGI machines while at an overseas high school program in Israel. When he returned to the States, Gorodetzky jumped feet first into Linux.

“It's fairly natural to go from Linux user to sysadmin. Of course, everything back then 10 years ago was more hands on,” said Gorodetzky. Simply knowing how to use Linux in those early days before the millennium, he added, was generally enough to qualify one for administration duties.

As for the success of Digg, Gorodetzky said it became clear fairly early on that the site was going to be big.

“We always hoped for some success, but this was a very side project. I was also involved with Kevin for Revision3 and Pownce [Rose's other Web startups], and we just happened to do a lot of work together. While you're in it, you're like, 'Well we're getting a little more traffic,' ” said Gorodetzky of realizing the site's success. “But it was a surprise. After six months in, it sort of became obvious that we got something there.”

Related Search Term(s): Database administration, server management, Digg

Share this link: http://www.sysmannews.com/link/32490

EMAIL THIS ARTICLE

SEND FEEDBACK