Linux Key to 24/7 Gaming Systems - Systems Management News On The Web

Sort By :DateRelevance

DEPARTMENTS

HOME

TOP STORIES

ON THE WEB

BZ MEDIA

Printable version

Linux Key to 24/7 Gaming Systems
By Alex Handy

May 15, 2008 — Right now, as you read this, there are at least 100,000 people in Sony’s fantasy worlds. They’re pretending to be elves, or trolls, or ninjas, or pirates, or cheese makers, or any of a thousand other fantastic things; and they’re all paying a monthly fee for the privilege. So when massively multiplayer fantasy role-playing games go online, they need proper change management, good network architects and customized remotely deployable Linux installs to do it.

That’s where Mark Rizzo, vice president of technology operations at Sony Online Entertainment, comes in. Rizzo has been running the back ends of massively multiplayer games since one of the first such games, Ultima Online, which he worked on at Electronic Arts in the late 1990s. As the first major massively multiplayer online role playing game, Rizzo said that the Ultima Online team was learning and building everything from scratch, resulting in numerous lessons learned by both the creators and players of the game.

A lot has changed in systems management since then. For Rizzo, this parallels the advances in Linux, which has matured to address many of the problems he said faced the Ultima Online team last millennium.

While today most of the problems faced by Rizzo’s team are technical or development related, back in the Ultima Online days, these were compounded by the unpredictable player base. In its day, no one had ever seen the psychological and sociological reactions of players in a massive online world before. Much of the control over societal movements in the game world was given to the players, something very few games have tried to do since. As a result, the player base became phenomenally violent to newcomers, and even the game’s patron saint and lead creator, Richard Garriott, found his in-game avatar constantly assassinated early on in the game’s life. Since Ultima Online, however, a lot has changed.

One thing that hasn’t changed, however, is the religious nature of Linux administrators. While customized network installable disk images have made deployment easier, the administrators behind those systems can still get caught up in holy wars around platform choices, said Rizzo. But he has a solution: “An architect who is over all those larger tech decisions. It helps resolve the issues where you may have some individuals who are talented, but maybe not as mature.”

Big Games, Big Systems
Rizzo heads up a team of around 50 systems administrators and troubleshooters. “Our group is responsible for all the live operations of the games, data centers, Internet access, security, and the network operations center,” said Rizzo. “I head that whole group and work to build strategy around quality of service as well as uptime. We also work a lot with the games when it comes to deployment. We have in the range of 6,000 to 7,000 servers.”

With new games coming online, and seven to eight games already hosting thousands of simultaneous users, change management has become a key issue for Rizzo’s team.

“There are so many moving parts. We have thousands of servers in pretty complex networks with a lot of requirements. [There are] lots of port ranges and constant changes of network access controls, as well as constant code updates. We did an April Fools’ prank in one of the games and we had to have some server code updated to pull the prank off. Change management allowed us to know who approved it, and know where it’s from, and where it’s going,” said Rizzo.

To manage that change, Rizzo said that Sony uses BMC Software’s Remedy trouble ticketing system and a mix of its own tools. “We continue to fashion around our processes some of the ITIL principles,” said Rizzo, though he added that his team picks and chooses the pieces of ITIL it finds useful and ignores other parts.

“To make it work for us, the big key component comes down to having good tools to remotely operate and install images,” said Rizzo.

Network Architects
All of this visibility comes thanks to Rizzo’s network architects. These are the folks Rizzo blesses with management duties and the ability to make hardware decisions.

“A good architect is a highly technical person who also understands the team dynamics. He knows who’s good at what, knows who’s not good at what. He has no direct reports, but has senior level responsibility. They also function as technical producers when it comes to products. It helps to be able to say things in a non-confrontational way. They have to be technically competent but fairly even keeled,” said Rizzo.

And most importantly, he said, architects aren’t bought, they’re made. “None of our architects are people we’ve hired in that way; they’ve always grown into that role,” said Rizzo.

Just Like Us
For a market that’s grown from small, single games in the late 1990s to a multitude of variants and sub-genres, online persistent game worlds are now a world unto themselves. Rizzo said that the systems management challenges are varied but not entirely unlike those found elsewhere.

“It’s a difficult analogy to draw to a typical enterprise application, where you have a stack of Web, database and interface. I would say it’s closer to something like a stock transaction system, something where you're having to process a lot of information,” said Rizzo. “It’s a lot more like you’re running these simulations in a sense; more like what research people do. The client is sort of a terminal into that world.”

Thus, uptime is the most important part of the online gaming plan. With players charged between US$10 and $15 a month for the privilege of access, time offline means thousands of angry customers. Fortunately, staying online is not heavily bandwidth intensive. On patch day, however, Rizzo’s servers can stream out huge amounts of information in multi-gigabyte software updates that need to be sent to all players.

With all those games running all the time, Rizzo felt that remotely installable Linux images were a hugely important asset for his team. When a server in the UK goes down, his team can reinstall a fresh disk image from their headquarters in Southern California.

“There have been systems built and put in place to help manage and image all the thousands of servers in a way that allows you to stand up equipment a lot faster than we used to be. Particularly, in Linux, it was very difficult to run and install on every machine. It took a long time to configure environments. There has been a lot of advancement in rapid deployment,” said Rizzo.

Related Search Term(s): Change management, Linux, networks

Share this link: http://www.sysmannews.com/link/32072

EMAIL THIS ARTICLE

SEND FEEDBACK