System Utilities You Can't Live Without - Systems Management News On The Web

DEPARTMENTS

HOME

TOP STORIES

DATA CENTER NEWS

EDITORIAL CALENDAR 08

EDITORIAL CALENDAR 09

ON THE WEB

PRINT EDITION

BZ MEDIA

AS OF 11/22/2008 5:37AM EST

System Utilities You Can't Live Without
By Alex Handy

August 1, 2008 — A poor craftsman blames his tools. But then, a craftsman is only as good as his tools. And as Mr. Natural always said: Use the right tool for the job!

Managers have their hands full monitoring, analyzing and testing their systems. And managers have come to rely heavily on tools to help perform those tasks, whether they are purchased, taken from open-source communities or customized for their own purposes.

And there are certain tools that systems managers have come to realize they can’t work without. Here are what those administrators listed as their most valuable utilities.

Jesse Nelson, Unix systems architect at Military.com, first thought of mon, the open-source service monitoring daemon. While Nelson has used many monitoring tools over the years, it’s mon that tends to be his first resort. Military.com is a social network and job site specializing in current and former members of the military. The site runs on around 200 servers, 150 of which are based on Puppet, the open-source configuration management framework. Military.com runs on Java, LAMP and is currently rolling out Ruby applications internally.

“It’s still good code,” said Nelson of mon, “versus Nagios, which is a big, fat framework. We use mon and Nagios. We use mon to do lightweight stuff.” Nelson also festoons mon with another, older tool, Cricket, the open-source graphing tool.

“I’ve used Cacti, but it was more than I needed at the time. I just needed any kind of histogram. I’m checking out Graphite. There’s better stuff out there, but Cricket is pretty simple. I’m also an old fart, so these are the things I used eight years ago that still work well,” said Nelson.

Nelson’s list also included Apache’s mod_rewrite and SSH.

Spiceworks, however, is a newer utility that monitors and tracks networks' assets and statuses. It’s the favorite of Robert Baxter, CEO of Baxter IT, a consulting firm.

Baxter’s been running systems for 12 years, and he said that the best thing about Spiceworks is the price. “Most inventory systems are garbage,” said Baxter, but Spiceworks can find and catalog systems when given an IP range or the SNMP info.

Spiceworks breaks the mold by being ad-supported. It’s also something of a multitasking platform, and it can handle trouble ticket tracking and help desk needs. Baxter’s list wasn’t all newcomers, though. It included some old chestnuts, like network mapper Nmap and Windows terminal PuTTY. Cacti also appeared in his list, as did Nagios.

Dumping Love on tcpdump
Travis Morgan, owner of Canadian hosting firm BigFiber.net, filled his top five with old favorites as well, including the Bash shell, Nmap and Perl. He also mentioned tcpdump, the command-line packet capturing utility. BigFiber is a co-location and Web hosting company, so Morgan has many reasons to keep a watchful eye on the packets he’s carrying.

“It’s easy, installed on almost everything and very powerful,” said Morgan of tcpdump. “It’s great for checking if traffic is leaving a host for another host, if it’s reaching a host, monitoring malicious traffic, and troubleshooting firewalls and NAT.” As an old tcpdump pro, Morgan doesn’t mix it with grep or other search tools. “It has most of the filtering you’d need on its command line. You can specify host, port, protocol, what type of output when you run it.”

William Tiemann, systems engineer at Akamai, also likes tcpdump. His list also included Altova’s XMLSpy and perennial favorites cURL and Nmap. There was also one we’d never heard of before: Tamper. “Tamper is a plug-in for Firefox that lets you modify requests as they are sent. It’s more for people supporting Web apps, but it is quite useful,” said Tiemann.

Miah Johnson, senior security analyst at an investment group in Houston, loves Wireshark, which can be used to view tcpdump output. “I couldn’t do my job without that. I do a lot of instant recon stuff, and I have servers all over the world. I run can tcpdump on them. I pull down lots of pcap files and look at them with Wireshark,” said Johnson. Johnson has to maintain and monitor servers around the world in major financial centers, and when something goes wrong, Wireshark can be used to help build the forensic picture. Wireshark can reassemble packet streams and make them searchable in a GUI.

Justin Fitzhugh, director of IT at the Mozilla Foundation, is another lover of tcpdump. His list of five included Nagios, a tool for which we found both love and scorn amongst those polled.

Fitzhugh directed hosting of the record-breaking download festival that was the Firefox 3 launch, and Nagios was at the top of his list.

“Nagios is probably one of our most important,” said Fitzhugh. “We also use Cacti, a trending tool for system load and infrastructure. We’ve customized both tools quite a bit to fit our environment. Nagios is a simple Web page, it’s not that thick. It’s a little hard in terms of your initial ramp up for it. It is extremely configurable and it’s open source. We’re in an environment where we have a lot of weird requests.”

Johnson also relies on Nagios. “I always set that up if it’s not already running. I can script custom checks, and all sorts of alerts. It’s very customizable,” said Johnson.

The Sunny Side
Ben Rockwood, director of systems at cloud hosting company Joyent, runs thousands of Solaris Linux and Windows machines, and he specializes in Ruby application hosting. As a Solaris user, Rockwood’s top five included DTrace, the application that pulled the clothes off the CPU stack and which was developed at Sun Microsystems for Solaris.

Rockwood's top five also included some interesting departures from the mainstream. With utilities like ClusterSSH, FileBench, Splunk and Zabbix, Rockwood appreciates a hands-on approach to administration.

ClusterSSH made the list because, “It allows you to open multiple SSH connections and input text to all those connections at the same time. That’s especially useful in a cloud environment such as mine, where I need to work on 16 machines simultaneously, and they’re all very similar. You could, say, use a vi session on 16 systems simultaneously. At that point the number of connections is determined by the size of your LCD.”

Rockwood also had good things to say about search tool Splunk. “The cool thing about Splunk is that it acts just like a search engine. It takes five minutes to install. Getting all of your logs redirected can be easy or hard depending on your infrastructure. From there it’s like knowing any search engine really well. That’s one of the nice things about a good utility like Splunk. It’s easy to get started and you go from there. It’s like an Atari game: easy to play, but impossible to master.”

Miah Johnson also loves Splunk. He likes it because of “the fact that I can throw pretty much any data source at it and it can figure out how to handle it. If it can’t, I can make it.”

Despite all the cloud talk, Rockwood said he’s a storage guy at heart. As a result, he loves FileBench, “a disk benchmarking utility. More than being a benchmarking tool, it’s a workload generation tool,” said Rockwood.

Normal benchmarking tools tend to use static data, which don’t resemble the real world, said Rockwood. “The problem is in the real world things don’t work that way. FileBench can go out and pregenerate files and generate directories using certain constraints and randomizing. Then you have what looks like a real world file system, and you can create scripted workloads. Like, if I want to create 1,000 files and then delete 500 of them. You can create very useful, highly tuned benchmark data. It’s an amazingly awesome tool,” said Rockwood.

Rockwood’s final choice is a competitor with Nagios. “Zabbix is an open-source, agent-based monitoring solution. Most monitoring solutions use the traditional model of SNMP polling. If the system already presents something via SNMP, you can monitor it. The only way to get around that is for your monitoring system to SSH out to a box, run some command and bring it back, which is really horrible because you have thousands of connections originating from your server every five minutes. Zabbix has an agent on each server. It’s sort of a power user tool. Not something your manager’s going to love. The Web interface is overly geeky, but it’s more powerful and raw than anything else out there,” said Rockwood.

Window to Utility
Miah Johnson’s list included two of the only Windows tools mentioned in our polling. Process Explorer and PuTTY rounded out his list, which also included Splunk. Said Johnson, “PuTTY is the best SSH, telnet or serial console client you can get for Windows. It’s small, doesn’t require an install and you can put it on a USB stick.”

Process Explorer, said Johnson, allows him to browse tasks running on Windows servers quickly and efficiently. With it, he can find out who started processes, how long they’ve been running and what’s wrong with them.

For Sam Quigley, security architect at financial management start-up Wesabe, Windows doesn’t enter into it. In fact, utilities soon won’t enter into it at all for Quigley: the company is moving to an entirely Ruby-based infrastructure, which will also include custom tools and utilities.

“To tell the truth, though, we’ve been moving away from individual utilities at Wesabe and more toward libraries that can be called from within Ruby. I’d just as soon not ever have to log into a box manually if I can help it ... I don’t think ‘Ruby’ really counts as a sysadmin tool, though,” said Quigley.

Until Wesabe successfully moves, however, Quigley still uses normal utilities. He professed love for Splunk and Wireshark, but also included two we’d not seen before. collectd is a system statistics collection daemon that packages up said information however the administrator wants. Typically, this is an RRD file, which can be sent along to any standardized monitoring and logging aggregator.

Quigley also mentioned MySQL Master-Master, a database management tool that handles replication from masters to slave nodes. MMM, as it’s called, can monitor MySQL servers and detect errors and problems. When they’re found, failover can be triggered. Additionally, MMM can handle the replication across nodes, and keep stuck servers from being constantly poked. Certainly a precious tool for anyone who’s basing his or her systems on MySQL.

Perhaps the most commonly mentioned, yet least elaborated upon utility, was Nmap. This open-source network mapping tool is a mainstay of the hardcore administrator, as is evidenced by no one mentioning any graphic layovers for the tool in our survey.

Originally written by Gordon Lyon, Nmap has been a mainstay among both hackers and legitimate users since it was created in 1997. While none of our respondents took the time to extol Nmap’s virtues, many considered it indispensable. That’s not surprising, since the tool can be used to find and identify servers across all types of networks.

Nmap can also be used to find services being offered on machines, even when those services are not advertising themselves publicly on the Web. As a result, Nmap can work double duty as both an inventory tool and as a reconnaissance tool. Certainly figuring out what’s on a network is a big challenge and headache for many systems administrators, but Nmap can also find illicit services, such as backdoor IRC drop outs that send keystroke logs to hackers. While Nmap isn’t a standard Unix tool, it is one of those essential tools that administrators tend to install first thing on a new system.

As we already mentioned, these are by no means the end all and be all of systems administration. Not one administrator in our survey didn’t use custom tools or massively customized versions of open-source projects somewhere in their organization. As a result, many of the most used utilities in the wild are one-offs that aren’t to be seen in any other place.

As time passes, however, this may change. Every once in a while, a frustrated sysadmin jumps the fence and decides to write a universal tool, and if they remain committed, we can end up with another Nagios, another Nmap or another tcpdump.

Related Search Term(s): networking, systems management, Unix, Windows

Share this link: http://www.sysmannews.com/link/32611

EMAIL THIS ARTICLE

SEND FEEDBACK

MORE SPECIAL REPORTS