Sniffing Out Trouble

Permanent data probes and Web-based management software keep traffic flowing on Indiana's state government network.

by / September 3, 2003
Without appropriate tools, solving network problems is a matter of trial and error, sometimes taking days to pinpoint causes of poor performance. "You start troubleshooting from top to bottom," said Jeff Duke, senior network engineer for Indiana's Division of Information Technology (DoIT).

About four years ago, DoIT's network engineers added Sniffer Portable probes, from Network Associates Inc. (NAI), to their troubleshooting arsenal. When a problem arose, an engineer plugged a unit into the affected portion of the network to capture and analyze data packets as they passed through. But the devices offered only a snapshot. They could not define normal network conditions with which they could compare current behavior.

When DoIT started replacing its old fiber distributed data interface (FDDI) network, officials decided to install tools to watch over data traffic all the time and quickly pinpoint problems, instead of tapping in only when something went wrong.

"We tested some network management products that could suit us for troubleshooting the network and also monitoring and reporting on the network," Duke said. "After evaluating everybody through a rigorous process, we decided Network Associates had the best to offer."

Since last year, DoIT has installed about 25 of NAI's Sniffer Distributed data probes to continually monitor traffic on its new Cisco Enterprise campus-area network. DoIT also implemented two of NAI's centralized network protection tools ? Network Performance Orchestrator (nPO) Manager and nPO Visualizer. Duke said DoIT now solves network problems much faster. Often the technology helps DoIT find anomalies and fix them before they affect end-users.

DoIT's Gigabit Ethernet campus-area network serves approximately 20,000 state employees in Indianapolis and surrounding Marion County. Another 19,000 employees in about 200 state agencies in 800 offices throughout Indiana are connected with this infrastructure through a wide-area network (WAN). Duke and his staff administer the campus network. The Indiana Telecommunications Network (ITN), is responsible for the WAN.

Lug and Plug
In the past, with only the portable probes, troubleshooting was a time consuming process. "You'd lug a portable somewhere and plug it in, go back to your desk, wait a couple of days, go back, unplug it, and check out what you'd been capturing," Duke said.

With the new distributed data probes, it's easy to view data not only on current network behavior, but also on changes that occur over time. Instead of visiting the scene of a problem, an authorized technician can use a Web browser to view probe data from anywhere on the network.

"Instead of spending three business days, we're spending 30 minutes and figuring out the problem," Duke said.

NAI's Sniffer nPO technology allows DoIT to administer network probes from a central console and monitor activity on the entire network.

Sniffer nPO Manager allows managers to install updated software on all network probes at once, configure them remotely and set up different access levels for different users. Before DoIT implemented this tool, Duke configured and updated the probes manually. To avoid making the wrong changes on the wrong units, he said, he had to document every step in the process ? which was a lot of work.

Managers also use nPO Manager to establish alerts for various abnormal conditions, said Chris Thompson, vice president of product marketing at NAI. When a probe detects a security or performance problem, such as a series of invalid logon attempts or network congestion, an on-screen alarm appears.

Who's the Culprit?
Sniffer nPO Visualizer provides Web-based network analysis and trend reporting. "You'll see inconsistencies or discrepancies, and can drill down into that report and find out who the culprit is and what's going on before somebody actually calls about it," said Duke.

Using network probes and nPO Visualizer, network administrators can establish what normal network traffic looks like, Thompson said. "When there is a security outbreak, a misbehaving application or network segment, they see the change in behavior probably before a user will experience a service issue," he added.

"Without having eyes on that kind of information, you really don't know how your network is," Duke said. "It gives you the overall health of your network, all around traffic patterns, what's normal and what's not."

If the engineer in charge of a server complains it's not performing well, Duke said he logs into the probe nearest the problem server, checks out the network traffic, and examines several days or weeks of data for abnormalities. Within 5 minutes, Duke said, he can tell whether the network, the application, a desktop computer or some other factor caused the slowdown.

Cancel that Server
Mark Small, NAI's senior vice president of government, health and education, said quick diagnosis can eliminate unnecessary expenses, citing the recent experience of one government organization that implemented nPO Visualizer and quickly learned they could cancel an order for new servers.

"One application out of the 40 they were running was using 80 percent of the CPUs of a cluster system," he said, and the enterprise needed only to fix the application, not increase capacity.

In Indiana, faster diagnosis means less downtime for end-users. The campus-area network is completely redundant, so technical problems can't put it entirely out of business, Duke said, though that doesn't solve all problems.

"An agency might not have redundant connections to us," he said. "If they lose connectivity, their whole agency's affected."

Using network monitoring and management technology to solve network problems in an hour and a half rather than three days provides a benefit "that's definitely quantifiable to the productivity of the agency," he said.

The ability to monitor network use over time also helps DoIT plan for growth. "If [traffic] goes up 1 percent every month, I can say in 6 months it's going to be at 10 percent," said Duke. He can then build or upgrade accordingly.

The new tools provide upper management with historical reports that show as much or as little detail as each individual needs. They also e-mail a set of weekly reports to state agencies that use DoIT's network services. The reports show agencies how well DoIT lives up to its service commitments. Duke also allows agency network engineers to access probes connected to their infrastructure.

Probes at the Edge
The ITN and DoIT are trying to work more closely to share information, Duke said, because of the interface between the WAN and DoIT's network. ITN also installed Sniffer probes on the WAN, he said, which is important because data passes constantly between the two networks. DoIT would also like to place probes at the network's edge, where individual agencies' LANs connect to the wider infrastructure, but that will take time.

"We haven't developed a way to put 800 probes out at these different edges and report on those," he said.

DoIT hasn't calculated how much time and money it saved with the new technology, but the savings are clearly there.

"We spent probably half a million dollars on our network and about $300,000 initially on the network management solution," Duke said. "We have probably solved 30 problems with the Sniffers [this year] that could have taken 5 hours apiece, but maybe we only spent 30 minutes on each problem."

Now, network managers are evaluating another NAI product, InfiniStream Security Forensics, which captures and saves up to 2.3 TB of network traffic, allowing engineers to play back and analyze network events, such as activities of a user they suspect of introducing a virus. DoIT is also evaluating NAI's IntruShield intrusion detection solution, Duke said.

DoIT's network management solution has proven more useful than Duke and his team originally expected. "Unless you have protocol analysis and online reporting, you really don't know what you're missing," he said. "You could be managing all your network devices, but without actually looking at the traffic, you're not really managing your network."

Contributing Writer Merrill Douglas is a freelance writer based in upstate New York. She specializes in applications of information technology. E-mail
Merrill Douglas Contributing Writer
Platforms & Programs