Network Security: San Francisco Incident Raises Questions for CIOs

Rogue administrator hijacks city network; experts tell how to maintain control of network resources.

by / December 3, 2008

Nothing gets people talking like a good story and in July 2008, headlines and blogs were abuzz with the news that Terry Childs, a network engineer in San Francisco's city-county government, was arrested for seizing control of the fiber network, locking out co-workers and denying officials the passwords to get back in. Mayor Gavin Newsom had to visit the jailhouse to obtain the passwords from Childs in a secret meeting that even the district attorney knew nothing about.

The press reported the story with enough drama to pique readers' interest. In a July 14 article, the San Francisco Chronicle referred to Childs as a "disgruntled city computer engineer" who received a bad performance review. The Washington Post published on Aug. 11 said he "hijacked the system" under the user name "Maggot617" and that police found diagrams of San Francisco's network, $10,000 cash and bullets at Childs' home.

No conviction has been made, but Childs' motives are on many minds. Most can only speculate at this point, but IT managers might do well to ponder why an employee would do this. After all, it happened in one government agency, what's to stop it from happening elsewhere?

Some speculated that Childs became too possessive of his work, though that's not easy to prove or verify. In a July 18 InfoWorld article, Senior Contributing Writer Paul Venezia wrote that "Childs apparently trusted no one but himself with the details of the network" and that administrators who build and maintain large networks often "care for them like children." But if Childs felt that way, that reason would ultimately have been no excuse, said Ron Vinson, chief administrative officer and deputy director of San Francisco's Department of Technology.

"[That] network does not belong to Terry Childs. It belongs to the city and county of San Francisco, and that's the major point there," he said.

How It All Began
In July, Childs was charged with felony counts of computer tampering. The network he hijacked was San Francisco's FiberWAN (wide area network) that handles payroll, e-mail, and law enforcement and jail documentation. Childs was one of five people who worked on the network.

"On July 9, in a process to complete our change control and change management system, I requested of Mr. Childs the user [identifications] and passwords for several devices on the FiberWAN network," said Chief Operations Officer Richard Robinson. Childs refused to comply. The devices in question were different types of network routers and switches.

According to Vinson, the department had already been implementing network security protocols, and when other network employees observed actions that weren't best practices, it led to Childs' questioning. Vinson said San Francisco paid at least $182,000 to Cisco and other contractors to help remedy the problem.

Childs made unauthorized and undocumented changes to the network, Robinson said. "The rest of the network engineering staff would not have the ability to continue to do any change control, any change management or any continued rollout of the FiberWAN," he said. "That being the case, it became a criminal issue because he was denying us administrative access to equipment that the city owned."

Robinson contacted the police department, and the officers also asked Childs to provide the passwords, but he still refused. According to the Chronicle, he gave authorities bogus passwords at one point. He was arrested three days later on July 12, Robinson recalled. The Department of Technology started working with Cisco, its network vendor, to handle the network as well as possible in the meantime. Once Childs gave the mayor the codes on July 21, network engineers regained control. But even now, Robinson and his team are still trying to determine the extent of the changes Childs made.

"We have found that Mr. Childs had put other devices, unknown to management or other network engineers,

within the network and that this is undocumented and unauthorized equipment," Robinson said.

In its August story, The Post reported that Childs set up unauthorized network doorways that allowed him "unfettered and undetectable access." He obtained pages of user names and passwords and downloaded gigabytes of city data to a personal storage device. At the time, Childs wasn't the only engineer with control over the network, but his control increased as his co-workers moved to other projects.

Childs left many devices inaccessible; he had gone through and encrypted many of them, Robinson said. These devices don't hinder the department from administering the network, but they do make it harder for personnel to identify and correct problems Childs may have left behind.

Robinson said the Department of Technology was in the process of furthering its security protocols and mechanisms, and Childs made the changes difficult by initially refusing to offer the passwords. These security steps included revisiting the city's intrusion detection and intrusion prevention strategies, architecture and approach. The city was also working to enhance asset management, identity access and password management tools.

Internal Threats
Unfortunately insider sabotage comes with the territory. In 2007, the Journal of Computer-Mediated Communication published a graph of public- and private-sector security breaches from 1980 to 2006. In 2006 alone, of about 250 reported incidents, nearly 200 came from threats within the organizations that reported the breaches. This graph was reprinted in a 2007 report, Insider Security Threats: State CIOs Take Action Now!, published by the National Association of State Chief Information Officers (NASCIO).

The San Francisco story can also be seen as a warning, and CIOs have definitely taken notice.

Bill Schrier, chief technology officer of Seattle's Department of Information Technology, wrote about the incident on "The Chief Seattle Geek Blog." He advises that CIOs should be careful not to give one employee too much power over the network.

"That sort of responsibility has to be shared, and to actually share it, there ought to be multiple administrators who can do the same sort of work and have access to the passwords -- with management oversight to make sure that the job responsibilities are divided and supervised," he said.

Schrier isn't the only one with that opinion. "You must have a balance of authority across your security layer, either with the network or applications, so you don't have one person that is godlike and controls all your resources," said Rico Singleton, a New York state deputy CIO. "You typically have a decentralized or federated security model of which you have redundant levels of super-user administrator type of authority."

Redundant levels of access means there will be more than one way, or more than one high-level administrator, with top-level authority, to access the system. However, the ability to distribute network responsibilities between different people may not always be easy to attain in state and local government.

"Government budgets and government personnel policies make it much harder to sort of do things in an efficient way," said John Pescatore, a Gartner analyst who specializes in security and privacy. "We also see government staffs are often much more multipurpose." In other words, while a private company may have a large department of administrators, a much smaller government agency may only be able to afford one administrator who also performs other functions that are unrelated to IT administration.

"The more common problem people worry about, and the more common way things have gone wrong, is when an authorized administrator -- we typically call them a super-user -- oversteps their authority," Pescatore said. They might do things that they really shouldn't do just because they have administrative access, whether it's IRS taxpayer database surfing or passport application surfing, he said.

Pescatore also recommends that departments stay on top of

who needs access for what task, and remove the appropriate access when tasks are no longer necessary. This role-based access protocol restricts system access to authorized users only.

"That seems simple, but so many times security audits are done and you find 30 percent of the admin accounts are still active, even though that person doesn't work there anymore and hasn't for months. As soon as those privileges are not needed to do their job, those privileges or authorizations should be removed," he said.

Pescatore and Schrier recommend configuring security management software tools so the authorizations of two administrators are required before significant changes can be made. Implementation of this type of software in a network environment forces departments to avoid the risky practice of giving only one employee the keys to the digital kingdom. Pescatore said Tivoli and Computer Associates provide tools that create automated logs for documenting network changes and blocked accesses.

Pescatore said the software would log if an employee were trying to surf an unauthorized database, making administrators aware of suspicious activity. Although it can be expensive to install such software agents on every necessary server, that price might be dwarfed by the cost and humiliation of having to notify 27 million citizens that their information was compromised, he said.

Keeping a Watchful Eye
The NASCIO report identified malicious employees as the No. 1 insider threat to a department's data, and it offers ways to deal with them. According to the report, IT employees may be too proficient for roles-based access and security awareness protocols. Their activity should be monitored and audited for abnormalities and dealt with quickly through severe consequences, including criminal charges if necessary.

NASCIO Executive Director Doug Robinson recommends departments pay attention to employees who are under stress.

"Perhaps they are going through a divorce, a foreclosure or financial instability," he said. "They are disgruntled because of a performance appraisal compensation or a raise that they didn't believe was adequate. They were demoted; they were fired perhaps."

Schrier said he also believes that managers should keep an eye out for employee stressors. "Certainly a warning sign for management is if an individual is taking a lot of overtime or is using a lot of overtime. It means that particular individual is probably being overworked," he said. "It requires some of the responsibilities to be spread around."

According to Schrier, if managers watch overtime and disperse responsibilities, they might also reduce the chances of IT workers getting too attached to their code. People who get too wound up in their work can become overly possessive, as if they're working on personal property instead of government-owned resources and projects.

IT shops should know as much as possible about prospective employees before hiring them by expanding background check procedures. The Post's August article reported that Childs "carried a list of convictions, including aggravated burglary, aggravated robbery and theft, according to court documents." He spent four years in a Kansas prison, but according to court documents he omitted those details from his employment application for San Francisco government service, The Post reported.

After the Storm
Although most of the dust has settled, San Francisco's Department of Information Technology still has a pile of digital debris to sift through. Apparently disgruntled network administrator Terry Childs left a networking device hidden on the city FiberWAN network that, as of this writing, IT staff are still trying to locate -- months following Childs' arrest.

The IDG News Service reported on Sept. 10, 2008, that an outside router was installed on the FiberWAN network that provided unauthorized remote access. City officials discovered it in August and don't have the correct user name or password, so they can't log on to the device and see what's going on. The prosecution has a screenshot of the message received when the improper login information is entered: "This system is the personal property of Terry S. Childs."


Hilton Collins

Hilton Collins is a former staff writer for Government Technology and Emergency Management magazines.

Platforms & Programs