No Time to Grieve
New York City's technology team took on immense new challenges in the wake of Sept. 11.
In the months following the Sept. 11 attacks on America, New York City and its people were in the national spotlight. For the men and women who repaired, created and managed the information technology systems that provided the city with critical emergency services, it began with a Tuesday morning that will be forever imprinted on the nation's memory.
"There are a thousand stories out there about 'where were you when the first plane hit,'" said Brian Cohen, New York City's former deputy commissioner for the Office of Technology and E-Government. Cohen was on his way to work and was temporarily trapped in the Brooklyn Battery Tunnel. He rushed to ground zero and had to run for safety as the first tower collapsed. His car was buried in the rubble and never found. "We had a hard time breathing," he recalled. "It was like a war zone, cars exploding all over the place, ashes five inches thick. It was dark, like walking through an eclipse."
Into the Fire
At the same time, Avi Duvdevani was at MetroTech, the city's highly secure IT facility adjacent to the New York Fire Department, embarking on a new job as acting commissioner of the Department of Information Technology and Telecommunications (DoITT). "For me, there were very strong feelings. The letter that announced my appointment was dated Sept. 11," he said. "I have this letter in my hand and then the world blows up. I felt obligated to the mayor to step up."
MetroTech, sitting at the foot of two bridges in Brooklyn, supports New York's multiprotocol network with over 3,000 terminals and 120 business critical application systems for finance, criminal justice and its Web portal. From this location, Duvdevani and his staff could look across the East River and watch the smoke billowing from the remains of the World Trade Center.
Because of extensive work done prior to Y2K, the city was positioned to use its technology infrastructure and to implement emergency measures. Deputy Mayor Joe Lhota said the IT team had drilled through tabletop exercises using a variety of emergency scenarios. "We knew exactly what to do if the world came to an end on Jan. 1, 2000," he said. "When you plan for things, you remember them."
A consortium of communication companies had been involved in theY2K preparation process and, on Sept. 11, was called to action. Consequently, when Verizon's communication center near the towers was severely damaged and service to 50,000 phones was lost, the city knew where to look for alternatives. Without those Y2K meetings, Lhota believes the situation could have been dire. "We could not have known that we could go to a cable company and get telephony," he said. Cell phone services were also critically impacted, as antennas and equipment installed on top of the towers went down with the buildings.
A Beautiful Portal Lost
While Duvdevani worked to restore phone service to the core of the city, staff in the Office of New Media scrambled to keep the city's Web site up with relevant content. Diane Witek, managing director of the Office of New Media, said she rushed to the MetroTech building and, not bothering to take her coat off, marched into the content manager's office for a status report. She said traffic issues were an immediate priority. Shortly after that, the provider of the T1 line to DoITT's server, located near the World Trade Center, went down. A temporary HTML page was quickly created.
"Our beautiful portal look was gone," Witek remembered, "and it was straight text that was serving this emergency information." The text format, however, proved useful and much quicker to use than graphic-laden pages. Witek created a form to smooth the flow of donations to the city. The system administration staff worked to create a common interface for an electronic form.
Companies and individuals had inundated the city's e-mail system with offers of material, equipment and assistance. The form allowed officials to evaluate what was needed and to act upon the offers. According to Marsha Kaunitz, deputy managing director of the Office of New Media, the generosity was overwhelming. "There were a lot of condolence messages and offers of equipment of all sorts -- and the offers were amazing," she said. "We needed to organize this and get the information to [the Office of Emergency Management] so they could decide what they needed."
Larry Knafo, managing director of e-government who had led contingency planning for the city's Y2K effort, stepped in to provide IT services to what became the Family Center for the OEM -- a 95,000 square-foot building on Pier 94. Computers had to be installed, connections brought in, networks built and work stations created. "We started building on Friday morning [Sept. 14] and were asked to have it open by Sunday morning at 8 a.m.," he said. "You walked in and it's an empty pier with a concrete floor and there's absolutely nothing else."
The OEM got much of what it needed through donations. Dell, IBM and Intel all donated computers. "We received well over $10 million in donated goods and services," Knafo said. "We needed all kinds of things that you wouldn't necessarily think about. We needed copy machines. Where do you get 30 copy machines right away? We needed 150 fax machines."
Pier 94 had no incoming phone lines, so Knafo and his team leveraged the technology already in nearby Pier 92, temporary headquarters of the OEM's command center. Running cable to the Family Center from Pier 92 was a major task and crews worked around the clock. Once that task was done, Knafo worked with Cisco to bring in voice-over IP. With phones, computers and Internet access from Time Warner, they started implementing new applications specifically designed for the emergency. A consortium formed by Filenet, Accenture and CTGI built the workflow applications.
By the Sunday following the attacks, the center was operational and gearing up to interview thousands of people. Recognizing that the tragedy was global, MCI WorldCom brought in a trailer with 35 phone stations and offered free long-distance calls to people who needed to phone family and friends abroad. America Online donated two-way pagers and a bank of Internet devices. Microsoft had teams of volunteers designing applications to help people locate friends and family.
Simultaneously, work was being done to provide critical information on the city's portal. NYC.gov became the single most reliable source for emergency information. According to Kaunitz, the site received an average of 35,000 visits a day prior to the Tuesday disaster. By Thursday, the portal was up and 1.7 million visits were logged. In a normal month, the portal gets about six million page-views. In the first half of September, that number soared to about 19 million.
"It was the best source for information," Cohen said. "It showed a lot of our agencies and a lot of our commissioners how important the Web can be. We'd never think we'd lose the phones. But when we lost the phones, we still had the Internet. IT becomes a whole new medium of communication."
The value of mobile technology came home to Lhota who depended on his Blackberry to send and receive messages during the communication blackout. Cell phones from Nextel, Verizon and AT&T also provided channels of communication. Blackberry donated 100 devices. Cisco installed a new phone system in City Hall, Hewlett-Packard and IBM donated laptops and other companies followed suit." The vendor community was extraordinary," Lhota said. "I never had to ask anybody for anything."
Everybody's a Hero
Generosity was not limited to the private sector. The city of Tampa, Fla., sent hundreds of orange traffic cones to New York -- a gift that Lhota said might seem minor but proved to be a major contribution to the recovery effort. "At two in the morning on Sept. 13, the first thing I [saw] was a fire truck from the city of Chicago," Lhota recalled.
The city's Geographic Information Systems Utility also became a mission-critical tool in rescue and recovery efforts, according to Lhota. Under the direction of Alan Leidner, GIS operations responded to over 1,500 requests for maps and data. Using the mapping technology, emergency responders were able to assess the safety of buildings and analyze their efforts to extinguish fires that blazed underground. Lhota pointed out a GIS map that helped fire fighters change their tactics in this underground battle -- showing how the injection foam was actually pushing fires out to new locations, rather than dousing the flames.
Inside the walls of DoITT, the newly appointed Duvdevani guided the quiet, yet arduous work of getting the city reconnected to the outside world. "Everyone wanted to go down to ground zero," he said referring to his staff of 40 telecommunications specialists. "There was such a commitment to do something. I kept saying that we can do more good here. There was a matter of pride about keeping the Web site up and running."
Sleep became a luxury as staff moved from one priority to the next. They had to develop new communication lines, assure the integrity of the city's portal and make decisions about content. Designers and engineers were challenged to create electronic solutions to unprecedented problems. "Everybody is a hero in their own provincial area of expertise," Duvdevani said of his team. "All those techies who never get to be part of the action became part of the action. People were loyal, committed, smart and capable." E-mail and Internet access were disabled 12 hours after the crisis began, but the MetroTech team redesigned and rerouted the Internet connection before dawn the following day.
With the help of Time Warner, telephone communications were restored to City Hall and critical locations, such as the Municipal Building. Duvdevani's staff engineered a solution to pick up an Internet connection outside of the "hot zone" and keep the Web site running: Telecommunications cables were temporarily run out of upper floor windows of Verizon's damaged headquarters.
Among the lessons learned, according to Duvdevani, is how important formal planning and exercises are in a crisis. Equally as important, however, is attention to what he calls "the human factor." It had become necessary to relocate city staff to manage the emergency and for offices to share technology resources. Along with the call to immediate action, came a sense of displacement and anxiety. Duvdevani has high praise for hundreds of technology professional who answered that call. "I am especially proud of their loyalty and commitment," he said. "People literally dropped their personal lives."
New Yorkers are still marveling at the extremes that continue to emerge from events of that Tuesday in September. The thousands lost, the 80 countries touched with death, the destruction of an international landmark.
At the same time, technology proved itself. It can never again be viewed as a mere convenience. It is a critical tool, an emergency lifeline and, in the wrong hands, a weapon of destruction.
Many of the top managers who led the city's IT force moved on to new positions after the change in administration. But even those who moved on, expressed confidence in the city's technology future under its new Mayor Michael Bloomberg. "He is very technology oriented and I feel confident that IT is here to stay," Lhota said. "I feel comfortable handing over the work we have done. It will be carried forward and I feel good about it."