The lights began to flicker in the Landmark Office Building in downtown Lansing about 4:15 p.m. on Thursday Aug. 14. Our leadership team was just wrapping up the biweekly business meeting, and when the lights went out, the 15 men and women in the room stared at each other in stunned silence.

Downtown streets quickly filled with people scurrying around wondering what was happening and how to get home. Getting out of the parking garage became a 30-minute challenge in accident avoidance. Several commuters volunteered to direct traffic on busy street corners.

Cell phones either didn't work or were constantly busy. The real shock came when I heard a radio announcement that the entire northeastern United States, including New York City, was experiencing a blackout. No one knew the cause. My mind was flooded -- was this another 9/11? Could this be a terrorist attack? What was going on?

Assessing the Situation

As Emergency Management Coordinator for the Department of Information Technology (DIT), I reported to the State Emergency Operations Center (SEOC), which was on generator power. As I walked into the underground facility, I thanked God we ran three emergency exercises in the past five months to prepare for moments like these. On arrival, I learned the full scope of the outage in Michigan and surrounding states. My job was to coordinate actions with other departments and ensure the DIT provided computer and communication assistance needed during the emergency.

I immediately contacted our DIT emergency coordination center, which was activated at our backup data center location -- also running on generator power. Many of our technical staff and emergency contacts were on vacation, but after working through wrong and unanswered phone numbers, representatives from every section of the DIT were connected into our phone bridge. This line was buzzing with activity for the next five days.

The SEOC quickly filled with emergency management representatives from all parts of state and local government, the Red Cross and the energy companies. Gov. Jennifer Granholm and her executive staff also were there. Slowly the activity in the room started to build as phones rang, and meetings and informal discussions formed.

An executive update was given by each organization every few hours. The governor walked around the room to hear each report and ask follow-up questions. I was impressed by her focus and hands-on approach to the crisis. I had at least half a dozen conversations with her during the blackout. During one of the briefings, I was surprised and encouraged when President Bush called the governor to promise federal support.

Over the next several days, our Public Service Commission (PSC) representative gave regular reports about the power outage's expected length in different areas. Maps on the walls showed which areas were still without power and which were still in a state of emergency. A pattern developed in which power was restored quicker than estimated, but in some cases the power was unstable and failed shortly after it was restored, hampering our computer restoration efforts.

The biggest issue was water. Many organizations, including the National Guard and the Red Cross, helped get water to southwest Michigan. Private companies donated water and others volunteered to truck it from one part of the state to another. Reports were also given by the Department of Community Health on hospital coverage and other health-related issues. The Department of Agriculture was active in resolving food spoilage issues and restaurant food safety.

Establishing Order

On Thursday night, the DIT faced numerous challenges and questions. Reports came from all over the state about whether services were up and running. Some computer servers went down when their Uninterruptible Power Supply (UPS) failed. The Executive Office wanted to update the state Web portal "" with regular messages from the governor and the Public Service Commission, but our connectivity was down.

We worked much of Thursday night to get things working again.

Over the weekend, the DIT was involved with workarounds to get unemployment extension letters out, update benefit card credits (formerly food stamp allocations), and assist in coordination for many other business processes. At one point, the Department of Community Health couldn't get an urgent e-mail to the Center for Disease Control in Colorado. They thought their emergency center's generator would enable them to keep all computer services going in emergencies like these, only to find their e-mail server was in a different building without power. Situations like these continued to arise through the following Monday.

Power was expected back in Lansing around 4 a.m. Friday. Should state employees report? Since cooling for state buildings in Lansing was provided by the utilities via chilled water, not air conditioning, would the computer rooms have enough cooling to bring up servers in time? Finally the decision was made to have Lansing employees report, even if computer networks were unavailable. Through the dedicated efforts of employees, most computer services were available by 9 a.m. Friday morning in Lansing.

In most of Detroit, power was unavailable until Saturday morning. Work continued through Monday morning as the DIT went through the same processes in Detroit that we followed in Lansing the previous Friday.

Looking Back

So what lessons did we learn? On the positive side, our previous exercises helped us. Having a common incident tracking system at the SEOC and DIT emergency coordination center was priceless. It allowed everyone at our department command center to see all the actions, alerts, logs and issues available at the SEOC. We could share common event logs and track actions taken across the enterprise.

On the negative side, we learned behaviors change when real emergencies occur. Most staff went home to check on their families before coming back work. What if power had been off in all of Michigan? Would everyone have reported as quickly? We were amazed at how an extended loss of power affected so many other areas. What if the outage would have been a week or longer? As a result, we updated several parts of our emergency plan activation procedure.

Looking back at the blackout of 2003, I realize again how vulnerable we are to emergency situations. From hurricanes to power outages to terrorists acts, we can prepare for emergency situations, but we can't control events.

For Michigan's DIT, the power outage enabled us to gain a more positive "can do" reputation with our customers. Our relatively new department has now lived through an emergency with the agencies we serve, which helped build trust. Not only did the blackout strengthen the new relationship with our client agencies, it showed the tremendous accomplishments we can achieve through teamwork in our own agency.

Despite a hectic workload and an intense atmosphere, I stepped back a few times to watch how things were running in the SEOC. I was amazed by the calm dedication and lack of panic. I was proud of the response everyone provided, and especially that our prior planning paid off. As a department, I'm proud of the excellence and teamwork we showed.

Dan Lohrmann Dan Lohrmann  |  Contributing Writer

Daniel J. Lohrmann became Michigan's first chief security officer (CSO) and deputy director for cybersecurity and infrastructure protection in October 2011. Lohrmann is leading Michigan's development and implementation of a comprehensive security strategy for all of the state’s resources and infrastructure. His organization is providing Michigan with a single entity charged with the oversight of risk management and security issues associated with Michigan assets, property, systems and networks.

Lohrmann is a globally recognized author and blogger on technology and security topics. His keynote speeches have been heard at worldwide events, such as GovTech in South Africa, IDC Security Roadshow in Moscow, and the RSA Conference in San Francisco. He has been honored with numerous cybersecurity and technology leadership awards, including “CSO of the Year” by SC Magazine and “Public Official of the Year” by Governing magazine.

His Michigan government security team’s mission is to:

  • establish Michigan as a global leader in cyberawareness, training and citizen safety;
  • provide state agencies and their employees with a single entity charged with the oversight of risk management and security issues associated with state of Michigan assets, property, systems and networks;
  • develop and implement a comprehensive security strategy (Michigan Cyber Initiative) for all Michigan resources and infrastructure;
  • improve efficiency within the state’s Department of Technology, Management and Budget; and
  • provide combined focus on emergency management efforts.

He currently represents the National Association of State Chief Information Officers (NASCIO) on the IT Government Coordinating Council that’s led by the U.S. Department of Homeland Security. He also serves as an adviser on TechAmerica's Cloud Commission and the Global Cyber Roundtable.

From January 2009 until October 2011, Lohrmann served as Michigan's chief technology officer and director of infrastructure services administration. He led more than 750 technology staff and contractors in administering functions, such as technical architecture, project management, data center operations, systems integration, customer service (call) center support, PC and server administration, office automation and field services support.

Under Lohrmann’s leadership, Michigan established the award-winning Mi-Cloud data storage and hosting service, and his infrastructure team was recognized by NASCIO and others for best practices and for leading state and local governments in effective technology service delivery.

Earlier in his career, Lohrmann served as the state of Michigan's first chief information security officer (CISO) from May 2002 until January 2009. He directed Michigan's award-winning Office of Enterprise Security for almost seven years.

Lohrmann's first book, Virtual Integrity: Faithfully Navigating the Brave New Web, was published in November 2008.  Lohrmann was also the chairman of the board for 2008-2009 and past president (2006-2007) of the Michigan InfraGard Member's Alliance.

Prior to becoming Michigan's CISO, Lohrmann served as the senior technology executive for e-Michigan, where he published an award-winning academic paper titled The Story — Reinventing State Government Online. He also served as director of IT and CIO for the Michigan Department of Management and Budget in the late 1990s.

Lohrmann has more than 26 years of experience in the computer industry, beginning his career with the National Security Agency. He worked for three years in England as a senior network engineer for Lockheed Martin (formerly Loral Aerospace) and for four years as a technical director for ManTech International in a U.S./UK military facility.

Lohrmann is a distinguished guest lecturer for Norwich University in the field of information assurance. He also has been a keynote speaker at IT events around the world, including numerous SecureWorld and ITEC conferences in addition to online webinars and podcasts. He has been featured in numerous daily newspapers, radio programs and magazines. Lohrmann writes a bimonthly column for Public CIO magazine on cybersecurity. He's published articles on security, technology management, cross-boundary integration, building e-government applications, cloud computing, virtualization and securing portals.

He holds a master’s degree in computer science from Johns Hopkins University in Baltimore and a bachelor’s degree in computer science from Valparaiso University in Indiana.

NOTE: The columns here are Dan Lohrmann's own views. The opinions expressed do not necessarily represent the state of Michigan's official positions.

Recent Awards:
2011 Technology Leadership Award: InfoWorld
Premier 100 IT Leader for 2010: Computerworld magazine
2009 Top Doers, Dreamers and Drivers: Government Technology magazine
Public Official of the Year: Governing magazine — November 2008
CSO of the Year: SC Magazine — April 2008
Top 25 in Security Industry: Security magazine — December 2007
Compass Award: CSO Magazine — March 2007
Information Security Executive of the Year: Central Award 2006