We're Up, Tired, But Smiling Again

This has been a rough week for our technology operations. The good news is that our critical Secretary of State systems are up and offices are open and helping customers.

by / May 21, 2011 0

This has been a rough week for our technology operations. The various headlines about two different (and unrelated) Michigan government outages tell you why our team is a bit behind on our sleep. The good news is that our critical Secretary of State systems are up and offices are open and helping customers. I’m happy again on a beautiful spring Saturday morning in Lansing.

Here are a few of the background articles covering the outages this week:

Michigan Recovers from Two major Computer Outages

SoS offices back in business

 Michigan Secretary of State offices up and running again; hours extended

Secretary of State Branches Extend Hours After Outage

 I know, enquiring minds want to know the “nitty gritty” about what caused the outages in the first place and specific details regarding what happened and how we recovered. That will come soon enough, with a detailed “Root Cause Analysis” (RCA) being performed on each situation. We owe those formal details to our agency customers and the public that was impacted. This RCA report will include steps we are taking to reduce the risk of such incidents reoccurring.

I also hope to do a longer article on this topic later this summer, with some behind the scenes conversations and perspectives on how we responded so quickly from two back-to-back situations. But for now, I felt I owed my blog readers an acknowledgement that the incidents did happen in Michigan – and say a few words about the Michigan outage articles. It was not a fun week. When it rains it pours.

While I’m all too aware of the reality that bad things happen in every technology organization, the key is how our teams respond and come back when apps are down. As I mentioned in an article last September, all government operations must be prepared for outages given various scenarios. (Though, I must admit, I never expected to be in this situation eight months later.)

As I have written over the years, this isn’t the first time, and won’t be the last time, that unplanned outages happen. However, the difference between these two outages and situations like the blackout of 2003 is that public perception and expectations are not the same. When large parts of the Northeast USA lost power, the public understood why services were down. But when an outage occurs as a result of internal people, process, or technology failures, all eyes are on your team to get back up quickly and effectively.

Most importantly, I want to thank our recovery teams who did an outstanding job of responding from the moment that these outages were reported. We have an excellent staff that “got going” when the “going got tough.” The communication demonstrated between the technology and business staff was a good sign of successful teamwork. Several of them worked more than 24-hours straight, and I am proud and thankful for their efforts.

More to come on this topic in the future. But for now, I’m hearing my kids laugh again. I’m enjoying the sunshine. I’m smiling again in Lansing. Now I get to mow my lawn.

Dan Lohrmann Chief Security Officer & Chief Strategist at Security Mentor Inc.

DAN LOHRMANN

Daniel J. Lohrmann is an internationally recognized cybersecurity leader, technologist, keynote speaker and author.

During his distinguished career, he has served global organizations in the public and private sectors in a variety of executive leadership capacities, receiving numerous national awards including: CSO of the Year, Public Official of the Year and Computerworld Premier 100 IT Leader.
Lohrmann led Michigan government’s cybersecurity and technology infrastructure teams from May 2002 to August 2014, including enterprisewide Chief Security Officer (CSO), Chief Technology Officer (CTO) and Chief Information Security Officer (CISO) roles in Michigan.

He currently serves as the Chief Security Officer (CSO) and Chief Strategist for Security Mentor Inc. He is leading the development and implementation of Security Mentor’s industry-leading cyber training, consulting and workshops for end users, managers and executives in the public and private sectors. He has advised senior leaders at the White House, National Governors Association (NGA), National Association of State CIOs (NASCIO), U.S. Department of Homeland Security (DHS), federal, state and local government agencies, Fortune 500 companies, small businesses and nonprofit institutions.

He has more than 30 years of experience in the computer industry, beginning his career with the National Security Agency. He worked for three years in England as a senior network engineer for Lockheed Martin (formerly Loral Aerospace) and for four years as a technical director for ManTech International in a US/UK military facility.

Lohrmann is the author of two books: Virtual Integrity: Faithfully Navigating the Brave New Web and BYOD for You: The Guide to Bring Your Own Device to Work. He has been a keynote speaker at global security and technology conferences from South Africa to Dubai and from Washington, D.C., to Moscow.

He holds a master's degree in computer science (CS) from Johns Hopkins University in Baltimore, and a bachelor's degree in CS from Valparaiso University in Indiana.

Follow Lohrmann on Twitter at: @govcso