How does Seabrook respond to natural disasters like hurricanes?
The city has a standard drill, and if looks like a hurricane, emergency managers prepare us for the worst. They look at the worst it could be and measure the response from that. If it could be even a Category 1, they usually get ready for the roofs to get blown off. We have black plastic that we cover our computers with and phone system - everything is covered in black like a little funeral.
How many times did you have to do that in 2008?
Three times [last] year. The third time [in September] we were pretty sure because of the size of Hurricane Ike that most of the Texas coast would be impacted. So we got ready for it. I finished up my stuff a day and a half before the hurricane and left town.
The main thing is having them backed up. When things really fail, the first question that anyone asks is, 'When was your last back up?' So I have always - for many, many years - if I don't do anything else, I back up. Because losing everyone's work is the most horrible thing that could happen to a network. The first time I backed up for [Tropical Storm] Edouard, I found out that it takes a very long time to back up, so as soon as that was over, I ordered gigabyte equipment to where I can back up most of the data in about two hours.
I had three or four - three minimum - fresh copies of the backup. I drove one set away on external drives, and I put one set up in the water tower. It's ironic that you put stuff in the water tower to prevent it from water, but the new water towers have a million gallons of water on top of them and they are built of solid concrete. So nothing will blow that away. If something blows that away, you might as well not come back because we wouldn't have a tax base anymore. I put my new equipment in the water tower - it's about 25 feet above ground - and we were actually getting warnings of a 20- or 22-foot storm surge the day before the storm. That would have really been terrible; I think we ended up getting about half of that.
Were you able to continue IT operations after evacuating?
Absolutely. We have two Internet connections, and I was able to use both until the storm hit. The eye of the storm came through here. I was able to connect remotely, both our T1 and Comcast connections worked. In fact, I was setting up a remote, VPN [virtual private network] portal just a couple of weeks before the storm. I was able to use that over Comcast, and I was able to VPN directly over the T1. And I was connected through the storm even after we lost power here - the generator went out about an hour into the storm, but I was able to still connect for about half a day on batteries.
What kind of work were you able to do during the storm?
Backing up - particularly the dispatch information, which was sitting on the first
floor. I was backing that up to the upstairs servers. I left two servers upstairs, and I really didn't care too much if they got blown away because I was prepared to rebuild them fairly quickly.
Generally I try to be prepared to rebuild the core infrastructure within about two hours, and about the only way you can do that is having offline images of those servers. I was pretty much ready for the whole building to go away. And you have to be; anything else is not enough.
What shape was your IT equipment in after Ike?
Amazingly our T1 came right up and ran. We were the only people we knew in our area that had an Internet connection other than the air cards. The air cards were barely working because of the traffic being so high on the wireless network. We had a megabyte and a half of Internet, which was really exciting. We all just came to City Hall and started using it to death. It was really the only way we could communicate with our citizens at that time. We couldn't take a phone call because our phone system died and we later learned that our circuits actually died.
How did you use the Internet access?
We used it for public announcements of all kinds: We were updating [the public] on the condition of the electrical system, the condition of the sewer system, which is very necessary because you can't let people back in until that's working. So we used it for communication; a lot of the phone calls were getting dropped and so e-mail was the most reliable means of communication for about a week.
What location were you working from?
We were working from our EOC [emergency operations center] at City Hall. All we had was two things: We had a tent for about three workstations and another smaller tent for administrative work. We had cables hanging out of the second floor. We had to run electrical cables for 150 feet out to get power because the only power we had was generator power.
What were the major lessons learned from the experience?
We learned that we were indeed prepared, and it was really fortunate that we had more than a month before Ike to learn what it takes to get our data away from here and get it backed up. We also learned that - they have never flooded at City Hall before - we really should have our critical equipment all upstairs; we can't have anything, like 911, downstairs. So we already started clearing space in our EOC for the 911 center, so all we have to do is take a couple of computers upstairs and keep going.
Do you have a game plan for the next hurricane season?
We are already moving our 911 center upstairs, and backup and networks already secure. For some of our databases, we have an off-site location where we can start in case everything - all of our backup, everything - goes away. We can set up operations for our critical finance operations about 400 miles north.
I am working on our virtual environment right now. In fact, I've started deploying virtual servers so that in case we completely get blown off the map, we can get that operational - it would probably take me four hours. We could actually start working in a completely virtual environment by next hurricane season so that we just set up anywhere, have a couple of workstations, have all of our domain completely configured and all of our central database is configured. That's really one of my goals - to be prepared for that. I didn't have to really have a complete plan for that in the past because I have never been prone to such complete disaster.
Follow Government Technology on Twitter.
