The response moved beyond City Hall, with a recovery effort that included Minnesota Information Technology Services (MNIT), federal and state investigators, private-sector cybersecurity specialists, and the Minnesota National Guard. Since the attack, officials have spoken to legislators, at conferences and at symposia, sharing their story in the hopes it can help other governments improve cybersecurity preparedness and response.
DETECTION AND DOWN TIME
When the network was shut down, it took internal networks, online payments and public Wi-Fi offline. A ransomware gang called Interlock was the attacker, and it uses the double extortion model — first exfiltrating data, then demanding a ransom to decrypt the data and prevent data leaks. One of St. Paul’s processes, however, is to create nightly backups, and this played a part in the city's decision not to pay a ransom.
As recovery got underway, the city prioritized 911, payroll and business services such as water delivery. Emergency services weren’t interrupted, while payment systems, the library, email and data storage were restored around the third week of August, with wider recovery taking several months.
During testimony before the state Legislature, CISO Stefanie Horvath credited “proactive investments” in cybersecurity operations for helping the city respond. Wascalus also said that St. Paul had an incident response plan well before her arrival in 2022.
STATE OF EMERGENCY
Even with those preparations, city officials determined additional support was needed. St. Paul reported the incident through MNIT’s cyber incident reporting portal and engaged a contracted cybersecurity firm. Within days, Gov. Tim Walz issued an emergency executive order activating the Minnesota National Guard's specialized cyber unit to the city of more than 300,000.
“State resources will augment the local government when the needs generated by the incident exceed the capability of local government to respond,” said Lt. Col. Brian L. Morgan, the cyber coordination cell director for the Minnesota National Guard.
Requests for National Guard support go through a vetting process that looks at factors such as impacts to public safety and health, as well as whether the entity needs help beyond its capacity. Morgan said the goal is to deploy the guard only for "the worst of the worst emergencies.”
The guard's cyber mission also extends beyond emergency response. Teams regularly train on ransomware incidents, threat hunting and critical infrastructure protection, while also building relationships with local, state and federal partners before incidents occur. Those connections help speed coordination when assisting in a major cyber event.
The guard's 177th Cyber Protection Team is made up of about 50 volunteers and a small full-time staff. A few things they provided to St. Paul included connectivity via FirstNet, laptop deployment, manpower and installing enhanced endpoint detection across city departments.
In August and still reeling from the attack, officials launched Operation Secure St. Paul, a citywide global password reset and device security check at a 5,000-seat arena. It required all employees to arrive in person.
“I wanted to make sure that everybody who was on our network was a legitimate person who belonged there,” Wascalus said. “It was a huge logistical undertaking that I think took about five days to plan but probably should have taken months.”
The effort ultimately brought more than 3,000 employees back onto city systems in three days.
“At the same time, the National Guard gave us even more people,” she said. “We changed over 3,000 passwords in person, that’s MFA credentials, and we had their devices checked to make sure that they had the right software on them.”
CONVERSATION AND COLLABORATION
In sharing their cyber recovery story, St. Paul officials are following a path taken by other governments that have publicly discussed major cyber incidents. Dallas and Nevada, for example, published public-facing after-action reports and also shared lessons at conferences and other forums.
St. Paul Mayor Melvin Carter has said that he spoke with the mayors of Atlanta and Baltimore, whose cities have also experienced cyber attacks. Wascalus said the experience changed how she thinks about collaboration and preparedness, noting that her CIO network also provided support through discussions and loaning equipment. Those experiences reinforced the value of relationships that existed before the attack.
Now, city officials are continuing to share their own lessons. St. Paul’s Digital Security Incident Info Hub remains online and an after-action report is under review. In June, Wascalus and Emergency Management Director Rick Shute are slated to speak about lessons learned at the League of Minnesota Cities conference.
“Essentially what we say is: This is what we learned. This is how you need to prep. This is what you need to be ready for in the moment,” Wascalus said.
