Clickability tracking pixel

Questions Arise Around Transparency of Idaho Crisis Data

When Idaho Gov. Brad Little unveiled his Idaho Rebounds plan, he promised data-based decisions on whether the state should progress through reopening. And that data, the plan stated, would be publicly displayed.

by Chadd Cripe, Rachel Roberts, and Audrey Dutton, The Idaho Statesman / May 15, 2020
Idaho Capitol Building Shutterstock/CSNafzger

(TNS) — When Idaho Gov. Brad Little unveiled his Idaho Rebounds plan April 23, he promised data-based decisions on whether the state should progress through a four-stage reopening.

And that data, the plan stated, would be publicly displayed.

Three weeks later, some of that coronavirus data still isn’t public — and other data is either incomplete or updated several days after the fact, casting doubt on the accuracy and timeliness of the numbers Little can examine on each decision day.

He announced Thursday that Idaho will move into Stage 2 of his plan, which opens restaurant dining rooms, hair salons and indoor fitness/recreation facilities.

The state posted a document to the governor’s Idaho Rebounds website Wednesday after the Idaho Statesman’s inquiry, showing the data Little used to enter Stage 1 of the reopening plan May 1. The document was updated as recently as May 8.

The data Little used for the Stage 2 decision was posted before Thursday’s press conference began.

The Idaho Department of Health and Welfare says that it’s working to provide more data related to Little’s criteria and that the COVID-19 pandemic is the first time the department has provided information this close to real time, which has highlighted the changing nature of epidemiological data.

The data that’s missing: testing, PPE, ICU

One of the six criteria in Little’s plan says Idaho can advance to the next stage based on the number of coronavirus cases or the percentage of positive tests over the “most recent reported 14-day period.” In both cases, a “downward trend” or a number below a predetermined threshold is required.

That data “will be displayed on the website,” the governor’s plan says, presumably referencing Idaho’s coronavirus site (coronavirus.idaho.gov) that features the state’s data dashboard.

The number of new confirmed and probable cases is posted, but not the data required to calculate the positive testing percentage (number of tests and positive results on a given day).

The Idaho Statesman was tracking Idaho’s testing percentage on a daily basis based on returned results reported by Health and Welfare until the state dumped more than 8,000 negative tests into its system April 28 — obliterating that data set. The tests were added as Health and Welfare tried to account for tests conducted by those other than the state lab and commercial labs — hospitals doing their own testing, for instance.

However, Health and Welfare says it is basing this metric on the positive testing percentage on the day a test was taken. That calculation requires knowledge of how many tests were performed on a certain day (information displayed on the Health and Welfare data dashboard, but updated only by week) and the number of positive results those tests created (which isn’t displayed at all).

The closest data Health and Welfare provides for this metric is weekly testing percentages. It reported 4% positive tests for May 3-9, 3.7% for April 26-May 2 and 3.8% for April 26-May 2. The data is updated through Saturday, May 9. The governor’s plan is broken into 14- and 15-day windows because it takes all of that time to see the impact of opening phases — so data through May 9 isn’t timely enough for a decision made on May 14.

The data also was incomplete as of Wednesday evening.

The Health and Welfare testing data through May 9 showed 1,711 positives and 34,287 total tests. But as of May 9, the state had reported 2,049 confirmed cases on its site. That means at least 338 positive cases were missing from the state’s weekly calculations — or 16.5%. Idaho hit the 1,700-case mark on April 23.

In addition, the total tests shown through May 9 is more than the total number the state reported through Wednesday (33,556).

Not all positives are included in the positivity chart because some labs don’t report their negative results, Health and Welfare spokesperson Niki Forbing-Orr said Thursday. Including the positives without the negatives would make the percentage inaccurate, she said. The difference in the total number of tests, she said, is because the numbers in the positivity chart are verified while the total tests number displayed every day is raw.

The testing percentage didn’t matter this time because Idaho’s rate of new cases trended down. But when it’s time to make a decision on Stage 3, set to begin May 30, it could be a critical piece of information.

Little will be provided more current data than is available in some charts on the Health and Welfare website, Forbing-Orr said.

“When he’s making these decisions, he’ll have daily data that he’ll be looking at to make his decision,” she said.

Little’s plan also requires certain standards for health care capacity and personal protective equipment (PPE): “At least 50 available (unused) ventilators, 50 ICU beds, and available 10-day supply of N95 masks, surgical masks, face shields, gowns, and gloves.” That information will be “displayed on the website,” the plan says.

But none of it was there until the ICU and ventilator stats were added late Thursday afternoon.

The governor’s plan refers to the “Idaho Resource Tracking System (IRTS) (through the Division of Public Health)” as the measurement tool for those requirements.

That system keeps track of how much PPE — such as masks and face shields used by health care workers to protect themselves and their patients — is available in Idaho. It also tracks how quickly hospitals and clinics are using up the gear. But the system operates in real time, and it wasn’t set up to maintain a historic record as of April.

The Idaho Statesman filed public record requests for the real-time snapshots, three Mondays in a row, to see how the inventory was holding up before and as Idaho received its allocation from the national stockpile. At that time, the system showed a more than 10-day supply of all PPE.

The Idaho Office of Emergency Management also has a survey to track PPE needs, which allows law enforcement and other nonhospital entities to let the state know when they need more equipment. That helps the IOEM manage requests for more PPE from the Federal Emergency Management Agency, spokesperson Janice Witherspoon said.

Idaho has a total of about 400 ventilators and about 555 ICU beds statewide. (More than 100 of those ICU beds are for newborns and children, who have been a fraction of the hospitalized patients for COVID-19.) For the April 15-28 window leading up to Stage 1, Idaho had more than 300 ventilators available every day but dipped to about 70 available ICU beds on April 20, according to the report posted to Idaho Rebounds on Wednesday. The ICU number dipped just below 100 on May 6 leading up to Stage 2, according to the data the Rebounds site posted Thursday.

The ICU and ventilator numbers are updated manually each day by someone at each hospital, Forbing-Orr said. Hospitals are required to report similar information to the federal government through a system under the U.S. Centers for Disease Control and Prevention, she said.

How new cases are figured

The April 15-28 charts posted to Idaho Rebounds showed that Idaho was calculating new cases for the governor’s criteria in a different way than is explained in his plan. The plan says the measurement will be from the “COVID-19 by Date Reported” chart on the coronavirus website. That’s based on the day the new cases are reported, which is the number followed by many people across the state.

The governor’s report, however, includes a chart of new cases based on one of four dates, depending on what’s available. Preference is given to the date of symptom onset, which is likely several days or more before the report date. That’s considered the best date to use for epidemiology — it’s closest to when someone actually contracted the virus. But it means cases are backdated to earlier dates than when they were reported — making it impossible for the public to track how the state is faring against the criteria, and likely moving cases into 14-day windows that have already closed. The coronavirus website does show cases by date of symptom onset, but because the governor’s report has four options for the date it uses, the two sources don’t match.

For April 15-28, the state reported 414 new confirmed and probable cases (29.6 per day). However, the chart in the governor’s report included only about 305 (an estimate based on the bar graphs in the chart) — or about 22 per day.

The state returned to the original “date reported” case calculation for the Stage 2 decision, Forbing-Orr told the Statesman on Thursday after this story published online. Event date is “more relevant to disease transmission than a report date” for epidemiology, she said. However, because the governor’s guidelines cite the report date, it was determined that using event date could be “confusing and inconsistent,” she said.

Hospital data concerns

Two other criteria that must be met to continue reopening the state rely on the state’s hospital data, which often is updated several days late. The criteria involve the number of emergency department visits by patients with COVID-19-like illness and the number of admissions to hospitals of those patients through the emergency department over the “most recent reported 14-day period.”

That data is displayed on the coronavirus website, and the facilities reporting represent 93% of estimated emergency visits annually, according to Health and Welfare. But there’s a delay in that reporting that could prove critical in these 14-day windows.

When Little announced his decision Thursday afternoon, the publicly available data was from Wednesday night. The hospital data was updated through Tuesday. And the ER visits number usually is artificially low — it gets increased the next day as reporting catches up. So on Thursday, the most recent full day of reporting of hospital stats was from Monday — which was day No. 11 of the first stage of reopening.

And there have been some sudden swings in that data. The ER visits for May 5, for example, jumped from two when it was first reported May 6 up to 15 visits on May 7. Three ER admits reported last week were later removed.

In the last two weeks of April, leading up to Little’s Stage 1 decision, Health and Welfare showed 1.1 emergency admits per day. Days later, updated data increased that number to 1.9 — a 73% increase. The ER visits average also increased, from 13.6 to 14.

That wasn’t the difference between opening and closing last time, and Idaho had significant breathing room on those numbers for the Stage 2 decision, but it could be a factor for Stage 3 or Stage 4.

Perhaps recognizing these challenges, Little’s April 30 announcement for Stage 1 was based on data from April 15-28. His Stage 2 announcement used hospital data from April 27 to May 10 — which ensured complete data but might not reflect illness increases caused by the reopening on May 1.

Changing data isn’t unusual for epidemiologists — but it can create issues when using that data to make decisions in a short time frame.

“Epidemiologic data are very frustrating and messy,” Dr. Kathryn Turner, Idaho’s deputy state epidemiologist, wrote in an email to the Idaho Statesman. “The data are collected from multiple sources (clinics, labs, people, proxy respondents, etc.) and the quality and completeness of information varies drastically. As an investigation goes on, the information can change and we epidemiologists are totally OK with that and used to it. That is why we have disclaimers all over the website that all data are preliminary and why it is extremely rare that we provide data in real time — in fact, this is the only time we have ever done it.

“The data changes every day and is fluid, and people can become frustrated by that.”

In addition to the case/testing data, health care capacity and two hospital stats, Idaho has criteria based on the number of health care workers who have contracted the COVID-19 disease (those numbers are publicly available, but not in the same form the governor uses) and whether the governor has an order in place to require crisis standards of care (he doesn’t).

The unanswered question

The Idaho Statesman has asked Health and Welfare repeatedly to provide clarity on how a “downward trend” in these metrics will be determined. The only answer we’ve received is that a 14-day moving average is used.

“The reason we use a moving average is that it produces a more stable and reliable trend compared with looking at day-to-day fluctuations when you are using data that might go up and down quite a bit,” Turner wrote.

It’s still unclear whether all that is required is for the 14-day average to be lower than it was two weeks earlier, or if that average spiking in the days before a decision is made would negate those gains. But the report for the April 15-28 time period provides a clue. On the new cases charts, the daily number was above average on the last day of the period but the dotted trend line — a mathematical calculation — indicated a steady decrease over time.

So if Idaho has 25 confirmed cases per day in a 14-day window and has 22 cases per day in the next window, that likely would be considered a downward trend even if that average moves upward at the end.

The last few days of data are critical to the decision-making process, Little said April 23 in a Q&A on Idaho Public Television.

“You have to do it in two-week increments,” he said of his reopening plan, “because what we do today, we won’t know the results until 14 days from now because of the incubation of the COVID-19 virus.”

©2020 The Idaho Statesman (Boise, Idaho). Distributed by Tribune Content Agency, LLC.

Looking for the latest gov tech news as it happens? Subscribe to GT newsletters.

E.REPUBLIC Platforms & Programs