Mathematical models help predict the trajectory of the coronavirus outbreak. But can they be believed?

As Gov. Jay Inslee weighs the economic cost of coronavirus closures against the health risks to Washington state residents, high on his daily reading list are the latest results from a suite of computer models.

Some are from top universities around the world. Others are homegrown. All are attempting to peer into an uncertain future and evaluate possible paths forward through a pandemic unlike any other in modern times.

From state houses to the White House and city council Zoom meetings nationwide, mathematical models have rarely been so influential — or so hotly debated.

Pundits on the right insist faulty models overestimated the peril and led to economically crippling shutdowns. Critics of the Trump administration say officials are cherry-picking the most optimistic models to downplay the epidemic’s severity and claim credit for reducing the toll. To most people, the models are mysterious black boxes. The numbers they spit out are constantly changing, creating confusion about their validity and value.

But epidemiologists — and some politicians — say it’s important to understand what models can and can’t do, and how best to use them at a time when decision-makers are largely flying blind with so much still unknown about the disease and its prevalence.

“Even with the smartest people in the universe — quite a number of whom live right here in Washington state — that does not mean they have a crystal ball,” Inslee said in an interview. “We’ve had to make assumptions recognizing that there’s a high level of uncertainty.”

Because testing has been so limited, no one knows how many people are actually infected with the new virus. Asymptomatic people can spread the disease, but how easily? Does post-infection immunity persist for years, or just a few months? As activities begin to resume, health officials are trying to figure out how much testing and contact tracing will be needed to keep the disease in check.

At their best, models can leverage imperfect knowledge about the new virus to simulate alternative futures and help guide decisions. The engines that drive them are mathematical approximations of epidemics built on more than a century of experience with infectious outbreaks. As new insights arise, the model inputs change — and so do the results.

“We try to take the best science we have and put that science in motion to get to logical conclusions,” said Daniel Klein, head of the computational research team at the Institute for Disease Modeling (IDM) in Bellevue. IDM’s modeling, which is tailored for Washington, is a prominent part of the portfolio Inslee and health officials consult regularly.

It’s a mistake to hang your hat on specific numbers generated by models, said Ruth Etzioni, a biostatistician at the Fred Hutchinson Cancer Research Center. “That’s expecting too much from the models, and you are going to be disappointed every time,” she said.

What models do best is provide ballpark estimations of potential impacts, and compare strategies — like reopening restaurants, allowing large gatherings or keeping schools closed. “They are not going to be perfect, but they can provide a little lab where you can try different approaches and find out which ideas are good or bad and what might be better,” Etzioni said.

A phrase that has become almost as common as “flatten the curve” these days is attributed to the late statistician George Box: “All models are wrong, but some are useful.”

Without models, leaders facing a novel pathogen would be forced to proceed on intuition, or by trial and error — with very high stakes.

“If we didn’t have models, the alternative would just be guesswork, or a finger in the wind,” said Klein. He and his colleagues are now focused on the issue on everyone’s mind: How to safely reopen while minimizing new infections, hospitalizations and deaths.

Early in the pandemic, modeling galvanized action. Projections that more than 500,000 Britons and 2 million Americans could die if the virus was left unchecked spurred leaders in both nations to impose lockdowns. But when the modelers from Imperial College London reported sharply lower death projections due to the restrictions, the new numbers were widely misinterpreted to mean their original results were bogus.

When modeling triggers action that alters the course of an epidemic, it can appear that the modelers were ridiculously off-base, said Dr. Georges Benjamin, executive director of the American Public Health Association.

“You have to anticipate that and make sure people understand that the numbers are going to change, and that that’s what success looks like,” he said.

Different models are designed to answer different questions, but modelers often aren’t transparent about the limitations of their approach and assumptions, said Etzioni — who would like to see the equivalent of a nutrition label on every model spelling out that information in plain language.

The workhorse of epidemiological modeling is an approach that categorizes members of a hypothetical population based on whether they are susceptible to a new disease, infected, recovered — and thus immune — or killed. Researchers make their best guesses about the biology based on what’s already been learned, run the model, then see how well it matches recorded deaths or other real-world benchmarks.

This is the approach IDM has been using to estimate the “effective reproductive number” — the number of people each infected person passes the virus to — in the Puget Sound area. Early on, the number was between two and three. Thanks to social distancing, it appears to have dropped below one — a key tipping point that means the number of cases should steadily diminish.

“These types of epidemiological models are battle-hardened,” Etzioni said. “They have been used successfully many times in the past.”

But another model that originated in Seattle takes a different tack — and has drawn harsh criticism. The Institute for Health Metrics and Evaluation (IHME) at the University of Washington incorporates no information about the virus in its model. Instead, researchers chart the number of deaths in countries and states where strict social distancing has been implemented and assume deaths in other places will follow similar trajectories.

IHME Director Dr. Christopher Murray defends the approach, which he says doesn’t require assumptions about transmission rates or other biological parameters. The group issues regular updates on a web site that forecasts daily and total deaths and peak hospitalization for every state and dozens of countries.

Murray confers frequently with top lawmakers and members of President Trump’s coronavirus task force, but critics say the model promises more than it delivers.

“The appearance of certainty is seductive when the world is desperate to know what lies ahead,” a group of epidemiologists wrote in a critique that raised concerns about “the validity and usefulness” of IHME’s projections. The model’s methodology makes it volatile, with death forecasts that have swung wildly in some locations over the course of a few days, the group pointed out.

Early results for Washington projected more than 1,400 deaths by early summer, which was scaled back to 600 10 days later. On Friday, the IHME model was forecasting a total of 877 deaths statewide by early August. That’s far lower than the 1,651 deaths projected by a MIT model that’s also part of Inslee’s daily review.

One of the biggest criticisms of the IHME model is that its main assumption — that social distancing will remain in effect everywhere until early summer — is unrealistic and likely to lead to an overly optimistic outlook.

With other models projecting more than 100,000 deaths in the U.S., IHME’s early April estimate that the virus would kill 60,000 Americans by August was cited by the Trump administration as evidence the pandemic would not be as bad as feared. But the country passed that 60,000 mark last week.

IHME’s coronavirus modeling started as a way to help UW Medicine prepare for the coming surge of patients — and it was extremely useful, even though initial forecasts were far worse than reality, said Lisa Brandenburg, who’s in charge of all UW hospitals and clinics. The first model runs suggested the peak number of COVID-19 patients on a single day could range between 236 and 950. Administrators prepared for the worst case.

The actual number of hospitalizations at UW facilities peaked at 122. But Brandenburg doesn’t regret the decision to brace for disaster, even though it meant putting all other patient care on hold and a big financial hit. The shocking projections from IHME helped persuade state and local leaders to impose restrictions to stem the virus, and they certainly saved lives, she said.

“Yes, I wish we had had more perfect foresight. But I also don’t think we could have made another choice with the data we had at the time,” she said.

However, with the possibility of another wave of coronavirus cases in the fall, Brandenburg said she plans to consult a range of models instead of just one — a hedge that’s universally recommended by modelers themselves.

As health officials try to anticipate what will happen when restrictions are lifted, another type of model is starting to play a starring role. Similar to standard epidemiological models, these “agent-based” approaches are far more detailed and sophisticated. People are treated as individuals, and communities can be simulated down to the census tract level, with realistic approximations of social interactions and travel patterns.

It’s like playing the computer game SimCity, with contagion factored in — and it requires such a huge amount of computing power that Google is donating access to its cloud capabilities.

That granularity is necessary to address what are now the most pressing questions, said Dr. Elizabeth Halloran, a biostatistician at Fred Hutch and a professor at the University of Washington. Questions like: When can kids go back to school? And what’s the most effective way to expand testing and contact tracing to prevent the virus from flaring up?

“These models allow you get at complexity,” Halloran said.

She works with a group at Northeastern University whose high-powered model is already trying to find some answers. One of their most recent simulations suggests that even a relatively modest testing regimen, aimed only at people with obvious symptoms, could be effective when coupled with modest levels of contact tracing. Only about 7 percent of the population would need to be isolated at any given time, while everyone else could resume their normal lives.

But that’s just one set of simulations, from one model.

“You start putting numbers on things and people think they’re true,” Halloran cautioned. “Nothing about any model is true.”

The models Inslee pays the most attention to are currently pointing in different directions. While IHME is projecting a steep drop in infections and deaths, IDM forecasts a much slower decline. The governor is hoping the forecasts will converge as more data comes in. But so far, the main point of agreement is not a pleasant one, he said.

“The most important thing we have derived from the models — and it’s unfortunate — is that they are confident that if we eliminated all social distancing right now, the pandemic curve would go up fairly dramatically, quite rapidly.”