Did you miss a day in Statistics class? The COVID-19 pandemic is here to help.
In the wake of a startling Stanford report that suggests as many as 81,000 people could already have been infected with coronavirus in Santa Clara County, number nerds are taking to Twitter to debate sampling methods, false positives and Bayesian inferences with a furor reminiscent of the banning of @BabyYodaBaby.
The heated debate over a few percentage points — has the virus infected 2.5% to 4.2% of county residents, as the study asserts, or is the number closer to 1% — captures our cultural zeitgeist, sheltering at home in fear of both a virus and an economic meltdown.
Critics claim the study’s methodology is dangerously flawed and question the political motives of the Stanford-led team. Others have pointed to the study as proof that COVID-19 is merely a partisan-driven flu hoax, as protests broke out this weekend in parts of the country over frustrations with the shutdowns.
In response, on Sunday, the study’s authors said they are planning to soon release a detailed appendix that addresses many of the criticisms and incorporates many of the suggestions into the paper itself.
“We have received a vast number of constructive comments and suggestions on our working paper over the past couple of days,” said Dr. Jayanta Bhattacharya, professor of medicine at Stanford University.
“This is exactly the way peer-review should work in scientific work, and we are looking forward to engaging with other scholars as we proceed in this important work,” he said.
The estimate, posted on the website medRxiv, comes from a first-in-the-nation community study of newly available antibody tests of 3,300 Santa Clara County residents in early April. Like all other emerging COVID-19 research papers, the work has not been peer reviewed. (Conventional publication can take as long as a year.)
The authors contend that between 48,000 and 81,000 of the county’s 1.9 million residents had been infected with the virus as of the first week of April.
That’s 50 to 85 times more than the number of official count of cases at the time.
If true, it suggests that the large majority of people who contract COVID-19 recover without ever knowing they were infected. If undetected infections are that widespread then the death rate in the county may be less than 0.2%, far less lethal than authorities had assumed.
It also implies that the virus probably cannot be eradicated at this high level of prevalence and that “contact tracing” — tracking down people who might have been exposed to infected person — could be nearly impossible.
Whether the true infection rate is higher or lower, Santa Clara County Executive Dr. Jeff Smith remains steadfast in his interpretation of the study’s findings: It suggests that more than 95% of the population remains susceptible to infection, and asymptomatic people spread the virus.
“That all means that there is more risk than we initially were aware of,” said Smith, lamenting how some are using the study to challenge Bay Area health officials’ unprecedented stay-home orders.
The study’s authors defend their technique, saying they adjusted for the test kit’s performance and sampling techniques to estimate the prevalence of the virus in Santa Clara County.
But over the weekend, some of the nation’s top number crunchers took to Twitter to challenge the Stanford research – saying it’s an extrapolation that rests on a flimsy foundation.
I don’t think there’s a way to say this diplomatically, but I think it’s important to tell the truth:
I have zero confidence in the Santa Clara serology study, and the recent work of Eran Bendavid generally.
— Alan Cole (@AlanMCole) April 18, 2020
They contended the Stanford analysis is troubled because it draws sweeping conclusions based on statistically rare events, and is rife with sampling and statistical imperfections.
“I think the authors owe us all an apology… not just to us, but to Stanford,” wrote Andrew Gelman, a professor of statistics and political science and director of the Applied Statistics Center at Columbia University, calling the conclusions “some numbers that were essentially the product of a statistical error.”
“They need to apologize because these were avoidable screw-ups,” he wrote. “They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong.”
From the lab of Erik van Nimwegen of the University of Basel came this: “Loud sobbing reported from under Reverend Bayes’ grave stone,” referring to the famed statistician’s technique. “Seriously, I might use this as an example in my class to show how NOT to do statistics.”
Loud sobbing reported from under reverend Bayes’ grave stone.
Seriously, I might use this as an example in my class to show how NOT to do statistics. Note that the CI on specificity includes false positive rates larger than the observed fraction of positives. https://t.co/SV7VwjU5yw— NimwegenLab (@NimwegenLab) April 17, 2020
“Do NOT interpret this study as an accurate estimate of the fraction of population exposed,” wrote Marm Kilpatrick, an infectious disease researcher at the University of California Santa Cruz. “Authors have made no efforts to deal with clearly known biases and whole study design is problematic.”
Others noted that authors had agendas before going into the study. Back in March, Bhattacharya and Dr. Eran Bendavid wrote an editorial in the Wall Street Journal arguing that a universal quarantine may not be worth the costs it imposes on the economy, community and individual mental and physical health. Their colleague John Ioannidis has written that we lack the data to make such drastic economic sacrifices.
The team’s data scientists made statistical adjustments to account for the sampling problem. Because volunteers were disproportionately white and female, relative to the county’s demographics, the team gave less computational “weight” to those participants. Latino and Asian volunteers, who were underrepresented, got greater “weight.”
One major problem with the Santa Clara County study relates to test specificity. It used a kit purchased from Premier Biotech, based in Minneapolis with known performance data discrepancies of two “false positives” out of every 371 true negative samples. Although it was the best test at the time of the study, that’s a high “false positive” rate that can skew results, critics say — especially with such a small sample size.
With that ratio of false positives, a large number of the positive cases reported in the study — 50 out of 3320 tests — could be false positives, critics note. To ensure a test is sensitive enough to pick up only true SARS-CoV-2 infections, it needs to evaluate hundreds of positive cases of COVID-19 among thousands of negative ones.
This potential error in the test can easily dominate the results, they said.
Statistician John Cherian of D. E. Shaw Research, a computational biochemistry company, made his own calculations given the test’s sensitivity and specificity — and estimated the proportion of truly positive people in the Stanford study to range from 0.5% to 2.8%.
Adjusting for demographics, Cherian’s calculations suggest that prevalence could plausibly be under 1% and the mortality rate could plausibly be over 1%.
The “confidence intervals” in the paper – that is, the range around a measurement that conveys how precise the measurement is — “are nowhere close to what you’d get with a more careful approach,” he said.
Trevor Bedford, a computational biologist at the Fred Hutchinson Cancer Research Center in Seattle, said “given how sensitive these results are to performance of the assay, I don’t think it’s safe to conclude that infections are ’50-85-fold more than the number of confirmed cases.’ ”
Even if the test was completely accurate, there would still be sampling problems in the Stanford study, critics said.
Biostatistician Natalie E. Dean of the University of Florida called it a “consent problem.” Participants weren’t randomly selected — they were recruited using Facebook. This means it might have attracted people who thought they were exposed to the virus and wanted testing. And exposed people may have recruited other exposed people for the study.
“The prevalence drops off quickly when adjusted for even a small self-selection bias,” wrote Lonnie Chrisman, chief technical officer at the Los Gatos data software company Lumina Decision Systems.
Addressing the critics, Stanford’s Ioannidis, professor of medicine and biomedical data science at Stanford University, promised an expanded version of their study will be posted soon. “The results remain very robust,” he said.
In the end, no single study is going to answer the question of how prevalent COVID-19 is in our communities, scientists said. More studies with different technologies and analytic approaches are needed.
That’s coming. A UC Berkeley project, which will begin in May, will test a large and representative swath of 5,000 East Bay residents. Scientists will take saliva, swab and blood samples from volunteers between the ages of 18 and 60 around the region.
Starting Monday, UC San Francisco and a privately-funded operation will test all 1,680 residents of rural Bolinas for evidence of the virus. UCSF will launch a similar effort Saturday in San Francisco’s densely populated and largely Latino Mission District, where it hopes to test 5,700 people.
Results are expected soon from seroprevalence surveys run by other groups around the world, including teams in China, Australia, Iceland, Italy and Germany.
“This pandemic,” wrote research scientist Ganesh Kadamur, “has been one giant Stats class for everyone.”
————————————————————————————————————–
Read the study: https://www.medrxiv.org/content/10.1101/2020.04.14.20062463v1
