When Dr. Anthony Fauci said in late May that there’s a “good chance” a COVID-19 vaccine will be ready by the end of this year, Steve Roth badly wanted to believe him. Roth, a 74-year-old New Yorker who endured fever, pneumonia and anxiety while fighting the virus, wants life to go back to normal as much as anyone. And he respects Fauci, the longtime head of the National Institute of Allergy and Infectious Diseases (NIAID), ”an awful lot,” he says. But he just doesn’t think Fauci’s timeline is realistic. Instead, he’s putting his proverbial money on mid 2021.
“Like everybody else, I’d like to see a vaccine today,” says Roth. “But what’s the real world?”
Roth is a semi-retired market researcher, not a biostatistician or epidemiologist, and hardly seems to be the kind of person you’d go to for insight into vaccine production. But in his spare time, Roth moonlights as a “superforecaster”— a member of a team of ordinary people who make surprisingly accurate predictions for the forecasting firm Good Judgment, Inc. In recent months, businesses, governments and other institutions have worked with superforecasters like Roth to help them understand how the COVID-19 outbreak might unfold.
That a group of semi-professional forecasters would somehow have accurate insight into anything as complex and important as the coronavirus pandemic sounds like the stuff of science fiction, or even ancient history—like the seers of old who told fortunes to kings and nobles. But the team behind Good Judgment, Inc. and the organization it spun off from (the research initiative Good Judgment Project) say they have established a rigorous system for identifying talented forecasters and sharpening their abilities. The company’s superforecasters undergo years of testing before they’re brought onto the team—in the early days, through tournaments sponsored by the Intelligence Advanced Research Projects Activity, and now on the open-to-the-public forecasting site Good Judgment Open. Under the current model, the best forecasters on Good Judgment Open are invited to an online community of superforecasters, where they can share ideas and contribute their predictions to a system that aggregates their forecasts. Good Judgment’s clients pay for answers to questions that are important to decision-makers; the superforecasters collect a share of the revenue. (Other prediction services, including Metaculus, use similar models, although like Good Judgment Open, Metaculus’ community is open to anyone; Metaculus also weighs all users’ input, giving more weight to better forecasters.)
Open Judgment’s superforecaster team has a track record of success, having made accurate predictions about world events like the approval of the United Kingdom’s Brexit vote in 2020, Saudi Arabia’s decision to partially take its national gas company public in 2019, and the status of Russia’s food embargo against some European countries also in 2019. Lately, they’ve taken their talents to the epidemiology world—in early February, Good Judgment’s team predicted there would be between 100,000 and 200,000 COVID-19 cases reported by March 20; the world hit the 200,000 mark one day earlier.
Unlike history’s prophets, forecasters like Roth do not claim to possess supernatural abilities. Instead, they say their accuracy is a result of using specific techniques to structure their thinking and constantly trying to improve their skills. Superforecasters also tend to share certain personality traits, including humility, reflectiveness and comfort with numbers. These characteristics might mean that they’re better at putting their ego aside, and are willing to change their minds when challenged with new data or ideas.
It’s unlikely that superforecasters like Roth could ever fully replace subject-matter experts. Michael Jackson, an associate scientific investigator at Kaiser Permanente Washington Health Research Institute, cautions that superforecasters are a “black box,” meaning their less-than-scientific methods make it impossible to vet their work in the same way that a scientist’s output would undergo peer review. And Philip Tetlock, a professor at the University of Pennsylvania and a co-founder of Good Judgment, acknowledges that there are times in which expertise is crucial (for example, he notes that some public health experts warned about the possibility of a coronavirus pandemic early in the outbreak.)
However, Tetlock argues that superforecasters have skills that experts may not: for example, they may also be more flexible than traditional scientists, because they’re not bound to a particular discipline or approach. Their predictions incorporate research and hard data, but also news reports and gut feelings. That way of working may increase their overall accuracy, says Tetlock. “Talented amateurs who pay attention to both the science and the news seem to be better at putting accurate probabilities on key outcomes in this phase of the crisis,” he says. “The experts were really good at warning us about the fundamental danger, but they may be less adept at adapting nimbly to the dynamics about this phase of the crisis.”
Superforecasters aren’t just smart, Tetlock says; they also tend to be actively open-minded and curious. They’re in “perpetual beta” mode, as he puts it in his book on the topic, Superforecasting: The Art and Science of Prediction— always striving to update their beliefs and improve themselves. Bryan Hartman, a 36-year-old computer science teacher in Illinois and superforecaster of six years, says that kind of flexibility can improve their predictions.
“[Superforecasters] provide a lot of counter information, which is called ‘red-teaming,’” says Bryan Hartman. “It’s not in a way to press anyone’s buttons. It’s always to make everyone see the whole picture … it’s very collaborative, and very few people take anything personally. We’re just trying to see if we can get it right.”
Superforecasters are particularly good at predicting how people’s choices will affect future outcomes, says Good Judgment Inc. vice president Marc Koehler. That kind of insight that could be particularly helpful during an event like the COVID-19 pandemic, given that people’s adherence to measures like social distancing and mask-wearing can have dramatic effects down the road.
“If you’re wondering how a virus rips through a herd, an epidemiologist is going to give you the best answers about whether it mutates and stuff like that,” says Koehler. “But when the herd turns into human beings who make decisions to comply or not comply with different guidelines, and when governments set policies about stay home or not at home, or keep schools open or not, when all of those different factors get involved, that’s where a group of human forecasters really excels.”
Jackson agrees that it’s possible superforecasters could better predict or accommodate for events like mass protests compared to typical viral modeling approaches. “The challenge is that what’s going to happen in the future isn’t just based on the properties of the virus itself,” he says. “It’s very much driven by human behaviors, and we’ve seen that those can change abruptly and unpredictably even in just the short course of this pandemic so far.”
Furthermore, superforecasters can assist experts in sounding the alarm early in a major crisis—which, in the case of COVID-19, can save lives. Shannon Gifford, a 61-year-old who has been forecasting for more than eight years and is the deputy chief projects officer for the Denver, Colorado mayor’s office, says her colleagues were taken aback when she nudged them as early as January to consider how COVID-19 could affect the city.
“I remember saying in large meetings, ‘Well, whatever’s happening in a couple months could be very different if this virus crashes the economy,’” says Gifford. “And people [were] looking at me blankly, like, ‘You’ve got to be kidding. What are you talking about?’
Gifford says that she wasn’t surprised that people in Denver and elsewhere underestimated the threat of COVID-19, calling it “human nature.”
“Part of it was we simply had no idea, at that point, how much spread had already taken place,” she says. “Because we had so little testing. And I think a lot of people were lulled into inaction believing we didn’t have much of a problem here, when we almost certainly did.”
Officials calling the shots, like mayors and governors, might be skeptical of the entire forecasting enterprise. But even if they ignore the superforecasters’ predictions, they could learn something from their methods. A willingness to change your mind when presented with new information, contend with your biases, challenge one another’s ideas, and break down problems into specific questions are all desirable qualities in people who make big, important decisions. “It’s the combination of thinking about what will happen, and why it will happen, that can be very useful to making decisions,” says Roth.
Correction, June 11
The original version of this story misstated the name of the head of the National Institute of Allergy and Infectious Diseases (NIAID). He is Dr. Anthony Fauci, not Dr. Anthony Facui.