Science's "Reproducibility Crisis" Is Being Used as Political Ammunition

This story originally appeared on Undark and is part of the Climate Desk collaboration.

David Randall and Christopher Welser are unlikely authorities on the reproducibility crisis in science. Randall, a historian and librarian, is the director of research at the National Association of Scholars, a small higher education advocacy group. Welser teaches Latin at a Christian college in Minnesota. Neither has published anything on replication or reproducibility.

But when a report the two men wrote, “The Irreproducibility Crisis of Modern Science,” was published by the National Association of Scholars on Tuesday afternoon, it received a Congressional reception. The launch took place in a House office building on Capitol Hill. The Texas Republican Lamar Smith, chairman of the House science committee and one of the most powerful science policymakers in Washington, spoke at the event. In a statement to Undark, he described the NAS report as an “important study.”

The report offers a lucid overview of the reproducibility debate. It also suggests 40 measures to help scientists produce more rigorous, reliable research. Most of these proposed reforms will sound familiar—and welcome—to scientists concerned about the issue.

Other details of the report, though, promise to raise concerns—starting with the involvement of Smith, who has dismissed climate science—or what he calls “the climate change religion”—as liberal alarmism. And indeed, the National Association of Scholars report alludes repeatedly, without offering much evidence, to a looming reproducibility crisis in climate science itself. The organization has a history of promoting climate denialism, and William Happer, a Princeton physicist and climate change skeptic who argues that extra carbon dioxide is good for the planet, wrote the report’s afterword, in which he accuses scientists of hoodwinking “deplorables.”

The report also calls for legislation, long championed by Smith, that many scientists argue could suppress research and helps Republicans in Congress roll back environmental regulations.

To be sure, reproducibility issues pose serious challenges for scientific communities. But what happens when those issues get picked up by political activists? And what, exactly, does the National Association of Scholars hope to achieve with the report?

The NAS (its acronym, confusingly, mirrors that of the National Academy of Sciences) was founded in the 1980s, with a mission to defend the values of the liberal arts. Its membership draws heavily from the ranks of conservative academics. “Pretty much from the beginning, we took threats to the integrity of good science as one of our topics,” NAS president Peter Wood told me. “Anyone who’s concerned about the health and welfare of sciences needs to be concerned about the amount of science that doesn’t stand up to scrutiny,” he added. “Just because that’s part of who we are and what we do, it seemed like it was about time to try our hand once again at a more systematic treatment of the problem.”

Randall, the report’s co-author, told me that reading a profile of John Ioannidis in The Atlantic had first gotten him interested in reproducibility issues. Ioannidis, now a physician and researcher at Stanford, wrote a blockbuster paper in 2005 expressing “increasing concern that most current published research findings are false.” In the years since, Ioannidis has become a prominent science reformer, and his colleagues have debated whether his claim—if it’s correct—reflects the natural, erratic progress of science, a deep flaw in the modern scientific enterprise, or something in between.

Certainly, the work of Ioannidis and others has helped draw attention to the way scientists review and publish each other’s research. And the NAS report echoes many suggestions that reproducibility advocates have made over the years, including raising the threshold for statistical significance, increasing funding for research replication, pushing researchers to share data, and setting standards that require journals to be more transparent about their process of peer review.

“There is a lot to like in the report,” Brian Nosek, executive director of the Center for Open Science and a prominent reform advocate, wrote in an email to Undark. Andrew Gelman, a statistics professor at Columbia University and a vocal critic of what he characterizes as lax standards in social psychology and other fields, agreed. “Overall, I was happy with the report,” he said, although he admitted that he had not read it closely. (Randall described Gelman to me as “a model public intellectual” and cited him as an inspiration for their work).

Randall and Welser insist that their goal is to help depoliticize science, by pushing for more objective, reliable research. And yet there is a clearly discernible political edge to their work—and that’s likely in part because of NAS’s own history with science and climate issues. The organization regularly publishes articles assailing mainstream climate science. Wood, an anthropologist by training, has written about “the bogus ‘global warming consensus,'” and in a Wall Street Journal op-ed published Monday and linked to the report, he and Randall describe “the whole discipline of climate science” as “a farrago of unreliable statistics, arbitrary research techniques, and politicized groupthink.”

The organization has received donations from the Charles Koch Foundation, one of the country’s most prominent anti-climate funders. It receives a sizable portion of its annual budget from the Sarah Scaife Foundation, which climate change activists have identified as a major backer of attacks on climate science.

Kerry Emanuel, a climate scientist at MIT and former member of NAS, says he was initially drawn to the organization because he was worried about what he saw as a growing relativism in the academy, evident in the work of deconstructionist philosophers like Jacques Derrida. NAS seemed to be taking a stand against those intellectual currents, Emanuel said—though he adds that he eventually became concerned about the organization’s stances on climate change, especially during a much-publicized incident in which hackers stole thousands of emails from a group of climate scientists and accused them of misusing data.

In a 2010 article published on the NAS website, Emanuel described the event as “a scandal” — but he didn’t see it as a challenge to the scientific consensus on climate change. The National Association of Scholars, on the other hand, sought to extrapolate the Climategate incident “into a universal condemnation of the field,” Emanuel told me. “It was just patently disingenuous.”

He left the organization soon afterward.

“It sort of revealed them not to be what they claimed to be—people who stood for scientific truth and scientific integrity. It was just another organization that used that as a front,” Emanuel said. “They’re basically a political organization posing as an organization dedicated to free inquiry,” he added.

Wood disputed Emanuel’s characterization of NAS. “It’s a false charge,” he said. “Professor Emanuel is someone who seems to have a closed mind on the issue of climate change.”

For Emanuel, though, the organization’s stance on climate wasn’t just misguided—it was a betrayal of their mission to challenge the deconstruction of truth. “They were publishing articles that were just as bad as the ones that the organization was founded to counter,” he said.

For many scientists and advocates, the most concerning part of the National Association for Scholars’ report is likely to be its advocacy for the Secret Science Reform Act—a piece of legislation, first introduced in 2015, that would bar the Environmental Protection Agency from using research that is not “substantially reproducible.” Lamar Smith has been the chief sponsor of the bill, which was recently renamed the HONEST Act. Scott Pruitt, the Trump administration’s embattled EPA director, has tried to apply some of its principles at the agency.

Supporters of the bill, including the NAS authors, argue that the legislation makes sure that policymakers only use rigorous research. The NAS report suggests expanding the policy to all federal agencies, and to federal courts. But major scientific organizations have been opposing the Secret Science Reform Act for years. Sean Gallagher, a senior government relations officer at the American Association for the Advancement of Science, expressed concern that the term “reproducible” in the act is so vague that it could be used to freeze good research out of policy discussions. “Reproducible means different things to different disciplines,” Gallagher told me.

Smith and others, Gallagher said, have done little to heed the concerns of AAAS and other organizations. Stressing that he had not yet reviewed the NAS report, Gallagher suggested that efforts like the HONEST Act might not be about upholding rigorous scientific integrity standards. “You do get the sense that they don’t like some of the regulations that used science, and therefore they attack the science.”

Nosek expressed similar concerns in his email to Undark about the NAS report. “Restricting policymakers to use only evidence that meets the highest transparency and reproducibility standards would prevent policymakers from using the best available evidence in many different situations,” he wrote.

Welser and Randall said they were sensitive to scientists’ concerns. Their report, Randall said, “is a framework for further conversation and debate, which absolutely must involve scientists and policymakers together.”

Perhaps more than anything, the National Association of Scholars—and the ensuing backlash—serve to underscore some fundamental questions that linger at the center of the reproducibility debate: What does serious, constructive criticism of the scientific enterprise look like? And when does that criticism tip over into unwarranted skepticism—or even feed ideologically motivated attacks on researchers’ work?

Naomi Oreskes, co-author of “Merchants of Doubt,” an influential study of foundation- and industry-backed attacks on science, said the reproducibility debate has already been exploited by political activists. “These guys are loving it,” Oreskes, a historian of science at Harvard, told me. “Any time scientists themselves admit there’s a mistake or a problem, they’re all over it. They have a feeding frenzy because this is exactly what they want. And what they want to do is use this now to try to discredit all science.”

Just as concerning as ideological attacks, though, is the possibility that scientists will be hesitant to speak openly about the messiness of the work that they do. It’s striking that, even in an era of skepticism and conspiracy theories, scientific communities have still been able to have such open—and often contentious—debates about reproducibility.

“Ideology is a powerful influence over all our behavior,” Nosek said. “The best that we can hope for is exposing it with transparency, and correcting for it with open, constructive debate.”

Related Video

Science

Inside the Robot-Run Lab of the Future (Do Watch Your Step)

Genetics is often just moving tiny amounts of liquid around. So Counsyl, a genetic testing company has hired some robots. Actually, they didn’t just hire them; they built their own to do the work most scientists hate doing.