Publication Bias In Climate Science

Some recent research I've been doing has lead to an interesting experience. I'm always frustrated at the way science is communicated to the public. This was another example of something that disappointed me.

I was trying to figure out if there is publication bias in climate science. More specifically, I was looking for a funnel plot for the "climate sensitivity," something that would quickly and graphically show that there is a bias toward publishing more extreme sensitivity values.

Climate sensitivity is the response of the Earth's average temperature to the concentration of CO2 in the atmosphere. The relationship is logarithmic, so a doubling of CO2 will cause an X-degree increase in average temperature. To increase it another X-degrees would require another doubling, and so on. Obviously there are diminishing returns here. It takes a lot of CO2 to keep increasing the Earth's temperature.

If we focus on just the contribution from CO2 and ignore feedback, this problem is perfectly tractable and has an answer that can be calculated by paper and pencil. In fact Arrhenius did so in the 19th century. (He even raved about how beneficial an increase in the Earth's temperature would be, but obviously many modern scientists disagree with his optimism.) A doubling of atmospheric carbon gets you a 1º Celsius increase in average temperature. The problem is that carbon is only part of the story. That temperature increase leads to there being more water vapor in the atmosphere, and water vapor is itself a very powerful greenhouse gas. So the contribution from water vapor amplifies the contribution from carbon, so the story goes. This doesn't go on forever in an infinite feedback, but "converges" to some value. There are other feedbacks, too, but my understanding is that water vapor is the dominant amplifier.

This is a live debate. Is the true climate sensitivity closer to 1º C per doubling of CO2, or 3º (a common answer), or 6º (an extreme scenario)? This is what I was looking for: a funnel plot of published estimates for the climate sensitivity would reveal publication bias.

I found this paper, titled "Publication Bias in Measuring Climate Sensitivity" by Reckova and Irsova, which appeared to answer my question. (This link should open a pdf of the full paper.)

Figure 2 from their paper shows an idealized funnel plot:

If all circles are actually represented in the relevant scientific literature, there is no publication bias. But if the white circles are missing, an obvious publication bias is present. The idea here is that for lower-precision estimates (with a higher standard error), you will get a big spread of estimates. But journal editors and perhaps the researchers themselves are only interested in effects above a certain size. (Say, only positive effects are interesting and negative effects are thrown out. Or perhaps only climate sensitivities above 3º per doubling of CO2 will ever see the light of day, while analyses finding smaller values will get shoved into a file drawer and never be published.) In fact, here is what the plot looked like for 48 estimates from 16 studies:

It looks like there is publication bias. You can tell from the graph that 1) low-precision low-sensitivity estimates (the lower-left part of the funnel) are missing and 2) the more precise estimates indicate a lower sensitivity. The paper actually builds a statistical model so that you don't have to rely on eye-balling it. The model gives an estimate of the "true" climate sensitivity, correcting for publication bias. From the paper: “After correction for publication bias, the best estimate assumes that the mean climate sensitivity equals 1.6 with a 95% confidence interval (1.246, 1.989).” And this is from a sample with a mean sensitivity of  3.27: “The estimates of climate sensitivity range from 0.7 to 10.4, with an average of 3.27.” So, at least within this sample of the climate literature, the climate sensitivity was being overstated by a factor of two. The corrected sensitivity is half the average of published estimates (again, from an admittedly small sample).

I read this and concluded that there was probably a publication bias in the climate literature and it probably overstates the amount of warming that's coming. Then I found another paper titled "No evidence of publication bias in climate change science." You can read the entire thing here.

My first impression here was, "Oh, Jeez, we have dueling studies now." Someone writes a paper with a sound methodology casting doubt on the more extreme warming scenarios. It might even be read as impugning the integrity or disinterestedness of the scientists in this field. Of course someone is going to come up with a "better" study and try to refute it, to show that there isn't any publication bias and that the higher estimates for climate sensitivity are more plausible. But I actually read this second paper in its entirety and I don't think that's what's happening. We don't have dueling studies here. Despite the title, the article actually does find evidence of publication bias, and it largely bolsters the argument of the first paper. Don't take my word for it. Here are a few excerpts from the paper itself:

Before Climategate, reported effect sizes were significantly larger in article abstracts than in the main body of articles, suggesting a systematic bias in how authors are communicating results in scientific articles: Large, significant effects were emphasized where readers are most likely to see them (in abstracts), whereas small or non-significant effects were more often found in the technical results sections where we presume they are less likely to be seen by the majority of readers, especially non-scientists.

 Sounds kind of "biased" to me.

Journals with an impact factor greater than 9 published significantly larger effect sizes than journals with an impact factor of less than 9 (Fig. 3). Regardless of the impact factor, journals reported significantly larger effect sizes in abstracts than in the main body of articles; however, the difference between mean effects in abstracts versus body of articles was greater for journals with higher impact factors.

So more prestigious journals report bigger effect sizes. This is consistent with the other study linked to above, the one claiming there is publication bias.

From the Discussion section of the paper:

Our meta-analysis did not find evidence of small, statistically non-significant results being under-reported in our sample of climate change articles. This result opposes findings by Michaels (2008) and Reckova and Irsova (2015), which both found publication bias in the global climate change literature, albeit with a smaller sample size for their meta-analysis and in other sub-disciplines of climate change science.

I found the framing here to be obnoxious and incredibly misleading. The Michael’s and the Reckova and Irsova paper (the later linked to above) both found significant publication bias in top journals, and the “No evidence of publication bias” paper found essentially the same thing. In fact, here is the very next part:

Michaels (2008) examined articles from Nature and Science exclusively, and therefore, his results were influenced strongly by the editorial position of these high impact factor journals with respect to reporting climate change issues. We believe that the results presented here have added value because we sampled a broader range of journals, including some with relatively low impact factor, which is probably a better representation of potential biases across the entire field of study. Moreover, several end users and stakeholders of science, including other scientists and public officials, base their research and opinions on a much broader suite of journals than Nature and Science.

So this new paper looking at a larger collection of publications and published estimates confirmed that top journals publish higher effect sizes. It’s almost like they said, “We did a more thorough search in the literature and we found all those missing points on the funnel plot in Reckova and Irsova.” See the effect size plot, which is figure 3 in the paper:

Notice that for the full collection of estimates (the left-most line marked "N = 1042"), the average estimate is close to the 1.6 estimate from the other paper. Essentially, the first paper said, “We found a bias in top-level, high-visibility journals. We filled in the funnel plot using a statistical model and got a best estimate of 1.6.” And the second paper said, “We found a bias in top-level, high-visibility journals. We filled in the funnel plot by looking at more obscure journals and scouring the contents of the papers more thoroughly and got a best estimate of 1.6.” The later paper should have acknowledged that it was coming to a similar conclusion to the Reckova and Irsova paper. But if you just read the title and the abstract, you’d be misled into thinking this new “better” study refuted the old one. If you Google the name of the paper to find some media reports on it, you will see that some reviewers read the title only, or shallowly skimmed the contents and didn’t read the papers it’s commenting on.

 Here is more from the Discussion section:

We also discovered a temporal pattern to reporting biases, which appeared to be related to seminal events in the climate change community and may reflect a socio-economic driver in the publication record. First, there was a conspicuous rise in the number of climate change publications in the 2 years following IPCC 2007, which likely reflects the rise in popularity (among public and funding agencies) for this field of research and the increased appetite among journal editors to publish these articles. Concurrent with increased publication rates was an increase in reported effect sizes in abstracts. Perhaps a coincidence, the apparent popularity of climate change articles (i.e., number of published articles and reported effect sizes) plummeted shortly after Climategate, when the world media focused its scrutiny on this field of research, and perhaps, popularity in this field waned (Fig. 1). After Climategate, reported effect sizes also dropped, as did the difference in effects reported in abstracts versus main body of articles. The positive effect we see post IPCC 2007, and the negative effect post Climategate, may illustrate a combined effect of editors’ or referees’ publication choices and researchers’ propensity to submit articles or not.

Remember, this is from a paper titled “No evidence of publication bias in climate change science.” Incredibly misleading. This entire paragraph is about how social influences and specific events have affected what climate journals are willing to publish. 

“What is the true climate sensitivity?” is really a central question to the climate debate. The 3⁰ C figure is frequently claimed by advocates of climate interventionists (people pushing a carbon tax, de-industrialization, etc.), but the 1.6⁰ C figure is more plausible if you believe there’s a publication bias at work. The actual concentration of carbon has gone from 280 parts per million in pre-industrial times to 380 parts per million today, and the global average temperature has risen by about 0.8⁰ C. (Maybe it's actually more than 0.8⁰ C; 2015 and 2016 were record years and some commentators are extremely touchy about this point. Apologies if I'm missing something important here, but then again any conclusion that depends on two data-points is probably not very robust.) If the sensitivity is low, then we can keep emitting carbon and it’s really no big deal. If water vapor significantly amplifies the effect of carbon, then we’ll get more warming per CO2 doubling. There is a related question of “How much warming would it take to be harmful?” To do any kind of cost-benefit analysis on carbon reduction we’d need to know that, too. But clearly the sensitivity question is central to the climate change issue. If there’s any sort of publication bias, we need to figure out how to correct for it. People who cite individual papers (because they like that particular paper) or rely on raw averages of top journals need to be reminded of the bias and shamed into correcting for it, or at the very least they need to acknowledge it.

This is just the beginning of a new literature, I’m sure. There will be new papers that claim to have a “better” methodology, fancier statistics, and a bigger sample size. Or perhaps there will be various fancy methods to re-weight different observations based on…whatever. Or different statistical specifications might shift the best point estimate for the climate sensitivity. (I can imagine a paper justifying a skewed funnel plot because the error is heterosketastic: “Our regression assumed a non-normal distribution, because for physical reasons the funnel plot is not expected to be symmetric…”) I’m hoping this isn’t the case, but I could easily imagine a world where there are enough nobs to tweak and levers to pull that we’ll just get dueling studies forever. There are enough "researcher degrees of freedom" that everybody can come to their preconceived conclusion while convincing themselves they are doing sound statistics. Nobody will be able to definitively decide this question of publication bias, but each new study will claim to answer the critics of the previous study and prove, once and for all, that publication bias does exist (oops, I mean doesn’t exist). My apologies, but sometimes I’m an epistemic nihilist. 

_______________________________________________________________________

It seems weird to me that there are only a few publications on publication bias in the climate sciences. "Publication Bias in Measuring Climate Sensitivity" was published in September 2015, and "No evidence of publication bias in climate change science" was published in February 2017. I remember trying to search for the funnel plot in early 2015 and not finding it. Possibly the September 2015 paper was the first paper ever to publish such a plot for climate sensitivity. If there is a deeper, broader literature on this topic and it comes to a different conclusion, I apologize for an irrelevant post. (Sometimes the literature is out there, but you just don't know the proper cant with which to search it.) But it looks like these two papers are the cutting edge in this particular vein. If more studies come out, I'll try to keep up with them.