John Ionnidis on Econtalk

Very good Econtalk with John Ionnidis. I first heard of Ionnidis in this article from The Atlantic which features his work on medical literature. If you’ve heard the claim that “most published medical research is wrong,” this is where it’s coming from. Ionnidis has turned his sights on economics.

I found the discussion of the minimum wage (~15 minute mark) very interesting. The takeaway: studies claiming to find no effect of the minimum wage on employment will usually not find the effect even if it’s real. I know there’s been a lot of recent work trying to empirically measure the effect of minimum wages on employment. The book Minimum Wages, which is kind of a meta-study on the empirical literature, was published in 2008 and is probably already out of date. There have been a lot of dueling studies of increasing sophistication (statistical if not economic) saying “There’s a significant disemployment effect when you raise the minimum wage” versus “No there isn’t.” I thought that the “No there isn’t” crowd were employing some kind of sneaky statistical wizardry to make the effect disappear. Maybe not. Ionnidis claims that, for most of these studies there will only be a ~10% chance of finding a statistically significant effect. (8.5% is the "median power" of these analyses, if I understand him.) Someone trying to build the case for “there is no disemployment effect” could do so by essentially publishing studies at random. I don't think this kind of trick would work in today's world; a funnel plot will reveal the publication bias. Still, taking Ionnidis's point, we should look skeptically at any one study. 

Here’s an excerpt of the conversation. (This seemed crystal clear and well-laid-out when I hear the podcast, but reading the transcript it’s hard to find an excerpt of the piece I found interesting. I’ve noticed this before with transcripts of conversations. “The point” is never crystallized into a single quote as a careful writer might do on purpose; it’s always distributed across several exchanges between the conversants.):

Russ Roberts: Let's do that again. Let's say that again. So, let's try to put it in the context of an actual empirical question that might be examined in economics. One of the ones you mentioned in the paper is the impact of a minimum wage on employment. And a caveat: Of course, there are many other aspects and impacts of the minimum wage besides whether you have a job or not. It can affect the number of hours; it can affect the training you receive; it can affect the way you are treated on the job. And it bothers me that economists only look at this one thing--this 1-0 variable, job-or-not. Number of jobs. Without looking at the quality, outside of the monetary, financial aspect. But, that's what we look at, often. And it is the central question in the area of minimum wage policy: Does it reduce or even expand potentially--which I think is crazy, but okay, a lot of people don't agree--whether it expands or reduces the number of jobs. Now, in such an empirical analysis of the minimum wage, how would you describe the power of that test? Meaning, there's some effect that we don't know of that impact. The power is--fill in the blank--the probably that?
 John Ioannidis: Right. So, for that particular question, the median power if I recall that we estimated was something like 8 or 9%.
 Russ Roberts: It is. I looked at it; I've got it right here. It is 8.5%.
 John Ioannidis: There you go.
 Russ Roberts: That means--so, what does 8.5% mean, in that context?
 John Ioannidis: It means that, if you estimate for each one of these studies that have been done, what are the chances that they would have found that effect? That they would have found a statistically significant signal, if the effect is what is suggested by the largest studies, for example? Their median chance would be 8.5%. So, 50% of the studies would have 8.5% chances or less to be able to detect that signal. Which is amazing. I mean, if you think of that--
 Russ Roberts: It's depressing--
 John Ioannidis: Or depressing, actually. I mean, they basically have no chance of finding that. Even if it is there.
 Russ Roberts: So, does this work on both sides of the question?
 John Ioannidis: It is very, very difficult for them to pick it up.
 Russ Roberts: Does this work on both sides of the question? Meaning: It obviously depends on your null hypothesis. So, if your null hypothesis is: Minimum wages have no effect, and I'm going to test whether they have an effect, you are going to say: Does that mean I'm going to find that I only have an 8% chance of finding that effect?
 John Ioannidis: Yeh. It would mean that even if that effect is there, you would have an 8.5% chance of detecting it.
 Russ Roberts: So, most of the time, I would not find it.
 John Ioannidis: So, most of the time you would find a non-significant result. Called a null result. Or, seemingly null result. Even though there is some effect there.
 Russ Roberts: But it could go the other way, too. Because your null hypothesis could be that the minimum wage has an effect; and I'm testing whether there is no effect. And I might not be able to find no effect. Is that correct to go in that opposite direction?
 John Ioannidis: So, what happens in the opposite direction is that when you are operating in an underpowered environment, you have two problems. One is the obvious: That you have a very high chance of false negative. Because this is exactly what power means. It means that 92%, if you have an 8% power--92% of the time, you will not be able to pick the signal. Even though it is there. So, it's a false negative. At the same time, you have the problem of having a very high risk of a false positive when you do see something that has a statistically significant p-value attached to it. And, it could be an entire false positive, or it could be a gross exaggeration of the effect size. And, um, it could be that the smaller the power that you are operating with, if you do detect something, even if it is real, the magnitude of the effect size will be substantially inflated. So, the smaller the power, the greater the average inflation of the effect that you would see, when you do detect it. So, two major problems. With low power: lots of false negatives. Second problem: lots of false positives and gross exaggeration of the effect sizes.