Beware the Man of No Theory

Scott Alexander has a great post from a few years ago titled Beware the Man of One Study. You should read the whole thing yourself, or listen to it on the SlateStarCodex podcast, which is basically a podcaster named Jeremiah reading Scott’s posts. (Did you know there was such a thing? Pretty sweet, right?)

Scott warns against putting too much faith in any single study. He even points out that you can’t trust any single meta-study and points to conflicting meta-studies on the minimum wage reaching opposing conclusions. In the academic minimum wage wars, one side can present a letter signed by 500 economists opposing a minimum wage increase, while the other can present a letter signed by 600 economists supporting it. There is simply no consensus about whether the minimum wage is good or bad on net. No single study, in fact no single body of work, proves definitively one way or the other. 

Scott then presents a funnel plot, which is evidence of publication bias. (I wrote about this topic wrt climate sensitivity.) Take a look at the figure in Scott's post; it references Doucouliagos and Stanley (2009), which I presume is a paper titled Publication Selection Bias in Minimum Wage Research? (with the question mark as part of the title). When I looked at this figure, I thought the scale on the x-axis was crazy. It spans from -20 to +10. It looks like there’s a point at +5, meaning a 10% increase in the minimum wage results in a 50% increase in employment. Of course that’s nuts, and its position low on the y-axis tells you it’s not a credible estimate. Neither are the very large negative values. The academic debate about the minimum wage isn’t about whether the elasticity is -5 or +1. It’s about whether the elasticity is closer to -0.1 or zero, or perhaps even very slightly positive. The scale of the x-axis obscures where the action is at. Just eye-balling the figure won’t tell you whether the bias-corrected average is zero or -0.1 (an estimate preferred by Neumark and Wascher’s book Minimum Wages), because the scale of the x-axis doesn't allow your eye to make out differences that small.

 I don’t know where Scott got that figure, but here’s the same chart from a published version of the (presumably same) paper.

On this scale it’s clearer that the bias-corrected estimate should be close to zero. The paper actually gives some estimates of the bias-corrected elasticity and a discussion of their statistical model. See Table 3. Depending on the exact model specifications, this is telling us that a 10% increase in the minimum wage results in a 0.09% decrease in employment, or a 0.04% decrease, or a 0.06% decrease…

Let’s say I totally buy the “publication bias” story and want to use these as my bias-corrected estimates. Does anyone really think we can increase the minimum wage by 100% and it will only cause a 0.9% reduction in employment? In other words, doubling the minimum wage will not even cause 1% of the affected workers to lose their jobs? Will a 300% increase (raising the federal minimum to $29/hour) only cause a 3.6% decrease in employment? Does anyone think these are remotely sensible estimates for the size of the effect?

I'll augment Scott's warning about the "Man of One Study." Beware the Man of No Theory. Don’t trust anyone who says that their opinions are all “evidence based” or that they’ve crafted their worldview simply by looking at “the data.” There is no such thing as a theory-free interpretation of "the data." We always need some kind of grounding in common sense, mathematics, logical consistency, and various academic disciplines to inform our interpretation of the data presented to us. In this case, the common-sense notion that "When the price goes up, people buy less" should ground us in reality. (I hope it's very clear that I'm not accusing Scott of being a "Man of No Theory." As far as I can tell he doesn't fall for naive empiricism.)

If a minimum wage advocate said "We can increase the minimum wage from $7.25/hour to $29/hour and it will only cause a 4% reduction in employment," something has gone wrong. Even pro-minimum wage economists who advocate increasing the minimum tend to suggest modest increases and issue caveats about how an elasticity measured for a small increase doesn't necessarily apply to a large increase. Read what the pro-minimum wage economists actually say. They don't extrapolate the elasticity estimates very far beyond the (usually modest) minimum wage increases that they are calculated from. Arin Dube suggests half of the median wage as a reasonable target for the minimum wage. And his congressional testimony offers many suggestions for how businesses deal with minimum wage hikes without having to fire workers. He and economists like him are clearly grounded in the Econ 101 framework. They feel some need to explain why that story doesn't apply to the low-wage labor market, and the careful ones will often caution that it does apply when you raise the minimum high enough. In other words, they are grounded in theory. They're not coming from a place where any result makes as much sense as any other. They are not naively empirical.

(Read Dube's written testimony to congress. There is a discussion of how employers can adjust by hiring higher-skilled workers. There is discussion of how employees engage in more intense job search and hang on to jobs longer than they otherwise would, absent a minimum wage hike. It's like he's acknowledging that the Econ 101 story should be true, that it's a perfectly reasonable a priori assumption, and the apparent null effect on employment is a mystery that needs explaining. He doesn't pretend like the null result needs no explanation at all, as if we can just totally ditch economic theory and common-sense intuitions about how the labor market should work.)

Not all points on that funnel plot are created equal. It turns out that there is some very recent research, using a richer and more detailed dataset than anything that was previously available, studying a relatively large increase in the minimum wage, that shows significant disemployment effects. It yields an elasticity greater than 1, which would make it an outlier on the funnel plot above. But that same study also duplicates the "no significant effect" result when it restricts itself to the data available to other studies. The new study has access to everyone's actual wages (hours worked and total earnings) before and after the minimum wage hike. It doesn't have to rely on proxies for "minimum wage worker", like "restaurant workers" or "teenage workers", not all of whom are minimum wage workers. And it can detect changes in "hours worked", which is presumably more responsive than actual job losses.  Maybe I'm ignoring Scott's advice and being a "man of one study" here (actually two separate papers by the same group of researchers on Seattle). But when a ground-breaking new study 1) uses a much richer dataset 2) finds a result that is more consistent with theory than previous work and 3) replicates results of prior work when it restricts its dataset to the detail available in previous studies, I tend to put a lot of faith in the new study. You need some theory to tell you which studies are more credible. I believe the following: "Having access to actual wages of individual workers gives me a better estimate of the effects on low-wage workers than using a crude proxy, like 'teenagers' or 'restaurant workers'." I could do some crude back-of-the-envelope, assuming a known real effect of a minimum wage increase, and showing that the crude proxy for minimum wage workers yields a smaller, less statistically significant result. I'd be using a little bit of math and some statistics. I'd be using theory to inform my worldview.

Scott says of the funnel plot,

The bell skews more to left than to the right, which means more studies have found negative effects of the minimum wage than positive effects of the minimum wage. But since the bell curve is asymmetrical, we interpret that as probably publication bias. So all in all, I think there’s at least some evidence that the liberals are right on this one.

Emphasis mine. I'm not sure if Scott is saying the liberals are right that there's a publication bias present (in which case, they are right), or if he's saying that they're right to ignore the effects of minimum wage on employment (in which case, they aren't right). If it's the latter, I'm going to push back on this hard and say, No, theory still matters.

"All these minimum wage studies show that raising the minimum wage has no effect on employment."
"All these minimum wage studies show that 'teenage workers' and 'restaurant workers' are a poor proxy for 'minimum wage workers'."
"All these minimum wage studies show that jurisdictions only raise the minimum wage when the local labor market is ready for a hike. Jurisdictions that are likely to have negative labor market impacts anticipate this and decline to raise the minimum wage."
"All these minimum wage studies show that businesses can effectively anticipate a minimum wage hike, given the time it takes to make it through the political process and the phase-in attached to most legislation."
"All these minimum wage studies show that reality is very messy, with many causal factors coming together at once. Real effects can be socially significant but still be numerically small, swamped by noise in our measurements. In this kind of world, it's hard to measure the effect of one thing on another thing."

All of these explanations are consistent with the new minimum wage research. You need theory, a worldview informed by some kind of prior belief, to decide which ones are most appropriate.

By the way, the Congressional Budget Office (CBO) got a larger estimate for elasticity when adjusting for publication bias:

 On the basis of that review, CBO selected a central estimate of that elasticity of -0.075; in other words, a 10 percent increase in the minimum wage would reduce employment among teenage workers by three quarters of one percent.
…
Second, CBO considered the role of publication bias in its analysis. Academic journals tend to publish studies whose reported effects can be distinguished from no effect with a sufficient degree of statistical precision. According to some analyses of the minimum-wage literature, an unexpectedly large number of studies report a negative effect on employment with a degree of precision just above conventional thresholds for publication. That would suggest that journals’ failure to publish studies finding weak effects of minimum-wage changes on employment may have led to a published literature skewed toward stronger effects. CBO therefore located its range of plausible elasticities slightly closer to zero—that is, indicating a weaker effect on employment—than it would have otherwise.