This story has been making the rounds. It was actually e-mailed to me by my boss. A study by the Highway Loss Data Institute (HLDI, pronounced "Hildy") supposedly finds that states that recently legalized recreational cannabis have seen an increase in auto accidents. They reach this result using a naive sort of time-series trend analysis. This is highly dubious. I wanted to get down a few quick reactions.
- Why are they looking just at auto collision? Why not also look at the liability coverages? Presumably liability property damage should show a similar trend. If there are more stoners and these stoners are hitting things with their cars, it should show up here. Liability bodily injury should ideally show the same trend, although admittedly this is a lower frequency coverage, so noise can swamp even a real trend. Still, it's another data source to validate their results on collision.
-In Colorado the overall collision frequency is about 5%. (As in, 5% of motorists will file a claim in a given year.) A 3% increase to this (the effect overall effect size found in the HLDI study) changes this to a whopping 5.15%. We're talking about small potatoes here.
-And yet...the effect is actually too large to be plausible. According to this document (second table, past month cannabis use rates), the population of users in Colorado rose from 7.3% in 2008/09 to 14.7% in 2014/15. So let's say 7% of the population were previously non-users but have become somewhat-regular users of cannabis (this figure would be 4% for Oregon and also 4% for Washington). For 7% of the population to drive a 3% increase in accidents, the accident frequency for those drivers would have to increase by an implausibly large 43%, which I think is higher than what anyone actually believes. Maybe if all of these new "past month users" were high all the time, but even then this is near the high end of what anyone believes is a plausible effect size.
-The calculated increase in collision frequency was 14% in Colorado, 6% in Washington, and 4.5% in Oregon. But when calculated on a "states combined" basis, the effect size was only 3%, which is smaller than any individual state. There's no mystery here; see Simpson's Paradox. The aggregate effect size can be smaller than any group's effect size (generally, whatever you're measuring) for a number of reasons. I just want to note the very wide range of estimates. It would be a mistake to pick any single one of them as the effect size. Considering my argument in the preceding bullet point, the by-state effect sizes are even more implausible than the overall effect size.
-If you look at a time series of collision frequency (industry-wide data) for Colorado and its comparison states (Utah, Nebraska, and Wyoming), nothing really jumps out at you. Colorado and Utah are sort of trending up, Nebraska is sort of flat, and Wyoming is sort of trending down. But you could easily look at the pattern and say that collision frequency is basically flat, but oscillating randomly around the 4-5% range. Any trends picked out by your eye or by regression analysis are likely to be spurious. The study is looking for very small trends (in the -5% to +5% annual change range), and saying: "We should predict that Colorado would be X based on how its neighbors are trending, but instead it's 14% above X." And it's attributing this difference entirely to cannabis legalization. Similar for the other states. This is not even close to identifying cannabis as a causal factor.
Perhaps I'll have more later.
One more thing. From the link up top: "Moore of the Highway Loss Data Institute said they hope the study's findings will be considered by lawmakers and regulators in states where marijuana legalization is under consideration or recently enacted." Matt Moore is a fine gentlemen. I've met him once and I've had several e-mail contacts with him. But I sincerely hope that lawmakers will not use his study. The study is a fine piece of time-series analysis, perhaps a good exercise for a first course in econometrics. But it's pretty lousy social science.
No comments:
Post a Comment