Why I’m Skeptical of the CDC’s Drug Overdose Figures

There has been much reporting of very large and very recent increases in the number of drug overdoses. If you believe the CDC’s raw figures, the total drug overdose deaths have increased from about 6 per 100,000 in 1999 to about 15 per 100,000 in 2014. Overdoses involving opioids (painkillers, heroin, methadone, etc.) have increased from about 3 to 9 per 100,000 in the same time period. A ~3 fold increase in overdose deaths is indeed alarming. But is it real? Does it make sense? Does the world really change that much in 15 years? What are the implications for drug policy? How is this possible when illicit drug use is basically flat over that time period? These are the questions that drove me do some pretty extensive research on this topic. I found that, unfortunately, nobody has written the definitive book, or even the longish article, on the question I wanted to answer. So I’ve done some digging and here’s what I’ve found.

Inaccurate Determination of Cause of Death

Steven B Karch. Learn his name. Read his textbook, Pathology of Drug Abuse. If you want to have an informed opinion on drug policy, you should understand how drug overdoses happen, which drugs are the biggest “offenders”, how a determination of cause of death is made, how often a mixture of drugs (rather than an “overdose” of a single drug) is the culprit, and a host of other issues discussed in his textbook. I first saw his name in this article by Radley Balko, which is also skeptical of the CDC’s alarmism on recent overdose deaths.

“Dr. Steven Karch, who has written a widely used textbook on drug abuse and pathology, says because tolerance for opioids can vary so much from person to person, there's no scientific way to definitively say that a death was caused by an opioid overdose. "There are plenty of people walking around with levels of opioids in their bodies that would be declared toxic if they were dead on a slab in a medical examiner's office," Karch says. "Toxicology is the least important part of making a diagnosis."

"I don't know where they got their numbers," Karch says of the CDC estimates. "There's no peer review of those figures. You follow the footnotes, and it looks like they're getting the information from medical examiners. But it doesn't say how the medical examiners are concluding that these were overdoses--if, say, they're just relying on toxicology results." Asked if that's usually how overdoses are diagnosed Karch says, "That fits my experience."

So determining the cause of death isn’t exactly a science. A toxicology screening showing very high concentrations of metabolites might not actually tell you anything about the cause of death.

He’s also quoted in this article in Time:

In other words, Karch says, the "lethal dose" of these drugs is impossible to standardize, even for patients who are taking a single drug rather than a combination of many. A 2000 study by Karch compared methadone patients who had died from clear causes, such as car accidents, with those who had died of a suspected overdose. Karch found that there was no predictable difference in the postmortem blood levels of methadone between the two groups. In fact some of the patients who died of other causes had higher levels of methadone in their bodies than those thought to have overdosed. "You can die from a drug and you can die with a drug," says Karch. "When you have four orders of magnitude separating either end of the curve, many of these deaths may not have to do with drugs at all."

Here is a paper written by two other doctors saying essentially the same thing. (The link is to the abstract; the full text is available online but I’m having trouble finding a stable ungated link to it).

Current data collected by medical examiners and coroners are incomplete and inadequate to evaluate the factors that lead to fatalities involving prescription opioids. Determining cause of death is critically important. Two methods are proposed to improve consistency and accuracy in the collection and analysis of decedent data in opioid-related poisoning deaths. First, an improved death certificate is needed to collect evaluative data, including: extent to which opioids were judged to 1) cause, 2) contribute to, or 3) be present in investigated deaths; extent to which opioids as a cause of death were found 1) alone, 2) combined with other prescription drugs, 3) combined with alcohol, or 4) combined with illicit drugs; the time of death; the presence or absence of a valid prescription; and the estimated quantity of opioids taken proximal to death.

Apparently anyone who has specialized knowledge regarding drug overdoses is skeptical of the CDC’s figures. If the overdose deaths are counted inaccurately, we shouldn’t be crafting public policy based on those data.

Most of the reporting I see on drug overdose death rates comes from a CDC database that simply counts up deaths that are given a particular set of ICD-10 cause-of-death codes (mostly codes X42 and X44, although several other codes are considered “drug poisonings”, depending on the particular tabulation). If overdose deaths are being routinely misclassified, then these numbers are potentially misleading. Consider the following disturbing possibilities:

1)   Death rates attributed to drugs are overstated (or understated) in every year due to an overall bias.

2) Trends in drug-related death rates are skewed due to changes over time in assigning cause of death, as more (or fewer) deaths are attributed to drug overdoses.

If only 1) is true, then the disturbing recent trend in drug overdose deaths could still be real, but the actual death rate in every year could be overstated (or understated for that matter). If 2) is true, then the overdose death rate in one of the years between 1999 and 2014 may well be accurate, but the *trend* could be spurious. It may be an artifact of, say, more frequent toxicology screenings, more thorough autopsies, or a greater propensity to attribute a death to an overdose. If someone drops dead of an unknown heart condition, perhaps we’re more likely to investigate that person’s drug history than we were 10 or 20 years ago. So perhaps deaths are simply being shifted from a heart- (or stroke- or allergy- or what-have-you) cause of death to a drug-poisoning related cause of death. Death rates related to heart conditions have declined in recent years, and this is a *big* category responsible for a lot of deaths. So it's plausible that a small redistribution from this category could cause a large change in another category. All of this is of course consistent with the death rates or the trend rates of those death rates being accurate, but we should definitely view them with more skepticism.

CDC’s Public Database on Causes of Death

With all those caveats, let’s actually take a peek at the CDC’s own cause of death data. The CDC maintains the appropriately named Wonder database which contains every death that generated a death certificate, from 1999 – 2014. It will surely be updated as new years of data roll in. One caution: if you’re using the Multiple Cause of Death database, it will double-count some deaths, since multiple causes can contribute to any single death. I grabbed the data for 2000 and 2014 for every single cause of death with its own ICD-10 code, and the sum was 6.5 Million in 2000 even though only about 2.5 million people died that year. Even so, it’s useful because it assigns particular drugs as causes of death. If you want to know how often heroin, cocaine, or benzodiazepines (3 of the top four killers) are implicated in deaths, this database will tell you that. So here’s what I found.

This table is the death counts and rates in 2000 and 2014 by drug type, sorted from highest to lowest 2014 mortality. (Apologies for the shitty formatting.) It looks suspicious as hell to me, because almost every single category sees a huge increase in mortality. Even the categories that are extremely small. So are we really seeing increases in every goddamn category of drug, or are toxicology screenings and patient histories just more thorough than they were a few years ago? Cocaine use has been in decline for over a decade, and yet death rates are up 26% since 2000? (See use rates for major drug categories here.) Methadone death rates are up almost 200%? I thought people were shifting toward pills. Note the first category in the table to show a *decrease* is “other and Unspecified Narcotics.” The decline is small, as are the total number of deaths in this category. I'm not suggesting that the increase in other drug mortalities is completely explained by a redistribution from the "Unspecified" category. But the decline is consistent with the notion that we know more about what drugs a decedent was taking. Look at some of the smaller categories. The dreaded Antitussives are up almost  a factor of 8. “Cannabis (derivatives)” up by a factor of 6, admittedly from a very small beginning. It is widely known that it’s impossible to overdose on smoked marijuana, so I have no idea what’s going on here. Barbiturates are up slightly; that’s weird because my understanding is that these are used less frequently than in the past (citation needed). This bolsters my suspicion that at least *some* of the recent trend is driven by simply having more information on the deceased, such as toxicology screenings and drug use history. In one narrative, “People are using more drugs, and at a constant death-rate per user, we should expect more deaths.” But would anyone toting this narrative predict that death rates would be up for almost every single drug category? Shouldn’t drug users be shifting *away* from other drugs as they have gained easy access to prescription painkillers? Of course, drug interaction is a common cause of “overdose.” People who die with prescription painkillers in their system are often killed by an interaction between painkillers and, say, alcohol, benzodiazepines, psychiatric meds, etc. So it’s possible that more people are dying overall as a result of prescription painkillers (the CDC’s narrative) and we’re just marking down whatever else is in their system. But I would have expected some dramatic declines in some of these categories if the CDC’s narrative is correct.

By the way, notice how flat the illicit use of psychotherapeutics is in this document, page 17 and 18. There are more legal prescriptions and more people taking meds, but illicit use (as defined in the document) is pretty flat (or possibly *down* even) over the past decade. That’s another strike against the “out-of-control pain meds” narrative.

The above table gives two snapshots in time. Let’s take a more thorough look at the years in between:

(Once again, shitty formatting. Sorry.) The “Other Opiods” category started to level off around 2010 at just under 4 deaths per 100k. Heroin has been skyrocketing. It doubled from 2000 to 2010 (about 0.5 deaths per 100k to about 1 per 100k), and then more than tripled (it’s 3.4 per 100k in 2014, supposedly). I can’t find good data on rates of prescription opioid use since 2011, but the sources I’m seeing with some quick Googling suggest that rates have continued their historical increased. This is odd, and inconsistent with the “more painkillers lead inexorably to more overdoses” narrative. Here’s a better narrative. Drug cops have been increasingly harassing and arresting pain patients and pain doctors. (See this excellent CATO report on the topic ). As soon as a patient starts to “look suspicious”, they get cut off by their doctor (who doesn’t want trouble from the cops). They then turn to street heroin to get their fix. Cops have also been arresting and prosecuting pain doctors, which has had a major chilling effect on that specialty. Many of them have been retiring, and young doctors are mostly unwilling to enter the profession. So pain patients instead get treated by primary care practitioners who don’t seem to know what they are doing.  (See this interesting thread in a forum where many of the posters are pain doctors. They are criticizing primary care practitioners for not knowing what they're doing when prescribing painkillers. Incidentally, note "algosdoc"’s comment #64, where he suggests that the cause-of-death coders don’t really know what they’re doing. This is consistent with my suspicion that many “overdose” deaths are misclassified.) In other words, in their zeal to “do something” about the large number of overdoses, drug cops have driven people away from prescription painkillers to street heroin, which is much easier to overdose on. They’ve put out of business the doctors who best understand pain medicine (and its associated dangers), and the slack  is being picked up by doctors who lack that specialized knowledge. Another narrative is that the street price of heroin declined relative to the price of prescription painkillers (citation needed) so people naturally made the substitution.

The recent increase in rates of heroin use appear to be real but smaller than the increase in the overdose rates. Apparently, heroin has become a more deadly habit than it was 10 years ago, in addition to being more common. If you know anything about how heroin overdoses happen, you should realize that most of these deaths are preventable and wouldn’t happen in a regime of legalization. Countries that have taken a “harm minimization” approach have seen overdose rates plummet. 

I’m not certain which narrative or explanation best fits the above observations. But clearly overdoses can move independently of the overall use rates. We should all be skeptical of the “roughly constant mortality per user, so more users means more overdoses” story, because it doesn’t fit.

Miscoding the Cause of Death

I still don’t fully understand the ICD-10 coding process, but it’s fairly obvious that coding errors are frequent. See the discussion on page 75 and the accompanying chart in the 2007 Summary of Vital Statistics for New York City:

The New York City Bureau of Vital Statistics reports drug-related deaths as a combination of the following ICD codes: F11-F16, F18- F19, X40-X42, X44. “F” codes represent drug-related deaths due to chronic drug use and “X” codes represent accidental drug-related deaths. The total number of 2006 drug-related deaths in New York City was virtually the same when comparing manual to automated coding methods (979 vs. 973, respectively.) However, the number of drug-related deaths in each of the subgroups differed drastically: manual coding resulted in a significantly higher number of drugrelated deaths due to chronic drug use or “F” codes compared to automated coding (903 vs. 149 deaths) and a significantly smaller number of accidental drug-related deaths or “X” codes (76 vs. 824 deaths).

In other words, a sudden drastic jump in the number of drug poisoning deaths occurs due to changes in coding methodology. 

It’s plausible that this happens elsewhere. This CDC article suggests that there are only a few cause-of-death coders in the entire world. It seems they frequently assemble for conferences. With a small, tightly-knit group, one can imagine a kind of group-think slipping in. One can imagine a scenario where a widely reported trend feeds on itself. If the cause-of-death coders (“nosologists”) think that overdoses are more common, perhaps they’d be more likely to classify a death certificate they see as an X42 or an X44 (the most common drug poisoning categories). When studies have been done to measure the accuracy of cause-of-death coding, they tend to find sizable errors, though the exact magnitudes vary wildly (citation needed).

What It All Means

I wish I could arrive at a solid conclusion, like “So the trend is a statistical artifact…” or “Therefore the trend is exaggerated by x%...” or “While the trend is real, actual rates in any given year are over/understated by a factor of z…”  I can’t. The information required to “correct” the CDC’s morality data does not exist. One would have to do a thorough examination of every single overdose death over the last decade and a half (and we’d also have to review every death that *might have been* an overdose death that we miscoded as something else). Presumably some kind of sampling could be done to avoid reexamining the entire set, but the task would still be enormous. The people making the cause-of-death determination are often doctors. The people casting doubt on their judgments are also doctors. It looks to me like the highest levels of training and education are thrown at this problem, and it’s still riddled with errors. It’s going to be hard to gather a team of super-duper experts to “correct” their work.

I’m not going to endorse radical skepticism, where “we can’t know anything about anything.” I don’t approve of that kind of nihilism. But given the low credibility of our existing data set, we should fall back on our Bayesian priors. There are some stylized facts that we know about drug use and behavior, which can inform any policy response. Rates of illicit drug use tend to move of their own accord, with little regard for official policy. In a given year, overall rates may be up or down a little, while specific drug categories may be up or down more dramatically. But these movements are almost totally uncorrelated with the government’s official drug policy. Populations go through cycles, where a drug is discovered, becomes popular, becomes stigmatized, becomes banned, becomes forgotten, then gets rediscovered (see Ryan Grimm’s excellent book This is Your Country on Drugs for a thorough description of this process). The news media catches on just after the peak of the cycle, when interest in the drug has already started to wane. Given all this, we should be very hesitant to enact any kind of policy response to the latest hysterical drug scare.