Is the United States' High Incarceration Rate Inappropriate?

More precisely, how inappropriate is it?

Sorry, this is not one of my well-researched posts or one in which I can draw on my expertise. I am merely raising a question.

I get the sense that some developing nations with very poor governance just don’t bother to combat crime. Maybe they have some prisons, but these poorly-run nations don’t direct their meager government resources toward making civil society function better. Maybe these countries make greater use of corporal and capital punishment than we do. In anocracies, people probably have to resort to “self-help” to get justice. My guess is that if some of these nations didn’t execute so many criminals and let so many others go, at least some of them would have incarceration rates as high as in the US. Many would probably benefit from having more incarceration, perhaps as high as US levels.

I also get the sense that many OECD countries are too soft on their criminals. I recall watching a documentary on terrorism in Europe. It profiled a French terrorist who was a disaffected young Muslim immigrant. One of his prior offenses for which he’d served time was shooting at a police officer. I remember this being shocking, and I even said out loud to my wife, “In the US he’d be dead or in prison for life!” I realize it's a bad idea to generalize from a single example. Is this story typical of European treatment of serious criminals? I’m also thinking of a book chapter by Thomas Sowell on crime in England and the refusal of the government to dedicate the necessary resources to it. (Sorry, wish I could supply a better reference.) Maybe this impression, that Europe tends to be soft on its violent criminals, is wrong. But if not, that would explain a lot of the difference in incarceration rates.

Also, these other OECD countries often have lower actual crime rates than the US [edit: this probably isn't true as a generalization]. I’m thinking in particular of Iceland and the Scandinavian countries. These are homogeneous populations with very low crime rates. And in fact American populations with Scandinavian ancestry have similarly low crime rates. If we swapped populations or underlying crime rates with them, maybe they'd see the need for higher incarceration rates. What's the proper comparison here? All of the US to Norway, or Norwegian Americans to Norway? All of the US to Japan, or Japanese Americans to Japan? Yes, this gets quickly hits some touchy questions, but it would be daft to ignore demographics if that's one of the drivers of the difference. 

Maybe European nations have the magic formula for reforming violent criminals? Maybe the welfare state makes crime less attractive? I have serious doubts about this kind of story. Interventions aimed at social problems rarely work. It's just difficult to remold dysfunctional people into well-functioning adults. And crime has a negative expected-value payoff, so appealing to economic imperative as a rationale for crime makes little sense. My best guess is that maybe they are better at putting their ex-cons back to work, which would dramatically reduce recidivism.

The war on drugs is often provided as the reason for high American incarceration rates, but drug-related incarceration appear to be a small or modest fraction of the total [edit: this podcast with John Pfaff says it's in the 15-20% range; oh, there are important qualifiers for this figure]. I could, and I would, argue that prohibition leads to a lot of violent crime and property crime, which in turn leads to non-drug incarceration. But then again much of that crime does not get punished. I think this is a big part of the story, even if it's often overstated.

Long prison sentences? Is five years just as good as 30, and other nations have figured this out? Is "Lock up particularly rowdy teens and 20-somethings until they're at least in their late 20s" a good enough strategy, but we gratuitously keep them in prison for an extra decade or two?

My best guess is that the ideal level of incarceration is probably somewhere between the United States' level and that seen in OECD countries. It annoys me when people just report the bare statistics showing that America stands out and pretend the implications are obvious. "We lead the nation in incarceration, so obviously we're just big meanies." Or "We lead the developed world in infant mortality, so plainly we need to just copy the policies of OECD nations with lower infant mortality rates."

What should I be reading on this topic? This seems like a good start, but I'm certainly open to more recommendations. Helpful suggestions welcome, but any comment of the "[scoff] It's so obvious..." variety will not be published.

I wrote this because I think it's an uncomfortable question for libertarians. Those are exactly the kinds of questions we should concern ourselves with. What does "incarceration" look like in an anarcho-capitalist world anyway? (This isn't too much of a stumper; I've seen Bob Murphy give a pretty good answer.) Would a libertarian society have an incarceration rate closer to America's current rate, or closer to Europe's? If society's optimum rate of incarceration is high (perhaps higher than America's even), is that a way that libertarian societies are less free than societies with large states? Would we do more corporal and capital punishment? Would we tolerate private entities meting out these kinds of punishments?
___________________________________

I wrote everything above and then listened to this podcast with John Pfaff. He seems like the guy to read on this topic. If I read his book Locked In, I'll report back with what I learned. Apparently US crime rates are comparable to other OECD nations (excluding lethal violence, where we do stand out), so part of my post above is mistaken. He argues that the war on drugs doesn't really cause that much additional violence; essentially he's arguing that those (mostly) young men would be just as violent and find other violent things to do anyway. I think this is badly mistaken. It was the one "WTF?" moment in the entire interview. Columbia? El Salvador? Honduras? Mexico? These are  nations with the top murder rates in the world, and it's because of drug-market related violence. Drug markets in the US aren't quite so violent, but it's hard to believe that what's happening in these other countries isn't happening here (at least to a lesser degree). Plainly black markets make violence more attractive as a means of settling disputes; it would be hard to believe that American criminals uniquely fail to respond to this incentive. It's hard to believe they wouldn't get less violent if the rewards for murdering rivals/witnesses went away. Read Jeffrey Miron's Drug War Crimes on this topic; the relationship between prohibition and violence is very real and robust. But I'm being picky about an otherwise excellent podcast with lots of useful information. It answered a lot of my questions.