Saturday, April 28, 2018

When Anecdotes Are Like Data

In Albion's Seed by David Hackett Fischer, there is a long description of Andrew Jackson's courtship or his wife:

Rachel Donelson Robards was unhappily married to another man at the time. A series of complex quarrels followed, in which Rachel Robards made her own preferences clear, and Andrew Jackson threatened her husband Lewis Robards that he would “cut his ears out of his head.” Jackson was promptly arrested. But before the case came to trial the suitor turned on the husband, butcher knife in hand, and chased him into the canebreak. Afterward, the complaint was dismissed because of the absence of the plaintiff-who was in fact running for his life from the defendant.  Andrew Jackson thereupon took Rachel Robards for his own, claiming that she had been abandoned.
He goes on to say (paraphrasing this time because I cannot find the exact passage): For a cultural historian, the events themselves were not as informative as the reactions to them.

Some people have overlearned the lessons of modern science and statistics. Correlation is not causation. Rely on data, not anecdotes. The plural of "anecdote" is not "data." All excellent cautions. But if these are overdone, perfectly salient information can end up being ignored.

If a man abducts a wife and they go on to live happily ever after, that is merely an anecdote. But if a man does this in full view of his community, and the community approves, then you have some kind of signal that this sort of thing is normal and accepted.

If one man threatens another man over a jealous rivalry (and that's all the information you have), it doesn't mean much. It's just a single data point, probably an outlier, probably uncommon. But if the community approves of this kind of threatened violence (as in the man is not sanctioned by social norms or official legal punishments), then you know quite a lot more about the community from this apparently singular story.

I try to interpret news stories with this lesson in mind. A single nervous cop shoots an unarmed man. So what? Does this imply that there is a serious underlying problem? No, not really. But suppose the community almost unanimously backs the officer. Suppose that existing legal traditions exonerate him, almost reflexively. Change the example slightly. Suppose it was not just a nervous cop who perhaps thought they saw a gun. Suppose instead it was a cop who lost his temper and fired on a fleeing suspect, then attempted to plant a gun on him. And imagine that community norms and legal institutions still reflexively support him. I think you can fairly infer something about the community and underlying institutions; someone dismissing this story as "just an anecdote" would be a little bit obtuse.

Or imagine this is a case of a botched S.W.A.T. raid in which someone is  horribly injured or killed while the cops are serving a drug warrant. You don't necessarily have to know that in similar cases the cops are almost never charged with a crime or found civilly liable. You don't require a large sample of similar events to draw some useful conclusions. If a judge signs a warrant allowing a team of heavily armed police officers to raid a residence, you know that this kind of behavior is officially sanctioned. You can probably infer that this is some kind of formal practice that happens regularly without actually pulling a big data set together. On the other hand, if a team of armed burglars raids a home, it's just an anecdote. This singular event doesn't tell you anything about the likelihood that a similar burglary will happen in the future in the same community.

Too often I see police misconduct get written off as "a few bad apples." If every news story about shocking police conduct were about individual cops acting alone, I could believe it. But some of the stories I see implicate entire departments, and for that matter the communities they work in. These stories often involve police working in concert and official support for the police's questionable behavior. Radley Balko gives this example in his book Rise of the Warrior Cop: White police officers serving in a particular black community would often say racist things over the police radio. Balko gives a couple examples, which he acknowledges might be written off as outliers. But he points out that the fact they were saying these things over police radio implies that they felt comfortable doing so. They assumed people who might be listening would be sympathetic and share their values. Otherwise, presumably, the racists would keep their racist thoughts to themselves. You wouldn't necessarily require a full transcript or representative sample of all radio chatter to prove some kind of systemic racism, though of course that kind of information would be relevant, too. (I believe the example was taken from the 1970s, so I should caution that it has only glancing relevance to today's discussion of race and policing.) Even a few examples of this kind can be strongly suggestive.

Another random example comes to mind. Suppose one man beats another man to death by taking a rock to his head. Then change the example and suppose an entire village decides to bury a man up to his neck and throw rocks at him until he dies. I think in the second case you can infer quite a lot more about the cultural traditions of that society, and it has nothing to do with having "a larger sample size".  (Larger sample, as in more people involved. In a statistical sense these are really not independent observations anyway but are highly correlated. A public stoning is still a single observation.)

Other examples? I think this is a useful concept, even if I am wrong about police misconduct being a systemic problem. (That's not the point of this post!)

No comments:

Post a Comment