Occasionally I see stories about actuaries who behave unethically or even illegally. I don't feel any urge to leap to their defense. I have no impulse to shout down all criticism. I don't feel the need to explain to the silly uninitiated public that you can't make actuaries accountable for their financial estimates and opinions because some things are unpredictable and subject to chance. I think, "That's outrageous and contrary to the public interest. That person should be punished, perhaps jailed."
Some of these stories involve some kind of accounting fraud. A government actuary signs off on an unsound, underfunded public pension. Perhaps they use a life table from the 1970s, thus understating mortality and thus understating the total liability of life-long pensions for retirees. Or they use an absurdly generous discount rate, assuming the fund will earn 8% on investments forever, rather than responsibly discounting liabilities at the risk-free rate. (This is how my home state of Illinois has fraudulently accounted for pension liabilities for decades.) Or maybe they are less sophisticated about the deception and sign off an an unsound pension fund. The real hit doesn't come until decades later, when the fund suddenly becomes illiquid and retirees stop receiving their checks.
I would say the actuary in these cases acted unethically. Sure, the government, perhaps bowing to a public employee union leader, shopped around for the shadiest actuary they could find. Inevitably they find someone who will invent a paper fiction by which to cram oversize liabilities into the Procrustean bed of inadequate revenues. Even if this result is sort of inevitable, I still say the actuary who signs off is unethical.
So I don't understand why police officers, teachers, and medical professionals are so defensive when one of their tribe gets publicly shamed for malfeasance. You see the same thing when someone suggests measuring and rewarding good performance. There is this weird defensiveness that "performance is impossible to measure."
"You can't scrutinize how police respond with violence," says the police, "because scrutiny even of a bad actor will incentivize the wrong response when another situation appropriately calls for the use of force." There is certainly something too this. You have to strike the right cord of balancing false positives and false negatives. But the reaction is far too strong, and in some instances cops stridently defend incredibly poor judgment. You have to hold some of these people accountable. The impulse to jump to the defense of every bad actor makes cops looks sleazy, like they are anticipating the need to cover their own bad behavior.
"You can't scrutinize teacher performance," say teachers, "because children are responsible for studying and learning, and some students don't do their parts." There is a version of this for the medical profession, too: "You can't scrutinize doctor performance, because patients so often fail to follow doctors' instructions." Of course, it would be foolish to hold these people responsible for any given pupil's failure to learn or any given patients failure to improve. But to say you can't average over a large population of students and measure the "value added" of a teacher? That's kind of implausible. Same with a doctor or clinic, treating thousands of patients a year. "Selection effect!" you say? As in, good teachers get sent the best pupils, and conversely good doctors get sent the worst patients. That's why you measure "value added", not merely value. You don't just measure, say, test scores for pupils or health status for patients. You measure these things against some baseline expectation. If school masters in third-world countries can measure the competence and dedication of their teachers, a developed country like the United States can figure out how to do the same. It should be easier, not harder, with modern computers, databases, and a well-funded bureaucracy. (By the way, even notice the tension between "You can't measure teacher quality" and "You have to have this certificate to be a qualified teacher in this state"? One wants to ask, "How did you figure out the things you need to know to be a qualified teacher? Did you somehow measure teacher quality and figure out what characteristics drive it? Or did you just make it up?")
It's possible that I may one day sign off on some fund or reserve estimate that is fundamentally unsound for reasons beyond my comprehension. Or I may build a predictive model that isn't truly predictive; it fails for reasons I can't anticipate. I hope these events, if truly isolated, wouldn't lead to the end of my career. But if someone uncovers even a single act of gross negligence, deliberate deception, or a pattern of repeated failures due to incompetence, I hope I would be punished for it. Most people want some level of scrutiny and responsibility for outcomes, so they can be rewarded for doing a good job. Denying any responsibility for outcomes looks suspicious, to say the least.
I'm not quite sure why other professions act so clannish. Maybe they are public or semi-public employees and can thus afford to completely insulate themselves from the scrutiny of their customers? Maybe because they are powerful and sympathetic interest groups, and they form constituencies that politicians cater to? Certainly, McDonald's, Walmart, Ford, and Merck can't afford to say, "We're not responsible for quality" or "We're not responsible when things go wrong." McDonald's and Walmart have to anticipate the worst behavior of the very worst 1-in-a-million employee, at the very bottom of the distribution, on his worst day of the year and say, "We're going to see this happen once in a while. We've got to try to prevent it, or at least own it when it happens." Ford and Merck have to anticipate the 1-in-a-million dumbest use of their product by the very dumbest user on his worst day, and then own the problem that isn't really their fault. They have to make the product safe, not just for the average user, but for the very worst leftmost-tail-of-the-distribution user. If you have a government job (teachers, police) or are paid on a government-set payment formula (hospitals and clinics), your best bet is to sit tight and argue that problems aren't your fault. I think that explains at least some of the clannishness.
The comparison between unethical or illegal behavior and performance measures, let alone outcome measures, is misguided. What does being a poor performer have to do with being someone who doesn't follow the rules?
ReplyDeleteAlso, when most people claim that "performance is impossible to measure," they usually seem to mean something more like "performance is impossible to measure in a way that doesn’t result in large amounts of exogenous variation at the individual level for which it is unfair to hold those individuals [doctors, teachers, etc.] accountable."
Anyway, what do the professions that you’re talking about have in common? They’re all street-level bureaucrats. They deal directly with the public, they have sizable discretion over their work, and their decisions are an important intermediary between how policy is made and how policy is implemented.
(This also means that when policy is self-contradictory, vague, or impossible to implement given available resources, it’s in the work of street-level bureaucrats where this becomes apparent and must be covered for.)
Regarding your first paragraph, both are things that certain professions get tribal about, which is the topic of my post.
ReplyDeleteRegarding your second, I discuss this in a paragraph in my post. I expressed incredulity at the idea that you can't measure performance because some outcomes have an element of random chance to them.
Third paragraph: Interesting, I hadn’t thought of that. But I’m guessing other professions that don’t deal with the public involve a lot of discretion, too. My job certainly does. I’m not sure this feature explains clannishness/defensiveness about scrutiny.