There is a long tradition of regulation in personal lines insurance. The state department of insurance, or DOI, has the authority to approve or disallow the insurer's rates, depending on statutory authority, although sometimes they go well beyond their statutory authority in restricting how insurers can rate. I wrote about the process here and here. The insurer must file a "rate filing" with the DOI whenever they wish to change their rates, and all rate changes require some kind of actuarial justification. Insurers set the overall price, as in the amount they need to cover total expenses and claims and earn a (usually slim) profit margin, and also the relative price of their various customers, as in the rate differential between 16-year-olds and 40-year-olds or the difference between people who have recently had accidents and people who haven't.
Traditionally the justification is based on expected cost. Setting the overall rate level depends on how well historical premiums have covered claims and expenses. A calculation based on these numbers, along with some sensible adjustments (trending costs for inflation, adjusting historical premiums to the current rate level, etc.) will tell the insurer, for example, "We need to increase rates in Illinois by 3% this year." (This calculation that yields the indicated rate increase is often simply called "the indication.") The relative prices between insurance customers is usually determined by some kind of predictive model. "My generalized linear model tells me that 16-year-olds are 3 times as likely to have an accident as my 40-year-olds. Multiply the base premium by a factor of 3 to get the 16-year-old's rate. Repeat for all rating characteristics..." There is often controversy about the use of certain rating characteristics. Some people think it's unfair to use credit history in insurance pricing, even though from a purely predictive standpoint it is a highly significant predictor of future claims. Certainly it's not allowable to use race anywhere in rating, although there is some discussion of whether some rating variables are a proxy for race. Zip code correlates with race, for example, so some people argue that insurers are sneakily pricing for race without admitting they're doing so, using their territorial rates as a proxy. Some people make the same argument about credit: credit correlates with race, with some races having generally poorer credit scores compared to whites. I happen to think this is wrong; allowing credit-based and location-based pricing means you can identify good and bad risks regardless of their race. In other words, it allows insurers to, say, write a lot of business in predominantly black zip codes because credit history allows them to identify the good risks in those zip codes. They'll even write the bad risks in those zip codes if they can determine the right price for them, and credit history makes this a lot easier to do. Absent credit, the same insurer might avoid that zip code by not placing an agent there or not directing its marketing activity there. (States can regulate the pricing, but there's no way they can tell them "You must place an agent in this zip code, and you must direct a marketing campaign to this one." I don't think there's any version of this mandate that would pass constitutional muster.) All this controversy aside, most insurers have a relatively free hand in using credit history and location (zip code-based or otherwise) to set prices, so long as they can show that their rates correlate strongly with actual claims.
Price Discrimination and Price Optimization
Enter price discrimination. Price discrimination is the practice of charging two otherwise identical people different prices based on their willingness to pay, their "elasticity of demand." That is, I'm going to offer a price lower than my base price to that guy, because otherwise he won't buy what I'm selling, even though I already get a lot of customers at the base price. Unfortunately, most people describe this practice using the converse and feel moral indignation at the thought of ever getting charged more than the lowest possible price. As in, "I'm going to charge you more, because you are willing to pay more for this service." Both framings are technically accurate, but the second is so fraught with emotional baggage that I prefer to avoid it. Price discrimination actually lowers overall costs for customers as a whole. If I can attract more customers by offering different prices, that means I have a bigger customer base over which to distribute my fixed costs. The overall price level is lower, even though in some particular cases some individuals might be paying more than they would in a world of flat prices. That person who is seething with indignation over being denied a discount is probably paying a lower price than he would if that discretionary discount didn't exist.
In property and casualty insurance (P&C, meaning home and auto in this context), price discrimination is a hot topic. It's called Price Optimization (although the two terms aren't exactly synonymous; more on that below). To many regulators it's a big no-no. Because pricing is so heavily based on traditional actuarial methods, the language of statutes and regulations typically reference "loss costs" and expenses and "actuarially sound rates" (which implicitly means something that is cost-based and not demand-based). Many regulators are actuaries, and they are relying on the language of actuarial standards of practice for guidance. In this sense, actuarial accrediting societies (and I'm a member of one) can be regulators by proxy. If the standards of practice only ever reference "loss cost" and never mention propensity to buy ("elasticity of demand"), then regulators relying on these standards of practice will not allow for it. Indeed, under a strict reading, the relevant standards don't allow for price discrimination. When the CAS (Casualty Actuarial Society) tried to rewrite a standard of practice to allow for rating based on considerations other than loss costs, some prickly "watchdog" groups complained loudly. (The CAS was in a tough spot here. On the one hand, they didn't want to be in a position to say price optimization is contrary to actuarial principles, such that actuaries using it should be sanctioned for malpractice. On the other hand, they didn't want to be the ones to green-light price optimization everywhere. I suppose this is one of the hazards of being a guild; you are sometimes the de facto regulator of an industry and it falls on you to make difficult decisions.) States differ in how strict they are, but some states have statutes that explicitly forbid price optimization and others have regulators who assume it is implicitly forbidden by traditional standards of practice (perhaps interacting with existing regulation, which might reference "actuarially sound rates" or "unfairly discriminatory rates" from those standards).
By the way, I understand why people don't like price optimization. I hinted at this above: people hate the feeling that they aren't getting the lowest possible price for something. If I explained price optimization to the average insurance customer, I'm sure they'd balk. So regulators and "watchdog" groups are responding to the impulses of typical insurance customers. (Scare quotes around "watchdog" because many of these agencies act in ways that are contrary to the interests of consumers, as is the case here.) Don't get the impression that I'm some ideologically blinkered libertarian saying, "Gee, why would anyone ever want to regulate markets?" Or some antisocial economist saying, "Gee, why wouldn't consumers want a perfectly efficient market?" Or some morally compromised data scientist saying, "Gee, why don't consumers appreciate the beauty of my glorious pricing model?" I get it. My response is, What do consumers know anyway? Consumers balk at all kinds of commercial activity, even though economists can usually come up with "efficiency" justifications for those behaviors. In fact, economists often conclude that we'd be much worse off if those unpopular practices were outlawed. (We'd be far worse off if the government banned something every time a consumer felt indignant; read Defending the Undefendable by Walter Block for a long list of legitimate business practices that average people get indignant about.) If "efficiency" sounds bloodless, bear in mind that it usually means lower overall costs for consumers and more products available. Just so with price optimization.
Price Optimization On the Overall Rate Level
Let me start by defending the practice of measuring demand elasticity, which basically means the propensity of a customer to leave one insurer for another based on the magnitude of a price change. Suppose I'm the actuary in charge of rates in the state of Illinois. I do some actuarial calculations and determine that prices need to rise by 10%. I have some rating software that re-rates my book of business (meaning the full set of our insurance customers) at the new, higher rate level. I proudly report that this rate change will increase our Illinois revenue by 10%. Except this is wrong. It assumes that we retain 100% of our customers after the rate increase. I should know how much premium we're actually going to get if I increase rates by 10%, accounting for the propensity of policyholders to shop for insurance elsewhere. In fact, I have a duty to upper management, to the shareholders, and ultimately the customers (who are relying on the company remaining solvent) to accurately estimate the effect on revenues. At the very least, I should calculate elasticity so I can calculate the effect of a rate change on customer retention, which will give me a more accurate estimate of the effect on revenue. (I'm sort of using "premium" and "revenue" interchangeably here, btw, though insurers get revenue from sources other than their customers' premiums.) I don't want to say, "We're increasing premiums by 10%" when it's within my power to provide a better estimate. Suppose I say we're taking 10% but it's only 5% when factoring in retention effects. That's bad. It makes it harder for a company to plan for the long-term. Insurance companies need to be making these decisions with their eyes wide open, not making absurdly unrealistic assumptions, assumptions that can easily be relaxed with a moderately complex calculation. Actuaries are the guardians of capital at insurance companies. We're supposed to analyze risk and safeguard the billions of dollars of stockholder capital. We're supposed to ensure there is enough money held in reserve to pay policyholder claims for the indefinite future. We're supposed to do these kinds of estimates. And we're supposed to make them as accurate as is feasible.
This is where it gets sticky. Suppose my retention calculation affects the company's decision about how much to increase rates. An executive who sees that a 10% rate increase only leads to a 5% premium increase might say, "Okay, let's only take an 8%. Show me what that looks like." Or it could go the other way. Maybe my Illinois customers are relatively inelastic, and I could take a 15% rate increase and get pretty close to the 15%. The executive might use this information to take a rate increase that's larger than what's actuarially justified. In practice this is usually limited by the historical data and the actuarial methods. Actuaries have to calculate an indicated rate increase (once again, "the indication"), and DOIs usually don't allow you to go above them. (There is some amount of play here; maybe I can make a few adjustments and turn a 10% into an 11%. But I can't make the indication go arbitrarily high, and even eking out more than a couple of percentage points is unlikely.) But they don't mind you going below the indication. This question of "How far below my indicated rate increase can I deviate?" is where price optimization comes in. Traditionally insurers use rules of thumb to make these kinds of decisions. The indicated rate increase, based entirely on actuarial calculations straightforwardly applied to historical premium, loss, and expense data, is often higher than what's actually reasonable. That executive might say, "Hmm, a 10% increase is too high and will cause a lot of disruption in our book. Let's take 5% instead." ("Disruption" meaning lots of customers non-renewing their insurance policies.) State DOI's are usually accepting of these hand-waving statements about "We're not taking the full indicated rate increase because we're worried about policyholder disruption." But they are very opposed to us doing an explicit calculation to optimize the rate increase. The "rule of thumb" and the explicit calculation are both forms of price optimization, it's just that the former is much cruder. I don't think DOI's should be in the position of saying, "You can do X, as long as you do it crudely and inaccurately. If you get more sophisticated about doing X, we'll punish you." That is essentially the line some DOI's have taken with respect to price optimization.
Price Optimization at the Individual Customer Level
The previous discussion is about the overall rate level. Do I increase rates by the traditionally-indicated 10% or the elasticity-indicated 5%? A more sophisticated version of price optimization involves adjusting the rate for individual policyholders based on their willingness to pay. This basically takes the overall rate effect as a given, but allocates the rate impact based on retention considerations. I can build a predictive model that tells me "This group of customers will leave if I give them a 2% rate increase, but this group of customers won't leave even if I give them a 5% increase. I'm going to allocate more of the rate impact to the less elastic group." (Of course, these are all probabilistic statements. The model doesn't say, "Joey will definitely leave if we increase his rates 10%", but rather something like "Joey's probability of retention will fall from 90% to 80% if I increase his rates by 10%." Optimization is done on the basis of expected values, not "Will he leave? Yes/No?") This practice inspires some unwarranted fears that insurers will identify inelastic groups of people and permanently charge them a high rate. "Hmm. It turns out that soccer moms and rural single men are very price inelastic. Let's just keep increasing their rates every year." That is wildly implausible. The personal lines insurance market is far too competitive for this to actually happen. There are dozens, often hundreds, of insurers in every market. If there are demographics that are systematically overcharged by the industry, someone will come along and specialize in marketing to that group and take all of the customers. (Contra ProPublica and their atrocious article about territorial pricing in auto insurance.)
Here's a more likely scenario for how price optimization would be used at the individual or demographic grouping level. I spelled it out in an earlier post, but I'll repeat the points here. Suppose I build a new predictive model that tells me the price differentials between my various customers. My 16-year-old rate came down from a 200% surcharge to a 150% surcharge. The differential between the worst and best credit individuals used to be a factor of 2, but now it's a factor of 2.5. With dozens or even hundreds of rating variables changing in terms of their indicated surcharge/discount, each individual customer is likely to get something different from the overall rate impact. Maybe the overall rate effect is neutral, 0%, but almost nobody actually gets exactly 0%. If you build a histogram of customer rate impacts, you'd get something normally distributed around 0%, with a few customers getting large premium increases and a few getting large decreases. Well, just like I have a predictive model that tells me the expected costs for each individual insurance customer, I have a model that tells me each customer's elasticity of demand. I can then adjust my surcharges and discounts to optimize something (something other than the error function of my "expected claim costs" model). I can optimize, say, "growth in policy count", or "overall profit", subject to various constraints. (This is why price optimization is not exactly the same thing as price discrimination. Price discrimination simply refers to charging different prices based on willingness to pay. The term "price optimization" in insurance refers to a broad suite of optimization routines, and demand elasticity is simply one of many inputs.) Given a long enough timeline, insurers will ultimately approach their indicated rate differentials. Price optimization simply smooths the path so as to minimize the number of customers who are lost along the way. If my indicated rate for 16-year-olds drops from a 200% to a 150% surcharge, my price optimization routine might say to make this change over the course of three or four years, rather than doing it in one jump. If my surcharge for prior claims jumps from 30% to 50%, my price optimization routine might effectively say, "You're fine to do that in one jump." And it might be because those customers aren't price sensitive and won't leave, or it might be because they are price sensitive but they're also high-cost and we don't want their business anyway. Some other insurer has the right price for them, but maybe we don't. That's a win-win. It seems unlikely that such an optimization routine would in effect say, "You can permanently overcharge married family households with a single teenage driver by 50% over the model predicted premium, because they are just that price inelastic."
Once again, traditionally DOI's have accepted these practices of deviating from the indicated rates based on concerns about disruption.
Regulator: Why is your 16-year-old factor 3.0 when your model says it should be 3.5?That is, they allow us to do so as long as were using crude rules of thumb and not doing an explicit price optimization calculation. Why should we be confined to the cruder version of this calculation? If more sophistication is available, why not allow it?
Insurer: We are moving in the direction of 3.5 with this filing, but due to policyholder disruption considerations we are worried about moving the factor all the way in a single filing.
Regulator: Okay, that makes sense. (Stamps "Approved" on the filing.)
Another crude method of price optimization is rate capping. No single policy's premium will increase by more than, say, 15% in any one year. Clearly this is an attempt to mitigate customer disruption. If I just charged everyone the rates indicated by my new predictive model in a "Let 'er rip" fashion, the customers getting big premium increases would leave. Rate capping smooths the transition to higher rates. Again, price optimization is simply a more sophisticated method of doing something that is already a widely accepted practice.
I should point out here that price discrimination is common in every other industry. Airlines use price discrimination to set ticket prices. They might charge one customer a higher price than another on the same flight because their predictive algorithm says that the first customer is willing to pay more. And, more obviously, ticket prices generally get higher closer to the date of the flight. (Does it go in the opposite direction for flights that don't get filled? As in, "This flight isn't filling up. Let's discount the tickets.") Doctors used to charge different rates to different patients, giving away some free or low-price care to their indigent patients and making it up on their more affluent patients. (I find it interesting that this kind of "privatized redistribution" was once standard practice, but that mandatory health insurance effectively eliminates this "the rich pay a greater share of society's healthcare costs" dynamic.) I like to tell a story about my eyeglasses. The original quoted price was $425, and I must have visibly balked at this price. The sales person then said, "Of course, that's with the anti-reflective coating on the lenses. We can save $150 if you go without." I chose to opt out, thinking this was a useless add-on. They ended up making my glasses with the coating anyway, and still gave me the lower price. I assume they default to making the lenses with the coating and that it doesn't actually cost extra to add it. They just use it as a bargaining chip to win price sensitive customers who bridle at the first quoted price.
Price Discrimination Is Economically Efficient
Price discrimination generally enhances economic efficiency, because it means more customers are served. Companies are identifying price-sensitive customers and trying to attract them by offering discounts. In a flat-price world those customers don't get served, because they say "No" to the single flat price. Granted, these are customers on the edge of indifference between the money and the product. Plausibly they are reaping very little consumer surplus, somewhere close to the difference between the sticker price and the discounted price they are offered. But nonetheless the practice means more production and more served customers, which necessarily implies a greater surplus. In the case of insurance, maybe it's less plausible that price discrimination allows more "production", but it is still welfare enhancing. From the point-of-view of the insurer, they need to collect $X from their customers to cover their costs. Insurers are identifying people who don't mind paying and allocating slightly more of the $X to them, and slightly less to people who would mind paying.
Regulators Should Permit Price Optimization
Regulators ought to allow insurance companies to do sophisticated price optimization. They need to stop treating deviations from the pure risk-based price as something sordid or unethical or necessarily contrary to sound actuarial principles. Some states have passed statutes that explicitly ban price optimization. In those states the regulator's hands are tied. In other states, regulators have simply assumed they have the authority to ban price optimization. They will hold up or disapprove filings that employ these methods. Regulators should stop assuming authority that goes beyond the literal language of their state's statutes. As I hinted at above (and described in detail in a previous post), regulators will often broadly interpret statutory language. Often the law that grants the state the authority to regulate insurance will make reference to "actuarially sound rates" or say that rates shall not be "unfairly discriminatory," and this language often echoes actuarial standards of practice. Unfortunately, some regulators decide that anything they don't like is "unfairly discriminatory." Insurance pricing is discriminatory by its very nature, and it has to be. An insurer must charge higher rates to 16-year-olds and people with poor credit, otherwise they will get only 16-year-olds and customers with poor credit, their claims frequency will explode, their losses will spiral out of control, and they will eventually go insolvent (or perhaps become a niche company that only insures 16-year-olds and other very poor risks, but they would needlessly bleed capital in the process of reaching that equilibrium). If you want to see what insurance without risk-based pricing looks like, look at the disastrous market for health insurance. Or look at Medicare and Social Security, which are prone to shocks from changing demographics. Appropriately priced insurance is necessarily discriminatory, so statutes that reference "unfairly discriminatory" rates leave us at the mercy of a regulator's arbitrary opinion of what's "unfair." I have heard current and former regulators describe disapproving or holding up a rate filing because they just didn't like a new rating variable (e.g. an auto surcharge based on prior homeowners insurance claims), even though they had no explicit authority to ban it. Many of these regulators assume price optimization is banned by default. They push back against attempts to use price optimization because they just don't care for it, even if officially they might cite boilerplate statutory language about "unfarily discriminatory" rates to justify their decisions. Insurers need a free hand to charge appropriate rates and manage their books of business. They need to be able to innovate and make decisions about their idiosyncratic risk portfolios without being held hostage by arbitrary regulators. If a state passes legislation that officially bans a particular rating variable or outlaws differential pricing based on demand elasticity, that's another matter. Of course it's the regulator's job to apply the statutes. But other than that, they should stop hindering innovation in the price optimization space by insisting on strictly risk-based pricing. They should resist knee-jerk consumer reactions that such-and-such a surcharge "seems fishy" or is unfair.
Insurance customers generally have dozens or even hundreds of options. It's basically impossible for an insurer to "overcharge" a customer, because there are always other options. Any customer who bothers to get a few quotes will generally find a lower price than what their current provider is charging. It's quite absurd to worry about nickle-and-dime price differences caused by price optimization. But from the point of view of the insurer, price optimization could mean eking out the tiny margin necessary to keep the company solvent. It could spell the difference between solvency and liquidation, which generally means lay-offs and unpaid claims for policyholders.
