A Better Way To Do Drug Approval: Continuously Updating Reports

My (admittedly limited) understanding of the FDA's drug approval process is that it waits for a clinical trial to be complete, reads the drug company's study, then says "Yea" or "Nay." This is absurdly inefficient and no doubt leads to thousands of deaths and untold suffering. The bureaucratic delay of life-saving medicine is an atrocity, and the FDA's foot-dragging approval process for Sars-Cov-2 vaccines is a particularly stark example. A more reasonable approach to drug approval would be to have the company submit a continuously updating document that the FDA can monitor on a periodic basis. Better yet, the FDA can pre-specify an approval threshold, so the pharmaceutical company can anticipate if and when approval will be likely. Once a "statistically significant" drug effect has been demonstrated, the FDA can start ramping up production and recouping it's investment by selling the new drug. No need to wait for the clinical trial to play out to completion, though of course such trials should be completed and the results published. More time means more data and (hopefully) more certainty of a drug's effectiveness. In the meantime, some patients can have the benefit of a new drug that improves quality of life, or extends it. 

A while ago, I read the FDA briefing document for the Pfizer vaccine.  I'm fairly sure it was this document. My wife had printed off a copy, which I read and marked up with notes. As a front-line health worker, she'd be one of the first in line to get it. It was being discussed whether immediate family (my kids and I) would be near the front of the line, so we wanted to educate ourselves on what was known about the vaccine. (It turns out the kids and I will have to wait, like all the other normies. In my opinion, this is perfectly sensible as a triaging strategy. I'm happy to see older and more vulnerable individuals get the vaccine first.) 

I was struck by the repetitiveness of the document. It seems as though the same information is repeated in different sections, as if the authors were trying to conform to a template set by the FDA? (Can anyone confirm?) 

The other thing that struck me was the figure below. And when I say it "struck" me, I mean it filled me with a cold rage, and it still does:

The red line is the cumulative case count for the unvaccinated control group, and the blue line is the vaccinated group. It was evident by week 3 that there was a difference between the two groups, and by week 4 or 5 it was definitive. And yet we had to wait for Pfizer to complete it's proposed trial, then wait on the FDA to look it over and stamp it "approved." Now I know there is a great deal of paranoia about spurious statistical results, replication crisis and all. That really is a serious problem which casts doubt on a lot of published research, including research on the efficacy of new drugs. But I think people have been blinded by their zeal for academic rigor. There is this sense in the current zeitgeist (at least in academia) that a result isn't real, or is highly suspect, unless it's exactly what you set out to test for, using the exactly specified methodology. No doubt, pre-committing to a methodology prevents "p-hacking", where you run hundreds of tests on your data until, by sheer chance, one of them gets the result you want. With all of that duly acknowledged, sometimes a result is so strong you don't need to doubt it just because "It's not exactly what we set out to test." I think this is one such case. 

Here's my proposal. The FDA briefing doesn't have to be submitted fully formed. Pfizer doesn't need to wait until all the data is in, then take whatever time is needed to massage all the data into a narrative with various charts and graphs. They can build a document that runs and updates every day, pointing to a database that also updates daily (or weekly or whatever periodicity makes sense). It seems like there are all these choke-points in a process that should be continuous. In today's broken world, the FDA really can defend itself by saying "We couldn't have made a decision until December because the results weren't in yet," and Pfizer can defend itself by saying "We couldn't have moved any faster, because we are just reporting our results according to the FDA's regulatory structure." They need to come together and build a process that isn't held up by waiting for some kind of hand-off.  Pfizer should have been submitting a daily-updating report of its cumulative findings, and someone (or some dozens of people, given the importance of timeliness) should have been reviewing it on a regular basis. They would have discovered the diverging blue and red lines in the graph above much sooner. It would have been thoroughly obvious by, say, day 50 that the vaccine was effective, at which point it would have been unethical to delay approval any longer. The x-axis goes out to 119 days. We could have been at least two months ahead of where we are in terms of ramping up vaccine production and getting shots into arms. Instead, we're standing where we are today. Quite possibly, we could have beaten back the surge that started in November and saved hundreds of thousands of lives. Inexcusable bureaucratic foot-dragging has killed countless people, over a thousand a day since November (and peaking at above 3,000 daily). 

(A couple of examples of what I mean by "continuously updating report." In my job as an actuary, I use RMarkdown to build reports. Python users will be more familiar with Jupyter notebooks. I'm sure there are dozens of analogues that work with other programming languages and suites of statistical software. Basically you write some code (R, Python, or whatever you like) interspersed with narrative text. A human can write in the narrative and change that as needed, but the numbers and table themselves are updated by re-running the code. They will update as the data tables they're pointing to get updated.)

The FDA can pre-specify some kind of automated approval threshold. The statistical significance of the vaccines effectiveness became clearer and clearer as the days dragged on and the red and blue lines diverged. Instead of insisting on a pre-registered clinical trial fully playing out and taking weeks to review the resulting study, the FDA could simply say, "The vaccine is approved as soon as it passes the following statistical test..." Pfizer dutifully updates its report on a daily basis and can start selling doses of the vaccine the day it passes the statistical test. In addition to speedier approval, it would have the advantage of allowing Pfizer to anticipate the approximate timing of approval and start ramping up production. Pfizer is unlikely to pre-commit millions (billions?) of dollars to production line infrastructure if it is dependent on the whims of an arbitrary bureaucracy to give it the green light. On the other hand, if there is a pre-designated threshold, Pfizer can anticipate the likelihood and timing of approval.

There are narrative parts of the document that can't simply be updated automatically. For example, there is a section on Bell's palsy occurring in several of the treated individuals (which was probably not caused by the vaccine). Someone would have to notice this phenomenon and write it into the text of the document. But that's easily doable; some sections can robotically update based on incoming data and others can be updated "by hand" as the facts on the ground become known. 

If the FDA is already doing something like this, feel free to tell me I'm way off base. Whatever they are doing, even if it does involve periodic review of ongoing trials, it's still absurdly slow. Even the prospect of saving hundreds of thousands of lives hasn't spurred them to speedier action. Tyler Cowen and Alex Tabarrok at MarginalRevolution have been great on this point. They have been absolutely hammering the FDA for its intransigence. There is a species of "public intellectual" that has been standing athwart this push to approve vaccines faster. These individuals are making very poor quality arguments and not taking seriously the thousands of lives a day that are being lost. (I'm sure you'll find a good sampling of what I'm talking about if you just read the comments of a few of Tyler or Alex's blog posts.) I don't know what's in their heads. It must be something to the tune of "We need to preserve standards for pharmaceutical approvals, or else we risk a rash of approvals for harmful drugs." Or "It is, from a deontological standpoint, unethical to give medicine that's not thoroughly tested using the most rigorous standards of evidence." Okay, I hear all that, but...at the cost of thousands of lives a day? These people should tell us how many corpses they're willing to pile up at the altar of "rigorous approval standards." Is there even a limit? Do they think that the FDA's slow-moving approval process is, in an expected value sense, actually saving lives? No. There is something deeper going on here. The real issue is that this is a regulatory failure. It's a failure that has resulted from a dysfunctional bureaucracy running on auto-pilot. And some shallow intellectuals won't be caught dead on the side of those icky libertarians who have been saying so for decades. They lack the intellectual tools to even recognize the root cause of the problem, nor do they possess the moral vocabulary to denounce it as outrageous. 

_______________________________________________

Under a pre-registered approval threshold, could a drug cross the threshold, then "cross back"? Maybe some cancer drug gets approved according to the automated trigger, then the next day a couple of the treated patients die and it's back in "statistically insignificant" territory. Sure, but in most cases the company would still be able to anticipate which way things were going, whether an "approval" status is stable or still uncertain, and make a reasonable decision on how to proceed. And the FDA can provide clarity on how to handle such borderline cases.