The FDA is a sacred cow in medicine. Thus, it was refreshing when I saw a recent critique of the FDA’s current system for AI regulation published in PLOS: Digital Health. The authors include medical researchers from the Harvard T. H Chan School of Public Health, Beth Israel Deaconess Medical Center, and the UK’s National Institute for Health and Care Excellence. They are a bit shy about how they state it, but their proposal is rather radical — they call for a major curtailment of the FDA’s role in AI regulation and a shift towards decentralized systems for AI oversight.
I think you're missing part of the story here. Insurance companies rely on FDA approvals to decide what "medical devices" they will reimburse. FDA doesn't touch anything that doesn't make specific health claims, so patient facing chatbots come with no guarantees but also can't charge a lot because of this. The top AI companies are fine with the current setup because they'll get the approvals and regulatory barriers will prevent competitors from making any real money.
Strong work here, Dan. It seems like we're at a "damned if you do, damned if you don't" phase for regulating AI in medicine. The last thing we want to do is throw out the baby with the bathwater, and yet we have a strong obligation to prevent horrible things from happening.
I think they're going to happen, and we're going to deal with them and learn from them, but it's going to be really ugly for a few transitional years. I love the idea of having a dedicated "clinical AI" type department, and I do see this happening, but not right away. The interim is going to be very chaotic.
Excellent piece, Dan. I am currently writing a study for ASU law school on how the FDA is approaching AI governance, so this is very helpful. But I was surprised that you didn't spend more time talking about the FDA's important "Predetermined Change Control Plan for AI/ML Enabled Device Software Functions." I see a mention in fn 8, but I am wondering if you have written more about that because I think it represents in important evolution in the agency's approach to these technologies. I am skeptical it will work in the long run (because I can't really see how innovators will be able to accurately predetermine the many potential downstream uses/applications of their AI/ML tools well into the future as the FDA asks) but I think that the agency deserves some credit for at least trying to think outside the box (a bit) compared to their past approach on only technologies. Do you have more thoughts on this? Or perhaps I could interview you on background for my study? -- Adam Thierer
I think you're missing part of the story here. Insurance companies rely on FDA approvals to decide what "medical devices" they will reimburse. FDA doesn't touch anything that doesn't make specific health claims, so patient facing chatbots come with no guarantees but also can't charge a lot because of this. The top AI companies are fine with the current setup because they'll get the approvals and regulatory barriers will prevent competitors from making any real money.
Strong work here, Dan. It seems like we're at a "damned if you do, damned if you don't" phase for regulating AI in medicine. The last thing we want to do is throw out the baby with the bathwater, and yet we have a strong obligation to prevent horrible things from happening.
I think they're going to happen, and we're going to deal with them and learn from them, but it's going to be really ugly for a few transitional years. I love the idea of having a dedicated "clinical AI" type department, and I do see this happening, but not right away. The interim is going to be very chaotic.
Excellent piece, Dan. I am currently writing a study for ASU law school on how the FDA is approaching AI governance, so this is very helpful. But I was surprised that you didn't spend more time talking about the FDA's important "Predetermined Change Control Plan for AI/ML Enabled Device Software Functions." I see a mention in fn 8, but I am wondering if you have written more about that because I think it represents in important evolution in the agency's approach to these technologies. I am skeptical it will work in the long run (because I can't really see how innovators will be able to accurately predetermine the many potential downstream uses/applications of their AI/ML tools well into the future as the FDA asks) but I think that the agency deserves some credit for at least trying to think outside the box (a bit) compared to their past approach on only technologies. Do you have more thoughts on this? Or perhaps I could interview you on background for my study? -- Adam Thierer