医療現場の AI 活用と法的責任、専門家が議論

*AI tools are already being used in hospitals, clinics and doctors’ offices. Who is making treatment decisions, and who is responsible if an AI gets it wrong? To address these questions, Science and Technology editor Eric Smalley hosted a webinar with panelists Dr. Jodyn Platt, an Associate Professor Health Management and Policy at the University of Michigan, and Dr. David Kao, Medical Director at the Colorado Center for Personalized Medicine.*

*Platt’s work explores how data and technology can be used responsibly to improve health while earning and sustaining public trust. Kao’s research is focused on using big data analysis to personalize management of heart disease, most notably heart failure.*

Eric Smalley: What are the main types of AI that you see being used in healthcare?

David Kao: There’s a temptation to think of AI in healthcare in sort of a binary way: You’re either doing it like the old country doctor or like Skynet from Terminator where it’s automatically making decisions about living or dying for all these people. And the truth is it’s a continuum.

In Colorado, we focus on using AI to augment clinical decision-making. There are machine learning-based algorithms that will monitor the electronic health record continuously, and if someone starts to look worse, they’ll alert the team. They’re not decisions in and of themselves. They’re just better information on which to make a decision.

Jodyn Platt: Before we had large language models and generative AI, we used AI to make predictions about disease prognosis. This gives doctors clinical decision support by helping us make decisions based on patient data.

If you’ve been to see a doctor that uses a scribe that helps them write clinical notes, that’s a very recent example of AI that’s using a large language model. Whenever you engage in a chat bot around healthcare, that would also be another example of a large language model that’s underlying that tool.

The third type of AI that I’ve recently started talking about is the “hype”, the AI that promises great and amazing things or promises that it may destroy the world. It creates a fear of missing out, and it’s very based on feeling and not a lot of fact.

ES: Are folks required to consent when doctors use AI tools?

JP: Every state is different in terms of third-party recording and consent things like that, so consent may differ depending on where you are. So sometimes nobody will tell you, or increasingly consent is incorporated into your general consent to treatment.

A clinician can turn it on and off on their own volition also if they feel like they want to turn it off, they can. So asking if AI is being used at every visit is reasonable if you want to know for sure.

DK: It’s a hard proposition. If you think about imaging studies like an MRI, those may be processed offsite by a third-party vendor. But we don’t ask for consent to send the images to be processed externally. The problem is where is that line? You know, at what point do you start to need consent?

I’m not saying that that’s how it should stay, but that is kind of where the vagaries are because the fact of the matter is a lot of people’s data is going to a lot of business partners and has been for decades. That information is far more intimate than anything you would give an AI model right now, like truly.

ES: How can patients be assured of some degree of data privacy?

JP: Health systems are going to engage in relationships with third-party vendors that are compliant in terms of HIPAA and all the data privacy rules that you would expect. The question is whether you feel comfortable with whatever company folks are working with.

I think one of the bigger issues, if data privacy is important to you, is sharing your information outside of the healthcare system. It can take a long time to see a doctor, or there may not be a doctor. If you feed medical information to a ChatGPT or some external LLM system to help you understand it, your data is not protected.

ES: The FDA regulates medical devices along with drugs, but who regulates an AI that’s used in patient care?

DK: States have made their own regulations for AI in patient care which makes it super confusing because they don’t have any other medical regulatory capacity. Ultimately, the hope from an industry perspective is for a national standard or process to prove efficacy and safety just as we do with medications. It’ll probably be regulated by the Center for Devices and Radiological Health, the FDA’s arm of device regulation.

The difficulty is that evaluating AI is different. If a medication was different every time you took it, how do you say whether it’s safe? The established ways we think about that don’t really apply.

JP: I think what clinicians hear is that it’s their responsibility when a decision is made and AI is in the mix. So the clinic note is still signed by a person. Any email that’s assisted by AI is also signed by a real person, and they’re accountable for what’s in those messages.

I would say that’s the only way we can use it really at all right now. The regulatory process doesn’t exist, so there has to be a human-in-the-loop component to it.

ES: What are your concerns or thoughts about people deciding to treat themselves with AI?

DK: I think it’s unwise, but I know people are going to do it.

We talk about needing human-in-the-loop for health and AI, right? In this case, it’s the patient who’s the human in the loop, who is probably not a great judge of whether the recommendations are appropriate or not. It’s sort of like treating yourself, which almost never goes well.

ES: How can we ensure that this technology is developed and deployed in the best interest of the public as well as the individual?

JP: I recently learned about a framework that was developed by a researcher at the University of Pittsburgh, Miranda Yaver, who thinks about this as an equity question.

The big actors create part of the system. But she argues that the public needs to know that they’re also part of it. We have to know how to navigate the system and need to have the resources to be able to do that. The barrier to entry is high enough that we don’t usually learn to do it.

So I think finding ways that patients and the public can have that continuous access to be able to exercise choices and have their data and information protected in ways that they want is critical as we’re developing the system.