Shit is going to get really bad when a ton of medical professionals have no idea how to do major parts of their jobs because they got ChatGPT to do it. I guess that we’ll have to wait and see what happens.
I find it unlikely that this will happen for roles like doctors and nurses. There are large practical components of training, if they didn’t have the basic knowledge needed it would show through pretty quickly.
Our board exams can only cover so much, so there are little things that can slip under the radar. Like I said in another comment, one of my classmates in medical school used Chat GPT to summarize the reading and it swapped the warning signs for 2 different neurological conditions, one of which is transient and can be fixed with medications, the other is one that can be lethal if not recognized quickly.
Residency training will weed some of them out, but if they never see/recognize those zebras until they show up on the autopsy, that patient still suffered for their laziness and cavalier attitude towards their education.
Doctors spend months or years being supervised. If a doctor cheated on one test then maybe it would slip through, but I see this as no different to just forgetting some part of some learning from years ago, which surely happens.
If a doctor cheated on every exam, their supervisor is going to notice really quickly.
I think we might overestimate how qualified a junior doctor is after doing all the exams. This article (from 2009, well before LLMs) says junior doctors screw up in 8% of prescriptions they write, with half of the mistakes “potentially significant”. This is after any chance at having a supervising doctor review. It says pharmacists generally save the day by spotting the errors.
I also found local numbers showing about 16% of junior doctors never make it through training (the article is saying it’s actually 40%, but 16% is their “normal”). That will include burnout and other reasons for not continuing, but I’m pretty sure with such a decent proportion of people dropping out you can expect the ones that haven’t taken in enough understanding despite passing their exams are commonly dropping out as part of that group, and though LLMs may have increased the pool I doubt we can assume these people make it through training without learning what they need to know. Becoming a doctor is just so intense that it doesn’t seem likely.
As has been pointed out by someone else, our concern should probably lie in those that pass exams then go on to do medical (or other) roles without any supervision period.
Part of my concern is that APPs like nurse practitioners that have no supervised practice as part of their training are going to become even more poorly educated. Their curriculum is already algorithm-based, and because of the Nursing lobby pushing for more and more independence for NP’s, they have dwindling physician oversight requirements (in some places a physician only needs to audit 10% of their notes and never actually lay eyes on the patient themselves.)
As a patient, you do have the right to refuse to be treated by anyone. You may have to wait for a physician to be available, but no one can treat you without your consent and you can always ask for a provider’s title and licensure.
Ah this is a different risk than I thought was being implied.
This is saying if a doctor relies on AI to assess results, they lose their skill in finding them by themselves.
Honestly this could go either way. Maybe it’s bad, but if machine learning can outperform doctors, then it could just be a “you won’t be carrying a calculator around with you your whole life” type situation.
ETA: there’s a book Noise: A flaw in human judgement, that details how whenever you have human judgement you have a wide range of results for the same thing, and generally this is bad. If machine learning is more consistent, the standard of care is likely to rise on average.
Myself and my classmates (respiratory therapy) use AI to put together study guides, flash cards, and practice tests for us, using our lecture recordings, notes, and PowerPoints as reference material. It hasn’t hallucinated anything incorrect into our study guides. I’m no fan of LLMs and the like by any means but it’s been a huge time saver in this capacity. Less time spent formatting bullshit means more time for studying
Medical student here. Some of my classmates did the same thing with summaries and study guides and it scrambled a couple of fine but extremely important details. The mistake meant that my classmate mixed up two presentations of neurological problems, one of which is transient and fixable with medications and the other is something that can rapidly become lethal if not recognized fast enough.
RT’s are precious resources for physicians, but the stakes for us fucking up are profoundly higher. (And if the RT does something wrong and the patient suffers harm, it’s still likely to land on the physician to some extent in terms of liability.)
I get that, but to a degree it’s also on the student for not verifying the output either. One of my classmates has dyslexia (he does the flashcard sets) and makes frequent errors. Thankfully our class shares the burden of making study materials because we all act as a filter of sorts for him. Helping him notice the errors before he commits them to memory, and allowing us to have them edited with correct information. Same goes for AI stuff, you gotta double check it. Editing a few lines is still a lot quicker than creating these resources from scratch
The problem is that most people don’t double check or they check a couple things then think “good enough”, and turn off the critical thinking part of their brain. That’s how lawyers ended up submitting a case brief with fake case citations. The “citations” look real enough, but to verify it, you have to go read the source yourself.
This goes for people citing studies without reading them first. There are a lot of studies that squidge the numbers around to make things look better and you have to look for things like how they parsed the data for the results and conclusions. I’ve personally made pharma reps very uncomfortable by digging into things like how they did or did not parse complications by sex (ie one complication was parsed by sex, but the other was combined)
I may only be in a respiratory therapy program, but I’ve been an EMT for 10 years prior to that. If that experience is worth anything, I’d say verifying information before making a clinical decision is a far more important habit to build than memorizing two obscure values for a test (that you’ll almost certainly forget by the time you’re a licensed physician).
An AI study guide is liable to make mistakes, but the bigger problem here is a prospective physician who can’t be bothered to make sure that they had the correct information before acting on it. Ditto for the lawyers or researchers relying on AI to do the work for them (an inappropriate use of AI imo). Throwing a practice test together and drafting legal paperwork/writing an academic piece are planets apart
The AI alleviates the process of critical thinking though. I make my own review notebooks for my boards and for clinical rotations by taking the time to figure out what’s important and what I don’t know to put those things in my notebooks. I write these out by hand on paper, so I have to be judicious about what is going to actually be important, and just the process of making those priorities helps me to have a better understanding of my own deficiencies.
Making a good study guide requires critical thinking skills, and if that gets outsourced to AI, that means the critical thinking isn’t being done by the human that needs to learn that skill.
I feel like people are overly antagonistic towards the technology, when the ire really should be directed at the companies. The tech has problems, like all tech does, but it also has its uses. Saw someone earlier today that had created a newsfeed with headlines rewritten to not be clickbaity bullshit.
Shit is going to get really bad when a ton of medical professionals have no idea how to do major parts of their jobs because they got ChatGPT to do it. I guess that we’ll have to wait and see what happens.
it’s already been that bad, we just have chatgpt to blame now
I find it unlikely that this will happen for roles like doctors and nurses. There are large practical components of training, if they didn’t have the basic knowledge needed it would show through pretty quickly.
Our board exams can only cover so much, so there are little things that can slip under the radar. Like I said in another comment, one of my classmates in medical school used Chat GPT to summarize the reading and it swapped the warning signs for 2 different neurological conditions, one of which is transient and can be fixed with medications, the other is one that can be lethal if not recognized quickly.
Residency training will weed some of them out, but if they never see/recognize those zebras until they show up on the autopsy, that patient still suffered for their laziness and cavalier attitude towards their education.
Doctors spend months or years being supervised. If a doctor cheated on one test then maybe it would slip through, but I see this as no different to just forgetting some part of some learning from years ago, which surely happens.
If a doctor cheated on every exam, their supervisor is going to notice really quickly.
But once they get to be supervised, it’s “too late to fail them” (/cynic)
I think we might overestimate how qualified a junior doctor is after doing all the exams. This article (from 2009, well before LLMs) says junior doctors screw up in 8% of prescriptions they write, with half of the mistakes “potentially significant”. This is after any chance at having a supervising doctor review. It says pharmacists generally save the day by spotting the errors.
I also found local numbers showing about 16% of junior doctors never make it through training (the article is saying it’s actually 40%, but 16% is their “normal”). That will include burnout and other reasons for not continuing, but I’m pretty sure with such a decent proportion of people dropping out you can expect the ones that haven’t taken in enough understanding despite passing their exams are commonly dropping out as part of that group, and though LLMs may have increased the pool I doubt we can assume these people make it through training without learning what they need to know. Becoming a doctor is just so intense that it doesn’t seem likely.
As has been pointed out by someone else, our concern should probably lie in those that pass exams then go on to do medical (or other) roles without any supervision period.
Part of my concern is that APPs like nurse practitioners that have no supervised practice as part of their training are going to become even more poorly educated. Their curriculum is already algorithm-based, and because of the Nursing lobby pushing for more and more independence for NP’s, they have dwindling physician oversight requirements (in some places a physician only needs to audit 10% of their notes and never actually lay eyes on the patient themselves.)
These Nurse Practitioners are presumably already required to be highly skilled nurses? Please tell me that’s true 😑
Nope. They can (and these days often do) go straight from their nursing degree to an NP program with no real work experience.
Oh great. Just what I wanted to hear.
As a patient, you do have the right to refuse to be treated by anyone. You may have to wait for a physician to be available, but no one can treat you without your consent and you can always ask for a provider’s title and licensure.
It already is happening:
https://www.thelancet.com/journals/langas/article/PIIS2468-1253(25)00133-5/abstract
Ah this is a different risk than I thought was being implied.
This is saying if a doctor relies on AI to assess results, they lose their skill in finding them by themselves.
Honestly this could go either way. Maybe it’s bad, but if machine learning can outperform doctors, then it could just be a “you won’t be carrying a calculator around with you your whole life” type situation.
ETA: there’s a book Noise: A flaw in human judgement, that details how whenever you have human judgement you have a wide range of results for the same thing, and generally this is bad. If machine learning is more consistent, the standard of care is likely to rise on average.
Machine Learning is not LLM.
It’s not but the linked paper I responded to doesn’t mention LLMs?
The thread is about ChatGPT, which is an LLM bot, hence the confusion?
Myself and my classmates (respiratory therapy) use AI to put together study guides, flash cards, and practice tests for us, using our lecture recordings, notes, and PowerPoints as reference material. It hasn’t hallucinated anything incorrect into our study guides. I’m no fan of LLMs and the like by any means but it’s been a huge time saver in this capacity. Less time spent formatting bullshit means more time for studying
Medical student here. Some of my classmates did the same thing with summaries and study guides and it scrambled a couple of fine but extremely important details. The mistake meant that my classmate mixed up two presentations of neurological problems, one of which is transient and fixable with medications and the other is something that can rapidly become lethal if not recognized fast enough.
RT’s are precious resources for physicians, but the stakes for us fucking up are profoundly higher. (And if the RT does something wrong and the patient suffers harm, it’s still likely to land on the physician to some extent in terms of liability.)
I get that, but to a degree it’s also on the student for not verifying the output either. One of my classmates has dyslexia (he does the flashcard sets) and makes frequent errors. Thankfully our class shares the burden of making study materials because we all act as a filter of sorts for him. Helping him notice the errors before he commits them to memory, and allowing us to have them edited with correct information. Same goes for AI stuff, you gotta double check it. Editing a few lines is still a lot quicker than creating these resources from scratch
The problem is that most people don’t double check or they check a couple things then think “good enough”, and turn off the critical thinking part of their brain. That’s how lawyers ended up submitting a case brief with fake case citations. The “citations” look real enough, but to verify it, you have to go read the source yourself.
This goes for people citing studies without reading them first. There are a lot of studies that squidge the numbers around to make things look better and you have to look for things like how they parsed the data for the results and conclusions. I’ve personally made pharma reps very uncomfortable by digging into things like how they did or did not parse complications by sex (ie one complication was parsed by sex, but the other was combined)
I may only be in a respiratory therapy program, but I’ve been an EMT for 10 years prior to that. If that experience is worth anything, I’d say verifying information before making a clinical decision is a far more important habit to build than memorizing two obscure values for a test (that you’ll almost certainly forget by the time you’re a licensed physician).
An AI study guide is liable to make mistakes, but the bigger problem here is a prospective physician who can’t be bothered to make sure that they had the correct information before acting on it. Ditto for the lawyers or researchers relying on AI to do the work for them (an inappropriate use of AI imo). Throwing a practice test together and drafting legal paperwork/writing an academic piece are planets apart
The AI alleviates the process of critical thinking though. I make my own review notebooks for my boards and for clinical rotations by taking the time to figure out what’s important and what I don’t know to put those things in my notebooks. I write these out by hand on paper, so I have to be judicious about what is going to actually be important, and just the process of making those priorities helps me to have a better understanding of my own deficiencies.
Making a good study guide requires critical thinking skills, and if that gets outsourced to AI, that means the critical thinking isn’t being done by the human that needs to learn that skill.
I feel like people are overly antagonistic towards the technology, when the ire really should be directed at the companies. The tech has problems, like all tech does, but it also has its uses. Saw someone earlier today that had created a newsfeed with headlines rewritten to not be clickbaity bullshit.
Why did you post this message 4 times?
Some clients do this. No idea why. Have had it happen myself.