Thank you to all who attended our November discussion! We took a look at the following two articles from the 48/11 issue of The Journal of Hand Surgery:
- Do Surgeons Accurately Predict Level of Activity in Patients With Distal Radius Fractures?
- Evaluation of Online Artificial Intelligence-Generated Information on Common Hand Procedures
Our conversation is below! We look forward to having another fruitful discussion in December.
Check out previous #JHSJC transcripts (please scroll to the bottom of the page).
Brent Graham @BgTalkinman:
Welcome to the November #jhsjc ! Remember to use the hashtag in all of you missives. We will be discussing two papers tonight: https://jhandsurg.org/article/S0363-5023(23)00364-7/fulltext
https://jhandsurg.org/article/S0363-5023(23)00414-8/fulltext
Brent Graham @BgTalkinman:
Let’s begin with the article on #AI “Evaluation of Online Artificial Intelligence-Generated Information on Common Hand Procedures” #jhsjc
Brent Graham @BgTalkinmanL:
This is another article on this topic that looked at how #ChatGPT performed in answering questions about common hand sx topics. #jhsjc
Warren @whammert:
There is potential, but CHAT bot needs more training/ data to be accurate enough
Brent Graham @BgTalkinman:
Right now it is essentially trained on all of the pertinent data that is available, which is one of the intriguing aspects – it’s less discriminating than a #CPG but the idea is the sort of the same #jhsjc
RyanC @RPCalfeeSTL:
Certainly AI is a hot topic and likely to work its way into medicine many ways #jhsjc
Warren @whammert:
I think there is potential, but needs more data to be accurate#jhsjc
Brent Graham @BgTalkinman:
The study compared the responses to questions frequently asked by patients about 4 common hand conditions produced by ChatGPT with those provided in educational materials published by the American Society for Surgery of the Hand (ASSH) and posted at the HandCare website. #jhsjc
Brent Graham @BgTalkinman:
The evaluations used the DISCERN score, a validated measure of the quality of pt education materials, and the JAMA Benchmark criteria, a simple screening tool to assess websites for transparency and reliability, including authorship, attribution, disclosure, and currency #jhsjc
RyanC @RPCalfeeSTL:
Good to see that the scores used supported the quality of ASSH materials for patients #jhsjc
Brent Graham @BgTalkinman:
The ChatGPT responses scored lower than the ASSH material on both scales and the answers were also considered too high a reading level for the average American patient. #jhsjc
Brent Graham @BgTalkinman:
The authors also noted that the AI software repeatedly referenced the need to discuss these questions with a surgeon, the importance of shared decision-making and individualized care, and compliance with surgeon treatment recommendations. #jhsjc
Brent Graham @BgTalkinman:
Given that at the present time, the immediate impact of AI in hand surgery may be in the way in which it is used by patients, how should these findings be viewed? #jhsjc
Brent Graham @BgTalkinman:
I will give an answer to my own question! I would say that this is a major upgrade over pts searching Google for information like we know they do up until now. Do others agree? #jhsjc
Warren @whammert:
Maybe an upgrade, but different. and not always accurate#jhsjc
RyanC @RPCalfeeSTL:
I agree. The advice was pretty reasonable and I like the recs to ask a surgeon. #jhsjc
Zina Model @zinamodel1:
I agree that it’s still an upgrade from googling a question because it at least appears more so to synthesize a collection of data about a topic as compared to a single website that may be a paid ad
Dafang Zhang @DafangZhang:
I agree, and re: readability, you can always ask ChatGPT to rephrase answer in lower reading grade level #jhsjc
Brent Graham @BgTalkinman:
Does that capability exist presently? #jhsjc
Dafang Zhang @DafangZhang:
Absolutely! For example if you ask “tell me about brachial plexus injuries” and answer is jargony, you can ask “explain that at a 6th grade reading level.” #jhsjc
Brent Graham @BgTalkinman:
Fascinating! That hasn’t been mentioned in the submissions we’ve seen on this topic. #jhsjc
RyanC @RPCalfeeSTL:
Smart feature #jhsjc
Zina Model @zinamodel1:
I think thats a great use of AI! To make things more understandable for patients with lower health literacy #jhsjc
Brent Graham @BgTalkinman:
AI lost points on the DISCERN scale were related to “the lack of attribution, reliability and currency of the source material”. Given that the data on which the AI responses are generated is continually upgraded is really that a relevant deficiency? #jhsjc
RyanC @RPCalfeeSTL:
Important to know for now that references aren’t reliable but that will likely change #jhsjc
Brent Graham @BgTalkinman:
It’s likely that most clinical encounters in hand surgery probably don’t involve a pt previously consulting the HandCare website; the information they are given is from the surgeon. How accurate would surgeons likely to be in terms “attribution”, “reliability”, “currency”? #jhsjc
Brent Graham @BgTalkinman:
My guess is that surgeons would also lose points in those areas. #jhsjc
Brent Graham @BgTalkinman:
Aside from the fact that surgeons often also use words and explanations that patients cannot understand – and may not have much capacity for changing that – this seems likely to be an ongoing problem for AI-generated responses to patient questions #jhsjc
Warren @whammert:
I think it will be a challenge to get the reading score to the appropriate level #jhsjc
Brent Graham @BgTalkinman:
Evidently that can be done by just asking it to speak in 6th grade terms #jhsjc ask
@DafangZhang !
Brent Graham @BgTalkinman:
My guess is that this use of #AI is going to become quickly ingrained into our day-to-day as pts use it more and more. I think it’s promising. #jhsjc
RyanC @RPCalfeeSTL:
Fascinating for sure #jhsjc
Brent Graham @BgTalkinman:
Let’s move on to the other paper, “Do Surgeons Accurately Predict Level of Activity in Patients With Distal Radius Fractures?”
https://jhandsurg.org/article/S0363-5023(23)00364-7/fulltext #jhsjc
Brent Graham @BgTalkinman:
https://x.com/BgTalkinman/status/1724616513674428878?s=20 #jhsjc
Warren @whammert:
Seems like the surgeons could ask a few key questions and then predictions would be closer #jhsjc
RyanC @RPCalfeeSTL:
Shows we are often working off quick impressions and no surprises – we are imperfect #jhsjc
Zina Model @zinamodel1:
thanks for selecting! We think that this shows surgeons that we make assumptions about patients functional status in a short clinical encounter, and these assumptions can impact care #jhsjc
Zina Model @zinamodel1:
I think it puts some data behind an unconscious bias that we may have about patients from an initial visit #jhsjc
Brent Graham @BgTalkinman:
I think “unconscious bias” is a good way of putting it! #jhsjc
Brent Graham @BgTalkinman:
The study compared the self-assessments of activity level of patients with distal radius fractures with that of the surgeons treating them. #jhsjc
Brent Graham @BgTalkinman:
The patients’ assessment used the International Physical Activity Questionnaire, while the surgeons familiarized with the IPAQ & made a global assessment of the patient activity level. #jhsjc
Brent Graham @BgTalkinman:
@zinamodel1 is a co-author of this interesting article! #jhsjc
Brent Graham @BgTalkinman:
The overall interrater agreement between surgeons and patients was classified as “fair”. Surgeons correctly identified 73% of patients who were “high” activity on the IPAQ, but less than 60% of patients identified as “moderate” or “low” activity. #jhsjc
Brent Graham @BgTalkinman:
The surgeons didn’t fill out the IPAQ themselves, but were familiarized with it as they made their assessment. The authors took this approach because this is how these judgments would be made in the clinic setting. Was that a valid strategy ? #jhsjc
Dafang Zhang @DafangZhang:
this seems like the way to do it…to pit patients’ actual activity level by IPAQ against surgeons’ perception however that is arrived at #JHSJC
Warren @whammert:
I am ok with this approach, but once familiar, the surgeon could ask a few questions and maybe be more accurate #jhsjc
Brent Graham @BgTalkinman:
Hopefully they do ask questions but seemingly this data shows that those don’t add up to an accurate estimate of the pt’s own assessment #jhsjc
Warren @whammert:
it is imperfect. While one could argue it is justification for operating, there is an effort to understand the patients perspective #jhsjc
Brent Graham @BgTalkinman:
I agree both that it is imperfect and that surgeons may make an effort to understand the pt’s perspective but this data seems to show that understanding isn’t great. Introducing tools like the IPAQ might make it at least somewhat more objective #jhsjc
Brent Graham @BgTalkinman:
Sx treatment was correlated with activity level as assessed by the surgeon, but that association disappeared after adjusting for age, comorbidity index, and fx class. What does that suggest about the role of perceived activity level in decisions about surgical management ? #jhsjc
Dafang Zhang @DafangZhang:
In a new pt visit, surgeons use mental shortcuts to arrive at a conclusion about physical activity in a limited time window. I wonder if these shortcuts are a sophisticated derivation of some of these factors (age, comorbidities, etc) #jhsjc
Brent Graham @BgTalkinman:
I’m certain that heuristic thinking of that nature is key to all decision-making – it’s natural and efficient, but it turns out to be less accurate than might be hoped. #jhsjc
Dafang Zhang @DafangZhang:
Agreed. And I wonder if that’s why after adjusting for these factors, the association disappeared #jhsjc
Brent Graham @BgTalkinman:
For decision-making to be truly shared, it is important for patients and surgeons to be in general agreement regarding the parameters that are used to define the indications for various types of treatment. How can these findings be used to optimize that alignment? #jhsjc
Brent Graham @BgTalkinman:
I hope the discussion of both of these important topics – #AI and #shareddecisionmaking continues! Thanks to all for your insights! We return Dec 12 9 pm EST! #jhsj
