For this month’s discussion, we discussed these leading articles from the October issue of JHS:
- Exploring the Role of a Large Language Model on Carpal Tunnel Syndrome Management: An Observation Study of ChatGPT (https://www.jhandsurg.org/article/S0363-5023(23)00360-X/fulltext)
- Catastrophic Health Expenditures Associated With Open Reduction Internal Fixation of Distal Radius Fractures (https://www.jhandsurg.org/article/S0363-5023(23)00323-4/fulltext)
We thank all of our participants! We’ll be back next month on November 14th.
Check out previous #JHSJC transcripts (please scroll to the bottom of the page).
Brent Graham @BgTalkinman:
Welcome to the October #jhsjc! There is no chatroom. Just remember to use this hashtag in all your posts so everyone can follow the your points in the discussion.
Brent Graham @BgTalkinman:
Let’s begin with article no. 1 “Exploring the Role of a Large Language Model on Carpal Tunnel Syndrome Management: An Observation Study of ChatGPT” https://jhandsurg.org/article/S0363-5023(23)00360-X/fulltext #jhsjc
Brent Graham @BgTalkinman:
https://x.com/BgTalkinman/status/1711910401166774356?s=20 #jhsjc
Brent Graham @BgTalkinman:
The authors tested the ability of a large language model (LLM), ChatGPT to respond to basic questions regarding the diagnosis and management of carpal tunnel syndrome (CTS). #jhsjc
RyanC @RPCalfeeSTL:
certainly this technology is evolving quickly and a hot topic #jhsjc
Brent Graham @BgTalkinman:
The main deficiencies were in suggesting diagnostic pathways that the authors considered inconsistent with contemporary practice, mostly around the use of electrodiagnostic testing, and in providing supportive references that were inaccurate, or even which didn’t exist. #jhsjc
Brent Graham @BgTalkinman:
In responding about treatment, ChatGPT mentioned modalities like #diuretics, #yoga and some therapy modalities that have been reported in the literature, but which have not been included in clinical practice guidelines, like those developed by AAOS #jhsjc
Brent Graham @BgTalkinman:
Given the ever-increasing volume of medical literature and the inability if humans to synthesize it, can LLMs like ChatGPT provide more evidence-based information to both patients than clinicians? #jhsjc
RyanC @RPCalfeeSTL:
Interesting as this seems to provide decent information – better than many sites – but at least currently evidence or references cited not correct #jhsjc
Brent Graham @BgTalkinman:
Maybe I will answer my own question: I think that with future revisions to #LLMs and frequent updating, they will be a better source of evidence based info than will clinicians #jhsjc
Brent Graham @BgTalkinman:
I say that LLMs will be better because they will not only have access to the information, but will be less prone to bias #jhsjc
RyanC @RPCalfeeSTL:
That may be quite true about bias
Brent Graham @BgTalkinman:
The main immediate impact will be to provide better info to patients than they can get from searching the internet themselves. #jhsjc
Avi Giladi @theaviram:
LLMs still very susceptible to bias. And whatever info trains them could — in theory — be available on the internet as well especiall re hand surgery evidence. LLM complex synthesis potential appealing but I think in the distant future #jhsjc
Brent Graham @BgTalkinman:
I suspect that rather than the distant future, improvements are in the very near future #jhsjc
Brent Graham @BgTalkinman:
If it has access to all of the information out there how does bias creep in? Will some types of information be preferred by the #LLM? That would result in bias but I’m not sure I see how that would happen. #jhsjc
Avi Giladi @theaviram:
what if it trains on the ASSH ListServ? #jhsjc
Avi Giladi @theaviram:
https://forbes.com/sites/forbestechcouncil/2023/09/06/navigating-the-biases-in-llm-generative-ai-a-guide-to-responsible-implementation/?sh=3c063f835cd2 nice quick read if helpful #jhsjc
Brent Graham @BgTalkinman:
Ha! I’m actually not sure how it trains but I suspect the scope is wide and dominated by the published medical literature. I think that is reflected by most of the results of the study.
Avi Giladi @theaviram:
I hope that’s true but ultimately I think overly optimistic. There is so much internet noise. I worry clean LLMs will be nearly impossible #jhsjc
Brent Graham @BgTalkinman:
I just posted this. I think @HandSociety should consider a group to think about this. You and some other young, forward-thinking individuals could be very important participants in something like that. https://t.co/TX9w1hfgP3
Avi Giladi @theaviram:
Very interesting. An important consideration. Definitely something worth pursuing and I’d be happy to contribute.
Brent Graham @BgTalkinman:
Are hand surgeons encountering patients who report information they’ve learned from consulting #LLMs? If so, is that information accurate and helpful or inaccurate and an impediment to counseling about CTS? #jhsjc
RyanC @RPCalfeeSTL:
I haven’t encountered patients quoting this yet. #jhsjc
Brent Graham @BgTalkinman:
The references ChatGPT was asked to provide in support of the responses were frequently incorrect and other studies have reported that are sometimes non-existent. Is that actually a problem at this time when it is being used as a patient info resource? #jhsjc
Brent Graham @BgTalkinman:
The findings are consistent with what has already appeared in the literature and even with some studies that have been reviewed at JHS and which are likely to be published in coming months. What is the future of #LLMs in hand surgery? #jhsjc
Brent Graham @BgTalkinman:
We are going to discuss #LLM and #ChatGPT more in the future bc we will publishing more studies on this subject in coming months. #jhsjc
Brent Graham @BgTalkinman:
Let’s move on to article no. 2, “Catastrophic Health Expenditures Associated With Open Reduction Internal Fixation of Distal Radius Fractures”
https://jhandsurg.org/article/S0363-5023(23)00323-4/fulltext
podcast: https://jhandsurg.org/pb-assets/Health%20Advance/journals/yjhsu/podcasts/October_2023-1696169558683.mp3
https://x.com/BgTalkinman/status/1711918227125629327?s=20
Brent Graham @BgTalkinman:
The aim of the study was to investigate the incidence of catastrophic health expenditures (#CHE), defined as out-of-pocket expenses of 40% of post-subsistence income, among patients being treated for a distal radius fracture with open reduction and internal fixation. #jhsjc
Brent Graham @BgTalkinman:
In a sample of 394 patients, 121 (31%) met the definition for having incurred #CHE. Younger patients, those who were uninsured and non-White patients all had much higher odds of incurring CHE. #jhsjc
Brent Graham @BgTalkinman:
The study was done at a level I trauma center in the Midwest of the United States. The sample was almost 90% White and 70% of individuals had employer-sponsored insurance. Are the findings generalizable to the US population? #jhsjc
RyanC @RPCalfeeSTL:
I likely some differences in populations at different centers but remarkable number of patients at risk for substantial bills #jhsjc
Brent Graham @BgTalkinman:
Why are insured pts at such high risk for #CHE? #jhsjc
RyanC @RPCalfeeSTL:
I think gets back to variation in deductibles and out of pocket max’s on plans #jhsjc
Brent Graham @BgTalkinman:
Presumably that will be an insolvable problem without some … regulation! #jhsjc
Brent Graham @BgTalkinman:
The overall percentage of patients meeting the definition for #CHE was 31% but the proportion among patients with employer-sponsored insurance was even higher. What are some of the possible explanations for that surprising finding? #jhsjc
Brent Graham @BgTalkinman:
What practical solutions are there to address a situation where even a relatively common injury that often requires surgical treatment in patients with private insurance results in this kind of financial stress in almost one-third of individuals? #jhsjc
Brent Graham @BgTalkinman:
The authors conclude that policies aimed at minimizing cost-sharing for unexpected medical circumstances, such as trauma, are required to influence financial outcomes in patients and reduce potential disparities. Is that a practical idea? #jhsjc
Brent Graham @BgTalkinman:
Hopefully there will be more discussion of these important papers resulting from the comments made tonight! We return November 14 #jhsjc
