Academic Journal
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
| Title: | Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments |
|---|---|
| Authors: | Dana Brin, Vera Sorin, Akhil Vaid, Ali Soroush, Benjamin S. Glicksberg, Alexander W. Charney, Girish Nadkarni, Eyal Klang |
| Source: | Scientific Reports, Vol 13, Iss 1, Pp 1-5 (2023) |
| Publisher Information: | Nature Portfolio, 2023. |
| Publication Year: | 2023 |
| Collection: | LCC:Medicine LCC:Science |
| Subject Terms: | Medicine, Science |
| More Details: | Abstract The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was used to assess the models’ consistency. The performance of the AI models was compared to that of previous AMBOSS users. GPT-4 outperformed ChatGPT, correctly answering 90% compared to ChatGPT’s 62.5%. GPT-4 showed more confidence, not revising any responses, while ChatGPT modified its original answers 82.5% of the time. The performance of GPT-4 was higher than that of AMBOSS's past users. Both AI models, notably GPT-4, showed capacity for empathy, indicating AI's potential to meet the complex interpersonal, ethical, and professional demands intrinsic to the practice of medicine. |
| Document Type: | article |
| File Description: | electronic resource |
| Language: | English |
| ISSN: | 2045-2322 |
| Relation: | https://doaj.org/toc/2045-2322 |
| DOI: | 10.1038/s41598-023-43436-9 |
| Access URL: | https://doaj.org/article/94f4d0dbdf3c489198acd4ee0d464e25 |
| Accession Number: | edsdoj.94f4d0dbdf3c489198acd4ee0d464e25 |
| Database: | Directory of Open Access Journals |
| Full text is not displayed to guests. | Login for full access. |
| ISSN: | 20452322 |
|---|---|
| DOI: | 10.1038/s41598-023-43436-9 |
| Published in: | Scientific Reports |
| Language: | English |