Abstract Body (Do not enter title and authors here): GPT-3 and GPT-4, show promise in assisting healthcare professionals with clinical questions. Their performance in real-time clinical scenarios, particularly with cardiovascular symptoms, remains underexplored. This study aims to evaluate their precision and reliability compared to physicians.
HYPOTHESIS
We hypothesize that board-certified emergency department attendings at Maimonides Medical Center exhibit higher accuracy and reliability than GPT-3.5 and GPT-4 in generating differential diagnoses based on history and physical examination for patients presenting to the emergency department.
METHODS
Patient data from Maimonides Medical Center's emergency department, collected from January 1, 2023, to April 30, 2023, was analyzed. Demographics, symptoms, medical history, and discharge diagnoses recorded by emergency room attendings were examined. ChatGPT-3.5 and GPT-4 generated differential diagnoses, which were compared with those by attending physicians. Accuracy was determined by comparing each rater's diagnoses with discharge diagnosis, calculating the proportion of correctly identified cases. Precision was assessed using Cohen's kappa coefficient and Intraclass Correlation Coefficient to measure agreement between raters.
RESULTS
Patient demographics showed mean age of 49.12 years, with 57.3% males and 42.7% females. Chief complaints included fever/sepsis (24.7%), gastrointestinal issues (17.7%), and cardiovascular problems (16.4%). Diagnostic accuracy against discharge diagnoses was highest for Chat GPT-4 (85.5%), followed by Chat GPT-3.5 (84.6%) and ED attendings (83%). Cohen’s kappa demonstrated moderate agreement (0.7) between AI models, with lower agreement observed for ED attendings. Stratified analysis revealed higher accuracy for gastrointestinal complaints with Chat GPT-4 (87.5%) and cardiovascular complaints with Chat GPT-3.5 (81.34%).
CONCLUSION
Our study demonstrates that Chat GPT-4 and GPT-3.5 exhibit comparable diagnostic accuracy to attendings, highlighting their potential to aid decision-making in dynamic clinical settings. The stratified analysis revealed comparable reliability and precision of the AI chat bots for cardiovascular complaints which represents a significant proportion of the high risk patients presenting to the emergency department. This study contributes to integrating AI models into medical practice, enhancing efficiency and effectiveness in clinical decision-making