Abstract Body (Do not enter title and authors here): Background: Immune-checkpoint inhibitor (ICI)-induced myocarditis is the most fatal immune-related adverse event (irAE). Early recognition and treatment of ICI myocarditis is associated with improved outcomes. However, current approaches to the diagnosis of ICI myocarditis have limitations. Open-source large language models (LLMs) are an accessible and scalable method of answering queries from human-generated text, and therefore may assist in the real-time early detection of ICI myocarditis among at-risk patients.
Research Question: Can a free, open-source LLM detect cases of ICI myocarditis early in admission?
Aims: We investigated the ability of a LLM pipeline to screen for ICI myocarditis within one day of admission using hospital medical records.
Methods: Hospital admissions of patients on ICI therapy from November 4th, 2021, to September 5th, 2023 were retrospectively reviewed by a multidisciplinary immunotoxicity team using established, published definitions for the presence of ICI myocarditis. Progress notes written the day before, day of, and day after admission were fed into an open-source LLM pipeline built using Mistral 7B OpenOrca with retrieval augmentation generation (RAG). Performance of the LLM was measured via sensitivity and specificity in comparison to the gold standard of manual adjudication.
Results: Of the 1874 hospital admissions of patients on ICI therapy, there were 22 (1.2%) cases of myocarditis. The average time to initiation of treatment was 2.45 days. Using notes written within one day after admission, the LLM detected ICI myocarditis with 95.5±8.7% sensitivity and 95.4±1.0% specificity, spending 2.13 seconds per chart.
Conclusion: LLMs serve as a useful tool to screen for ICI myocarditis, detecting 95.5% of cases within one day of admission with 95.4% specificity. Future studies will investigate if integrating this tool into the real-time management of irAEs could lead to earlier diagnosis, faster initiation of treatment, and improved patient outcomes.