Abstract Body: Background: Providing thrombolytic therapy to eligible acute ischemic stroke patients is time-sensitive. Manual chart review to screen for thrombolysis contraindications may be time-consuming and prone to errors. We designed a novel tool based on a large language model (LLM) to automatically identify contraindications from clinical notes, and we tested its performance using synthetic clinical notes.

Methods: We first generated synthetic clinical notes using LLMs (ChatGPT-4o, Llama 3.1, ChatGPT-4o mini and Gemini 1.5 Flash), each containing a randomly predetermined set of thrombolysis contraindications. We then tested the performance of Llama 3.1 405B, an open-source LLM by Meta, equipped with a custom prompt. Using this prompt and a synthetic clinical note as input, Llama 3.1 405B generated a list of thrombolysis contraindications. Performance measures included sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and F1 score.

Results: A total of 150 synthetic notes were generated (ChatGPT-4o, n=40, 26.7%; Llama 3.1 405B, n=20, 13.3%; Llama 3.1 70B, n=50, 33.3%; ChatGPT-4o mini, n=20, 13.3%; Gemini 1.5 Flash, n=20, 13.3%). Each note contained on average 241.6 words (SD 110.7, range 80-549) and 1.5 contraindications (SD 1.1, range 0-5). Our tool had a sensitivity of 90.9% (95% CI, 86.3-94.3), specificity of 99.2% (95% CI, 98.8-99.5), PPV of 87.7% (95% CI, 82.7-91.7), NPV of 99.4% (95% CI, 99.1-99.6), accuracy of 98.7% (95% CI, 98.2-99.0), and F1 score of 0.892. False positives were due to the inclusion of irrelevant clinical details (n=24, 86%) and repetitive information (n=4, 14%). No hallucination has been observed. No evidence of sex bias was observed for sensitivity (p=0.06) and PPV (p=0.86). Our tool had better sensitivity for notes generated by ChatGPT-4o and Llama 3.1 405B than ChatGPT-4o mini and Gemini 1.5 Flash (95.6% vs 80.0%, p=0.003). It also had better sensitivity and PPV for notes generated by Llama 3.1 70B than ChatGPT-4o mini and Gemini 1.5 Flash (sensitivity 93.2% vs 80.0%, p=0.03; PPV 94.4% vs 83.0%, p=0.04).

Conclusion: Our LLM-based tool was able to identify stroke thrombolysis contraindications from synthetic clinical notes with very high sensitivity and PPV. We plan to validate the effectiveness of this tool in enabling faster and safer acute stroke thrombolysis decision-making with future prospective studies, with the eventual goal of acute stroke workflow integration.