Abstract Body (Do not enter title and authors here): Background
Major adverse cardiovascular events (MACE)—cardiovascular (CV) death, non-fatal myocardial infarction (MI) and non-fatal stroke—are highly relevant clinical outcomes. In global randomized trials, medical-record review by a physician clinical events committee (CEC) is the gold standard method for adjudicating MACE events but is labor intensive. Automated adjudication with artificial intelligence (AI) could reduce cost and improve reproducibility.
Research question
Is AI-based adjudication of MACE accurate compared to human CEC review in a global randomized trial?
Methods
We developed an AI-based adjudication system (“Auto-MACE”) that uses a iteratively refined prompt of the OpenAI o1-mini language model to generate a primary adjudication for each event based on discharge summaries. A complementary Clinical Longformer model trained on previously adjudicated events assigns a confidence level: confident accept, uncertain, or confident reject. We validated Auto-MACE against CEC adjudication in the PARADISE-MI trial, a global study comparing sacubitril/valsartan to ramipril in 5,661 patients with myocardial infarction complicated by a reduced left ventricular ejection fraction, pulmonary congestion, or both. We also compared the estimated treatment effect of sacubitril/valsartan vs ramipril on composite MACE using each adjudication method.
Results
The PARADISE-MI CEC reviewed 455 all-cause deaths, 658 potential non-fatal MI’s, and 167 potential non-fatal strokes. Auto-MACE agreed with the CEC adjudications in 86% of CV death events, 76% of MI events, and 84% of stroke events (Cohen’s Kappa of 0.70, 0.54, and 0.54 respectively). The model classified 41% of cases as uncertain (31% of deaths, 54% of MIs, and 19% of strokes). Among the remaining confident events, agreement with the CEC rose to 97% for CV deaths, 89% for MIs, and 88% for strokes (Cohen’s Kappa of 0.89, 0.78, and 0.50 respectively) (Figure). The estimated effect of sacubitril/valsartan vs ramipril on composite MACE was similar with Auto-MACE adjudication (HR 0.91 [95% CI 0.78-1.07]) and CEC adjudication (HR 0.90 [95% CI 0.77-1.05]).
Conclusion
AI-based adjudication of MACE showed high agreement with gold-standard CEC adjudication, especially for CV death and stroke, and for events in which the model was confident. A strategy of initial AI-based adjudication with human review of uncertain events may reduce adjudication workload while maintaining accuracy.