Abstract Body (Do not enter title and authors here): Introduction: In clinical trials, adjudication of clinical events is crucial for ensuring reliable
outcomes. This process is entirely manual and requires many hours of work from Health
Care Professionals. Here we present a hybrid machine learning (ML) model
designed for adjudication of Acute Coronary Syndrome (ACS) events together with a
human adjudication committee. Our approach offers a pragmatic solution for consistent
and timely adjudication of ACS events.
Methods: Data from four randomized cardiovascular outcome clinical trials (CVOT) were
combined to train an ML model to adjudicate ACS events (N= 2803). Site investigator
decisions, patient baseline characteristics, and text-extracted information from clinical
notes served as input features to this model. We established a hybrid adjudication
approach, where predictions with confidence scores below an established threshold
(determined from training data) were sent for manual adjudication. A simulation framework
was established to understand the effects of errors induced by this hybrid approach on the
estimated relative risk (RR) of confirmed ACS events in clinical trials.
Results: Based on training data threshold, we demonstrate 90.9% precision (95% CI:
[89.5%-93.0%]), 88.0% accuracy ([85.7%-90.2%]), 87.4% recall ([83.1%-
90.3%]), 89.1% F1 score ([86.4%-91.5%]), and a kappa statistic of 0.76 ([0.71-0.80]) on an independent test set. At this threshold, 80% of the ACS events were adjudicated by the model and 20% were manual. The impact of this split
showed that the ratio of estimated RR to true RR of ACS events was 0.9, where a score of 1
would mean perfect agreement between the hybrid model and fully manual adjudication.
Finally, using model explainability, we confirmed that major cardiac biomarkers reported in
clinical dossiers along with site investigator decisions were among the most important
features used by the model to generate the adjudication decisions.
Conclusion: Our study demonstrates that a hybrid model that uses clinical and text-
extracted features achieves high scores, indicating clinical trial utility. The estimated RR to
true RR ratio of 0.9 suggests that with a hybrid model, most adjudication cases (80%) can
be resolved with an ML model without compromising trial integrity. This framework also
allows users the flexibility to customize the desirable operating point, while keeping ahuman-in-the-loop. We demonstrate the eeectiveness and pragmatism of a hybrid human-ML adjudication approach in clinical trials.