Abstract Body: Introduction
Clinical risk scores are evaluated in terms of discrimination (correctly classifying patients) and calibration (accurately predicting individualized risks). When comparing new and existing models, however, the impact of additional variables is generally limited to measures of discrimination and reclassification, with no consensus on how to test calibration improvements. This study introduces the application of pseudo-values (PV) methods to evaluate calibration and recalibration.
Hypothesis
We hypothesize this approach to be particularly valuable when additional variables enhance risk prediction without necessarily improving discrimination, and for assessing model calibration at different time points without assuming proportional hazards.
Methods
Individuals PVs are estimated by comparing the full sample estimate to those when that individual is excluded (leave-one-out estimator). In survival analysis PVs are used to retrieve information on censored individuals, enabling linear modeling of time-specific event risks - the measure of interest for calibration assessment - and straightforward derivation of calibration plots. The deviance of a PV model summarizes model fit, with lower values indicating better agreement between observed and predicted risks, and statistical tests are available for comparing the fit of nested models. Moreover, PV models allow estimating time-specific risks without proportionality assumptions, providing a flexible tool for assessing calibration at different time points. As an illustration, we applied the PV approach to compare alternative versions of the TIMI Risk Score for Atherothrombosis in diabetes, evaluating clinical and biomarker data in 36,632 individuals.
Results
Figure 1 compares the PV implementation of the original risk score (left panel) and a second model with selected biomarker data (right panel). In this example, the expanded model provided similar overall performances (Brier score: 0.039 vs 0.040) and a moderate increase in discrimination (C-statistic: 0.735 vs 0.698). Calibration plots indicate better accuracy, confirmed by a significant improvement in model fit (p<0.001). Calibration was remarkably similar when predicting event risks at different times (Figure 2) with poorer performances only at the extremes of the risk distribution.
Conclusions
The PV approach complements existing methods for clinical prediction with survival data, offering advantages for the assessment of model calibration and recalibration.