Promising algorithms to perilous applications: a systematic review of risk stratification tools for predicting healthcare utilisation

Systematic review

Our review identified 3897 publications eligible for screening after duplicates were removed (figure 2). Of these, 3636 were excluded on the basis of their title or abstract alone leaving 261 that were sought for retrieval. Full texts could not be retrieved for 10 publications, thus 251 were reviewed in full. A total of 51 publications met our criteria and were included in our final analysis (online supplemental table 1).25–75 Further detail about the identified models, along with our risk of bias analysis, can be found in online supplemental materials.

The majority of studies were based in the USA (23), with the remainder set in the UK (10), Spain (9), Canada (2), Italy (2), New Zealand (2), Australia (1), Ireland (1) and Israel (1). Population sizes ranged from 96 to 5.4 million with a median value of 94 264 (IQR 12 800–434 027). Hospital admission was the most commonly predicted outcome (34), followed by healthcare costs (14), emergency department attendance (9), access to primary care services (8), mortality (5) and readmission (2).

19 studies reported the derivation and internal validation of a risk stratification model with 32 describing validation of a model in a separate population dataset. 10 studies reported the results of implementing PHM measures based on case selection by a risk stratification model in a real-world clinical pathway. These included five randomised control trials (RCTs), three prospective cohort studies and two retrospective cohort studies. PHM strategies used were case management (8), telemonitoring (4) and care coordination (3).

We identified 28 risk stratification tools across all studies. 42 studies examined a single model, whereas 9 studied the comparative efficacy of several models. Johns-Hopkins ACG was the most studied algorithm (20), followed by the Charlson Comorbidity Index (10), Hierarchical Condition Categories (8), the Chronic Illness and Disability Payment System (3), RxRisk (3), the Elder Risk Assessment Index (2), the Patients At Risk of Rehospitalisation algorithm (2) and QAdmissions (2). Of the remainder, four were proprietary ML algorithms.

Results of internal and external validation studies

A summary of the derivation characteristics of each of the 28 discovered models is compared with the results of subsequent validation studies in online supplemental table 2.25–84 The results of internal validation studies echoed previous reviews with C-statistics for various outcomes ranging from 0.67 to 0.90. Notably, three of the highest C-statistics within internal validation samples were displayed by models derived using ML techniques—0.84,67 0.8542 and 0.90.55

Half (14) of the discovered models underwent external validation. Of these, only the Charlson Comorbidity Index and the Johns Hopkins ACG System were validated internationally. Model performance in external validation studies generally resembled internal validation performance for each model, with C-statistics ranging from 0.53 to 0.88. Accounting for heterogeneity in study design and reporting, there was no evident association between validation context and model discrimination, with models broadly displaying consistent predictive performance when transported to external datasets.

Results of real-world evaluation studies

Two studies reported the implementation of risk stratification tools into care pathways within the same population used for development. The Nairn Case Finder73 and the Predictive RIsk Stratification Model (PRISM)25 algorithms were used to identify those that might benefit from case management, both in the hope of reducing hospital admissions. In a prospective stepped-wedge clinical trial conducted across more than 230 000 patients in 32 primary care practices, the practice resource allocation intervention linked to PRISM resulted in significantly increased hospital admissions (OR 1.44 (95% CI 1.39 to 1.50), p<0.001), as well as increased emergency presentations, time in hospital, and primary care workload. The intervention guided by the Nairn Case Finder significantly reduced hospital admissions (AD=42.5%, p=0.002) in a population of 96 high-risk patients from a single locality, when matched 1:1 on risk score to patients in a separate control population.

Eight of the discovered models were deployed as tools for case selection as part of a PHM strategy in a separate context from development. The Johns Hopkins ACG System was deployed in two separate studies, whereas each of the other models was deployed only once. Healthcare utilisation measures were not significantly influenced by interventions guided by the Hierarchical Condition Category71 and PacifiCare’s Medicare Risk Programme37 models. Similarly equivocal evidence for the efficacy of interventions linked with the Johns Hopkins ACG System was observed, with one study showing no benefit31 and the other demonstrating benefit in groups selected by the model (OR 0.91 (95% CI 0.86 to 0.96)) but reciprocal harm in non-prioritised groups (OR 1.19 (95% CI 1.09 to 1.30)).32 Interventions linked with the Elder Risk Assessment Index30 and QAdmissions48 algorithms led to significant increases in mortality (AD 10.8%, p=0.008) and hospital admissions (difference in difference 79.8 (95% CI 21.2 to 138.4), p=0.01), respectively.

Significant reductions in hospital admissions were achieved through interventions guided by the combined predictive model (AD=−0.9, p<0.001),39 Patients At Risk for Rehospitalisation algorithm (AD=−0.3, p<0.001)39 and SCAN Health Plan Model (AD=11.5%, p=0.02).51 Figure 3 summarises the main findings of this review, describing only the models that underwent external validation or real-world evaluation.