[ad_1]
Discussion
This study explored various EMR data-based case definitions for diabetes, uncovering algorithms with excellent performance. We used chart review labels as our gold standard. While the validated administrative data-based ICD-code algorithm demonstrated strong performance, the findings support our hypothesis that harnessing free-text notes can yield comparable or superior results to existing standard methods. The ML algorithm that included all document types of free-text notes was the top performer in this study cohort, with 0.95 SN and 0.94 PPV. Meanwhile, the combination of free-text algorithm, medication, and ICD codes improved the SN to 0.97 but experienced a decline in PPV to 0.87.
The current operational standards for defining diabetes for surveillance (ie, NDSS)12 and research purposes in Canada were shaped by the administrative data-based ICD code algorithm.13 These methodologies rely on the utilisation of ICD-code databases, and rely on readily available standardised ICD-code databases, like the DAD, established at both national and international settings. In the Canadian context, these DAD records are reliant on the quality of ICD codes produced by the trained coders who review the charts. Diabetes is a chronic condition which is heavily emphasised for ICD coding in Alberta, and yet the algorithms that solely use these codes resulted in a lower SN compared with the free-text algorithm. This discrepancy stems from the fact that ICD coders primarily review physician documentations from free-text documents within the EMR system for ICD coding in Canada, as dictated by the system design. Challenges and limitations encountered in ICD coding have been described in previous studies24 indicating the information overload experienced by the healthcare system and workers in various areas when dealing with EMR data.
A recent scoping review highlighted that diabetes definitions typically incorporate laboratory and medications data, along with ICD codes.8 Laboratory data typically employ values surpassing specific clinical thresholds to determine disease status. When a patient is being treated with antihyperglycaemic medication, these clinical values are presumed not reach that threshold due to the medication’s effect. In our study, the combined clinical diagnosis algorithm of laboratory and medication had a 0.90 SN and 0.80 PPV, which is comparable to algorithms described in the above-mentioned review. In a systematic review25 on the applications of NLP in diabetes care showed that out of 38 studies, 17 aimed to define diabetes, but most of these studies relied on single concept words or keyword-based definitions (ie, diabetes). In our cohort, the keyword algorithm had an 0.73 SN and 0.70 PPV, potentially reflecting the quality of documentation or the practice of data being entered into the EMR from the front end. Figure 3 showed that several consistent diabetes related medication terminologies (eg, metformin and insulin) were captured across multiple EMR document types. The ML-based algorithm which included all types of free-text documents performed the best in this study cohort, achieving a SN of 0.95 and PPV of 0.94 PPV, raising several important considerations. The ICD code algorithm had an 0.84 SN and 0.93 PPV. Combined algorithms often increased SN but reduced PPV, which was expected.
EMR systems, such as SCM1 and Connect Care (Alberta’s newly implemented province-wide clinical information system),26 based on Epic software (Madison, WI), typically have a front-end graphical user interface for delivering clinical care. It is important to note that not all healthcare workers or providers have access to complete patient charts, and access is typically determined based on assigned roles in the system. Information overload from EMR data can occur if too much information is given,27 and communication oversight could arise if insufficient information is provided.28 Additionally, the quality of clinical notes documentation can be heavily influenced by interactions between the care providers and patients or their family members, potentially triggering varying sets of orders and interventions documented in the EMR system. This project extracted all free-text notes from the back end of the EMR system and processed these documents using a standardised medical terminology dictionary (ie, UMLS). Our findings demonstrated that various types of healthcare workers and providers are documenting similar medical concepts across multiple EMR document types for diabetes. Therefore, analysing the commonality in documentation across roles to consolidate and centralise information for shared awareness would enhance information flow in clinical care settings and improve downstream processes, such as improving the quality of the administrative health databases.
Current diabetes definitions based on ICD-code databases are not integrated into clinical practices within the Canadian context, as DAD coding systems and EMR systems operate separately from each other. Alberta’s Connect Care clinical information system which includes EPIC-based EMR infrastructure, now in operations throughout AHS operated acute care and ambulatory facilities, has the capacity of integrating ML models,29 with potential outputs incorporated into dashboards. The integration of inpatient data-specific case definitions could facilitate easier identification of comorbidities, designing automated risk prediction algorithms within EMR which could be implemented into point of care as needed. As EMR adoption in Canada continues to rise,4 the implementation of EMR data-based diabetes case definitions from both inpatient and outpatient care30 has the potential to enhance the quality of DAD data for diabetes. This, from a research operations standpoint, could assist with cohort selection for epidemiological and clinical studies. The subsequent improvement in DAD will, in turn, enhance the surveillance capabilities of the NDSS for Alberta in the long run.
This study is not without limitations. First, as we used a single geographic setting, external validation from a different geographical setting is needed. Second, our algorithms do not differentiate between type 1 and type 2 diabetes, the two most common forms of diabetes. With the prevalence of both types increasing, as well as differences in management and care, differentiating between these types is important, this will be an area of future work. Also, we appreciate the immaturity of the proposed application in real-life practice but importantly this study is foundational work for ML in healthcare systems. We appreciate the limited interpretability by the prediction model. Importantly, in our study, we demonstrated the explainbility by showing that top features (figure 3) are coinciding with what is documented within clinical practices. This strengthens the application of our model in real-world practice. We also appreciate the lack of system infrastructure to implement models with existing EMRs not having the capacity to implement designed ML models. AHS has recently implemented EPIC-based clinical information system, which has the capacity to integrate ML models into EMR systems, in AHS-operated and partner acute and subacute care sites, ambulatory care locations, clinical lab services and diagnostic imaging areas. That being said, our study includes many strengths. Strengths include taking a multimodal EMR data approach to develop a case definition for diabetes and comparing to existing standards, integrating ML and NLP onto EMR data, and using the randomly selected chart review data as the gold standard.
Our future studies will expand to include Connect Care data and eventually validate this work in other jurisdictions. Furthermore, we will evaluate the implementation of our ML models into existing clinical information systems. Recent advancements in large language models have shifted the interest in developing such models for eventual deployment in healthcare systems from the NLP field perspective. While we acknowledge that we have not considered these deep learning NLP models for this study, a future study is in the planning stages, aiming to explore large language model methods on a study cohort with a much larger disease prevalence.10
[ad_2]
Source link
