tetano
Editor, Senior Moderator
J Med Internet Res
. 2020 Oct 2.
doi: 10.2196/24018. Online ahead of print.
Machine Learning to Predict Mortality and Critical Events in COVID-19 Positive New York City Patients: A Cohort Study
Akhil Vaid[SUP] 1 [/SUP], Sulaiman Somani[SUP] 1 [/SUP], Adam J Russak[SUP] 1 [/SUP], Jessica K De Freitas[SUP] 1 [/SUP], Fayzan F Chaudhry[SUP] 1 [/SUP], Ishan Paranjpe[SUP] 1 [/SUP], Kipp W Johnson[SUP] 1 [/SUP], Samuel J Lee[SUP] 1 [/SUP], Riccardo Miotto[SUP] 1 [/SUP], Felix Richter[SUP] 1 [/SUP], Shan Zhao[SUP] 1 [/SUP], Noam D Beckmann[SUP] 1 [/SUP], Nidhi Naik[SUP] 1 [/SUP], Arash Kia[SUP] 1 [/SUP], Prem Timsina[SUP] 1 [/SUP], Anuradha Lala[SUP] 1 [/SUP], Manish Paranjpe[SUP] 2 [/SUP], Eddye Golden[SUP] 1 [/SUP], Matteo Danieletto[SUP] 1 [/SUP], Manbir Singh[SUP] 1 [/SUP], Dara Meyer[SUP] 1 [/SUP], Paul F O'Reilly[SUP] 1 [/SUP], Laura Huckins[SUP] 1 [/SUP], Patricia Kovatch[SUP] 1 [/SUP], Joseph Finkelstein[SUP] 1 [/SUP], Robert M Freeman[SUP] 1 [/SUP], Edgar Argulian[SUP] 1 [/SUP], Andrew Kasarskis[SUP] 1 [/SUP], Bethany Percha[SUP] 1 [/SUP], Judith A Aberg[SUP] 1 [/SUP], Emilia Bagiella[SUP] 1 [/SUP], Carol R Horowitz[SUP] 1 [/SUP], Barbara Murphy[SUP] 1 [/SUP], Eric J Nestler[SUP] 1 [/SUP], Eric E Schadt[SUP] 1 [/SUP], Judy H Cho[SUP] 1 [/SUP], Carlos Cordon-Cardo[SUP] 1 [/SUP], Valentin Fuster[SUP] 1 [/SUP], Dennis S Charney[SUP] 1 [/SUP], David L Reich[SUP] 1 [/SUP], Erwin P Bottinger[SUP] 1 [/SUP], Matthew A Levin[SUP] 1 [/SUP], Jagat Narula[SUP] 1 [/SUP], Zahi A Fayad[SUP] 1 [/SUP], Allan C Just[SUP] 1 [/SUP], Alexander W Charney[SUP] 1 [/SUP], Girish N Nadkarni[SUP] 1 [/SUP], Benjamin Glicksberg[SUP] 1 [/SUP]
Affiliations
Abstract
Background: Coronavirus disease 2019 (COVID-19) has infected millions of patients worldwide and has been responsible for several hundred thousand fatalities. This has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods for achieving this are lacking.
Objective: We analyze Electronic Health Records from COVID-19 positive hospitalized patients admitted to the Mount Sinai Health System in New York City (NYC). We present machine learning models for making predictions about the hospital course over clinically meaningful time horizons based on patient characteristics at admission. We assess performance of these models at multiple hospitals and time points.
Methods: We utilized XGBoost and baseline comparator models, for predicting in-hospital mortality and critical events at time windows of 3, 5, 7 and 10 days from admission. Our study population included harmonized electronic health record (EHR) data from five hospitals in NYC for 4,098 COVID-19+ patients admitted from March 15, 2020 to May 22, 2020. Models were first trained on patients from a single hospital (N=1514) before or on May 1, externally validated on patients from four other hospitals (N=2201) before or on May 1, and prospectively validated on all patients after May 1 (N=383). Finally, we establish model interpretability to identify and rank variables that drive model predictions.
Results: On cross-validation, the XGBoost classifier outperformed baseline models, with area under the receiver operating characteristic curve (AUC-ROC) for mortality at 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days; XGBoost also performed well for critical event prediction with AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, XGBoost achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers for mortality prediction.
Conclusions: We trained and validated (both externally and prospectively) machine-learning models for mortality and critical events at different time horizons. These models identify at-risk patients, as well as uncover underlying relationships predicting outcomes.
. 2020 Oct 2.
doi: 10.2196/24018. Online ahead of print.
Machine Learning to Predict Mortality and Critical Events in COVID-19 Positive New York City Patients: A Cohort Study
Akhil Vaid[SUP] 1 [/SUP], Sulaiman Somani[SUP] 1 [/SUP], Adam J Russak[SUP] 1 [/SUP], Jessica K De Freitas[SUP] 1 [/SUP], Fayzan F Chaudhry[SUP] 1 [/SUP], Ishan Paranjpe[SUP] 1 [/SUP], Kipp W Johnson[SUP] 1 [/SUP], Samuel J Lee[SUP] 1 [/SUP], Riccardo Miotto[SUP] 1 [/SUP], Felix Richter[SUP] 1 [/SUP], Shan Zhao[SUP] 1 [/SUP], Noam D Beckmann[SUP] 1 [/SUP], Nidhi Naik[SUP] 1 [/SUP], Arash Kia[SUP] 1 [/SUP], Prem Timsina[SUP] 1 [/SUP], Anuradha Lala[SUP] 1 [/SUP], Manish Paranjpe[SUP] 2 [/SUP], Eddye Golden[SUP] 1 [/SUP], Matteo Danieletto[SUP] 1 [/SUP], Manbir Singh[SUP] 1 [/SUP], Dara Meyer[SUP] 1 [/SUP], Paul F O'Reilly[SUP] 1 [/SUP], Laura Huckins[SUP] 1 [/SUP], Patricia Kovatch[SUP] 1 [/SUP], Joseph Finkelstein[SUP] 1 [/SUP], Robert M Freeman[SUP] 1 [/SUP], Edgar Argulian[SUP] 1 [/SUP], Andrew Kasarskis[SUP] 1 [/SUP], Bethany Percha[SUP] 1 [/SUP], Judith A Aberg[SUP] 1 [/SUP], Emilia Bagiella[SUP] 1 [/SUP], Carol R Horowitz[SUP] 1 [/SUP], Barbara Murphy[SUP] 1 [/SUP], Eric J Nestler[SUP] 1 [/SUP], Eric E Schadt[SUP] 1 [/SUP], Judy H Cho[SUP] 1 [/SUP], Carlos Cordon-Cardo[SUP] 1 [/SUP], Valentin Fuster[SUP] 1 [/SUP], Dennis S Charney[SUP] 1 [/SUP], David L Reich[SUP] 1 [/SUP], Erwin P Bottinger[SUP] 1 [/SUP], Matthew A Levin[SUP] 1 [/SUP], Jagat Narula[SUP] 1 [/SUP], Zahi A Fayad[SUP] 1 [/SUP], Allan C Just[SUP] 1 [/SUP], Alexander W Charney[SUP] 1 [/SUP], Girish N Nadkarni[SUP] 1 [/SUP], Benjamin Glicksberg[SUP] 1 [/SUP]
Affiliations
- PMID: 33027032
- DOI: 10.2196/24018
Abstract
Background: Coronavirus disease 2019 (COVID-19) has infected millions of patients worldwide and has been responsible for several hundred thousand fatalities. This has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods for achieving this are lacking.
Objective: We analyze Electronic Health Records from COVID-19 positive hospitalized patients admitted to the Mount Sinai Health System in New York City (NYC). We present machine learning models for making predictions about the hospital course over clinically meaningful time horizons based on patient characteristics at admission. We assess performance of these models at multiple hospitals and time points.
Methods: We utilized XGBoost and baseline comparator models, for predicting in-hospital mortality and critical events at time windows of 3, 5, 7 and 10 days from admission. Our study population included harmonized electronic health record (EHR) data from five hospitals in NYC for 4,098 COVID-19+ patients admitted from March 15, 2020 to May 22, 2020. Models were first trained on patients from a single hospital (N=1514) before or on May 1, externally validated on patients from four other hospitals (N=2201) before or on May 1, and prospectively validated on all patients after May 1 (N=383). Finally, we establish model interpretability to identify and rank variables that drive model predictions.
Results: On cross-validation, the XGBoost classifier outperformed baseline models, with area under the receiver operating characteristic curve (AUC-ROC) for mortality at 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days; XGBoost also performed well for critical event prediction with AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, XGBoost achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers for mortality prediction.
Conclusions: We trained and validated (both externally and prospectively) machine-learning models for mortality and critical events at different time horizons. These models identify at-risk patients, as well as uncover underlying relationships predicting outcomes.