Original Research

A trial of a real‐time Alert for clinical deterioration in Patients hospitalized on general medical wards



With limited numbers of intensive care unit (ICU) beds available, increasing patient acuity is expected to contribute to episodes of inpatient deterioration on general wards.


To prospectively validate a predictive algorithm for clinical deterioration in general–medical ward patients, and to conduct a trial of real‐time alerts based on this algorithm.


Randomized, controlled crossover study.


Academic center with patients hospitalized on 8 general wards between July 2007 and December 2011.


Real‐time alerts were generated by an algorithm designed to predict the need for ICU transfer using electronically available data. The alerts were sent by text page to the nurse manager on intervention wards.


Intensive care unit transfer, hospital mortality, and hospital length of stay.


Patients meeting the alert threshold were at nearly 5.3‐fold greater risk of ICU transfer (95% confidence interval [CI]: 4.6‐6.0) than those not satisfying the alert threshold (358 of 2353 [15.2%] vs 512 of 17678 [2.9%]). Patients with alerts were at 8.9‐fold greater risk of death (95% CI: 7.4‐10.7) than those without alerts (244 of 2353 [10.4%] vs 206 of 17678 [1.2%]). Among patients identified by the early warning system, there were no differences in the proportion of patients who were transferred to the ICU or who died in the intervention group as compared with the control group.


Real‐time alerts were highly specific for clinical deterioration resulting in ICU transfer and death, and were associated with longer hospital length of stay. However, an intervention notifying a nurse of the risk did not result in improvement in these outcomes. Journal of Hospital Medicine 2013;8:236–242. © 2013 Society of Hospital Medicine

Copyright © 2013 Society of Hospital Medicine

Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]

The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.


Study Location

The study was conducted at Barnes‐Jewish Hospital, a 1250‐bed academic medical center in St. Louis, Missouri. Eight adult medicine wards were assessed from July 2007 through December 2011. The medicine wards are closed areas with patient care delivered by dedicated house staff physicians under the supervision of a board‐certified attending physician. The study was approved by the Washington University School of Medicine Human Studies Committee.

Study Period

The period from July 2007 through January 2010 was used to train and retrospectively test the prediction model. The period from January 2011 through December 2011 was used to prospectively validate the model during a randomized trial using alerts generated from the prediction model.


Electronically captured clinical data were housed in a centralized clinical data repository. This repository cataloged 28,927 hospital visits from 19,116 distinct patients between July 2007 and January 2010. It contained a rich set of demographic and medical data for each of the visits, such as patient age, manually collected vital‐sign data, pharmacy data, laboratory data, and intensive care unit (ICU) transfer. This study served as a proof of concept for our vision of using machine learning to identify at‐risk patients and ultimately to perform real‐time event detection and interventions.

Algorithm Overview

Details regarding the predictive model development have been previously described.[11] To predict ICU transfer for patients housed on general medical wards, we used logistic regression, employing a novel framework to analyze the data stream from each patient, assigning scores to reflect the probability of ICU transfer to each patient.

Before building the model, several preprocessing steps were applied to eliminate outliers and find an appropriate representation of patients' states. For each of 36 input variables we specified acceptable ranges based on the domain knowledge of the medical experts on our team. For any value that was outside of the medically conceivable range, we replaced it by the mean value for that patient, if available. Values for every continuous parameter were scaled so that all measurements lay in the interval [0, 1] and were normalized by the minimum and maximum of the parameter. To capture the temporal effects in our data, we retained a sliding window of all the collected data points within the last 24 hours. We then subdivided these data into a series of 6 sequential buckets of 4 hours each.

To capture variations within a bucket, we computed 3 values for each feature in the bucket: the minimum, maximum, and mean data points. Each of the resulting 3n values was input to the logistic regression equation as separate variables. To deal with missing data points within the buckets, we used the patients' most recent reading from any time earlier in the hospital stay, if available. If no prior values existed, we used mean values calculated over the entire historical dataset. Bucket 6 max/min/mean represents the most recent 4‐hour window from the preceding 24‐hour time period for the maximum, minimum, and mean values, respectively. By itself, logistic regression does not operate on time‐series data. That is, each variable input to the logistic equation corresponds to exactly 1 data point (eg, a blood‐pressure variable would consist of a single blood‐pressure reading). In a clinical application, however, it is important to capture unusual changes in vital‐sign data over time. Such changes may precede clinical deterioration by hours, providing a chance to intervene if detected early enough. In addition, not all readings in time‐series data should be treated equally; the value of some kinds of data may change depending on their age. For example, a patient's condition may be better reflected by a blood‐oxygenation reading collected 1 hour ago than a reading collected 12 hours ago. This is the rationale for our use of a sliding window of all collected data points within the last 24 hours performed in a real‐time basis.

The algorithm was first implemented in MATLAB (Natick, MA). For the purposes of training, we used a single 24‐hour window of data from each patient. For patients admitted to ICU, this window was 26 hours to 2 hours prior to ICU admission; for all other patients, this window consisted of the first 24 hours of their hospital stay. The dataset's 36 input variables were divided into buckets and min/mean/max features wherever applicable, resulting in 398 variables. The first half of the dataset was used to train the model. We then used the second half of the dataset as the validation dataset. We generated a predicted outcome for each case in the validation data, using the model parameter coefficients derived from the training data. We also employed bootstrap aggregation to improve classification accuracy and to address overfitting. We then applied various threshold cut‐points to convert these predictions into binary values and compared the results against the ICU transfer outcome. A threshold of 0.9760 for specificity was chosen to achieve a sensitivity of approximately 40%. These operating characteristics were chosen in turn to generate a manageable number of alerts per hospital nursing unit per day (estimated at 12 per nursing unit per day). At this cut‐point the C‐statistic was 0.8834, with an overall accuracy of 0.9292.

In order to train the logistic model, we used a single 24‐hour window of data for each patient. However, in a system that predicts patients' outcomes in real time, scores are recomputed each time new data are entered into the database. Hence, patients have a series of scores over the length of their hospital stay, and an alert is triggered when any one of these scores is above the chosen threshold.

Once the model was developed, we implemented it in an internally developed, Java‐based clinical decision support rules engine, which identified when new data relevant to the model were available in a real‐time central data repository. The rules engine queried the data repository to acquire all data needed to evaluate the model. The score was calculated with each relevant new data point, and an alert was generated when the score exceeded the cut‐point threshold. We then prospectively validated these alerts on patients on 8 general medical wards at Barnes Jewish Hospital. Details regarding the architecture of our clinical decision support system have been previously published.[12] The sensitivity and positive predictive values for ICU transfer for these alerts were tracked during an intervention trial that ran from January 24, 2011, through December 31, 2011. Four general medical wards were randomized to the intervention group and 4 wards were randomized to the control group. The 8 general medical wards were ordered according to their alert rates based upon the historical data from July 2007 through January 2010, creating 4 pairs of wards in ascending order of alert rate. Within each of the 4 pairs, 1 member of the pair was randomized to the intervention group and the other to the control group using a random number generator.

Real‐time automated alerts generated 24 hours per day, 7 days per week from the predictive algorithm were sent to the charge‐nurse pager on the intervention units. Alerts were also generated and stored in the database on the control units, but these alerts were not sent to the charge nurse on those units. The alerts were sent to the charge nurses on the respective wards, as these individuals were thought to be in the best position to perform the initial assessment of the alerted patients, especially during evening hours when physician staffing was reduced. The charge nurses assessed the intervention‐group patients and were instructed to contact the responsible physician (hospitalist or internal medicine house officer) to inform them of the alert, or to call the rapid response team (RRT) if the patient's condition already appeared to be significantly deteriorating.

Descriptive statistics for algorithm sensitivity and positive predictive value and for patient outcomes were performed. Associations between alerts and the primary outcome, ICU transfer, were determined, as well as the impact of alerts in the intervention group compared with the control group, using [2] tests. The same analyses were performed for patient death. Differences in length of stay (LOS) were assessed using the Wilcoxon rank sum test.


Predictive Model

The variables with the greatest coefficients contributing to the PT model included respiratory rate, oxygen saturation, shock index, systolic blood pressure, anticoagulation use, heart rate, and diastolic blood pressure. A complete list of variables is provided in the Appendix (see Supporting Information in the online version of this article). All but 1 are routinely collected vital‐sign measures, and all but 1 occur in the 4‐hour period immediately prior to the alert (bucket 6).

Prospective Trial

Patient characteristics are presented in Table 1. Patients were well matched for race, sex, age, and underlying diagnoses. All alerts reported to the charge nurses were to be associated with a call from the charge nurse to the responsible physician caring for the alerted patient. The mean number of alerts per alerted patient was 1.8 (standard deviation=1.7). Patients meeting the alert threshold were at nearly 5.3‐fold greater risk of ICU transfer (95% confidence interval [CI]: 4.6‐6.0) than those not satisfying the alert threshold (358 of 2353 [15.2%; 95% CI: 13.8%‐16.7%] vs 512 of 17678 [2.9%; 95% CI: 2.7%‐3.2%], respectively; P<0.0001). Patients with alerts were at 8.9‐fold greater risk of death (95% CI: 7.4‐10.7) than those without alerts (244 of 2353 [10.4%; 95% CI: 9.2%‐11.7%] vs 206 of 17678 [1.2%; 95% CI: 1.0%‐1.3%], respectively; P<0.0001). Operating characteristics of the PT from the prospective trial are shown in Table 2. Alerts occurred a median of 25.5 hours prior to ICU transfer (interquartile range, 7.00‐81.75) and 8 hours prior to death (interquartile range, 4.09‐15.66).

Demographics by Study Group
Study Group
Control (N=10,120)Intervention (N=9911)
  • NOTE: No significant differences between study groups. Abbreviations: F, female; ICD‐9, International Classification of Diseases, 9th Revision; IQR, interquartile range; M, male.

Age at discharge, median (IQR), y57 (4469)57 (4470)
Top 10 ICD‐9 descriptions and counts, n (%)
1Diseases of the digestive system1,774 (17.5)Diseases of the digestive system1,664 (16.7)
2Diseases of the circulatory system1,252 (12.4)Diseases of the circulatory system1,253 (12.6)
3Diseases of the respiratory system1,236 (12.2)Diseases of the respiratory system1,210 (12.2)
4Injury and poisoning864 (8.5)Injury and poisoning849 (8.6)
5Endocrine, nutritional, and metabolic diseases, and immunity disorders797 (7.9)Diseases of the genitourinary system795 (8.0)
6Diseases of the genitourinary system762 (7.5)Endocrine, nutritional, and metabolic diseases, and immunity disorders780 (7.9)
7Infectious and parasitic diseases555 (5.5)Infectious and parasitic diseases549 (5.5)
8Neoplasms547 (5.4)Neoplasms465 (4.7)
9Diseases of the blood and blood‐forming organs426 (4.2)Diseases of the blood and blood‐forming organs429 (4.3)
10Symptoms, signs, and ill‐defined conditions and factors influencing health status410 (4.1)Diseases of the musculoskeletal system and connective tissue399 (4.0)
Prediction ToolGenerated Alerts and Outcomes
Sensitivity, %Specificity, %PPV, %NPV, %Positive Likelihood RatioNegative Likelihood Ratio
  • NOTE: Abbreviations: CI, confidence interval; ICU, intensive care unit; NPV, negitive predictive value; PPV, positive predictive value.

ICU TransferYes (N=870)No (N=19,161)
Alert3581,99541.1 (95% CI: 37.944.5)89.6 (95% CI: 89.290.0)15.2 (95% CI: 13.816.7)97.1 (95% CI: 96.897.3)3.95 (95% CI: 3.614.30)0.66 (95% CI: 0.620.70)
No Alert51217,166
DeathYes (N=450)No (N=19,581)
Alert244210954.2 (95% CI: 49.658.8)89.2 (95% CI: 88.889.7)10.4 (95% CI: 9.211.7)98.8 (95% CI: 98.799.0)5.03 (95% CI: 4.585.53)0.51 (95% CI: 0.460.57)
No Alert20617,472

Among patients identified by the PT, there were no differences in the proportion of patients who were transferred to the ICU or who died in the intervention group as compared with the control group (Table 3). In addition, although there was no difference in LOS in the intervention group compared with the control group, identification by the PT was associated with a significantly longer median LOS (7.01 days vs 2.94 days, P<0.001). The largest numbers of patients who were transferred to the ICU or died did so in the first hospital day, and 60% of patients who were transferred to the ICU did so in the first 4 days, whereas deaths were more evenly distributed across the hospital stay.

Outcomes (ICU Transfer, Mortality, and LOS) by Study Group and Alert
Outcomes by Alert Statusa
Alert Study GroupNo‐Alert Study Group
Intervention, N=1194Control, N=1159Intervention, N=8717Control, N=8961
  • NOTE: No significant differences between study groups. Abbreviations: ICU, intensive care unit; IQR, interquartile range; LOS, length of stay

  • LOS significantly differed by alert status (P<0.01).

ICU Transfer
LOS from admit to discharge, median (IQR), da7.07 (3.9912.15)6.92 (3.8212.67)2.97 (1.775.33)2.91 (1.745.19)


We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).

Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.

We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.

As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.

The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]

Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).

An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]

In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.


Disclosures: This study was funded in part by the Barnes‐Jewish Hospital Foundation and by Grant No. UL1 RR024992 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and the NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. ClinicalTrials.gov Identifier: NCT01280942. The authors report no conflicts of interest.


Online-Only Materials

   Comments ()