Original Research

Mortality predictions on admission as a context for organizing care activities

Abstract

BACKGROUND

Favorable health outcomes are more likely to occur when the clinical team recognizes patients at risk and intervenes in consort. Prediction rules can identify high‐risk subsets, but the availability of multiple rules for various conditions present implementation and assimilation challenges.

METHODS

A prediction rule for 30‐day mortality at the beginning of the hospitalization was derived in a retrospective cohort of adult inpatients from a community hospital in the Midwestern United States from 2008 to 2009, using clinical laboratory values, past medical history, and diagnoses present on admission. It was validated using 2010 data from the same and from a different hospital. The calculated mortality risk was then used to predict unplanned transfers to intensive care units, resuscitation attempts for cardiopulmonary arrests, a condition not present on admission (complications), intensive care unit utilization, palliative care status, in‐hospital death, rehospitalizations within 30 days, and 180‐day mortality.

RESULTS

The predictions of 30‐day mortality for the derivation and validation datasets had areas under the receiver operating characteristic curve of 0.88. The 30‐day mortality risk was in turn a strong predictor for in‐hospital death, palliative care status, 180‐day mortality; a modest predictor for unplanned transfers and cardiopulmonary arrests; and a weaker predictor for the other events of interest.

CONCLUSIONS

The probability of 30‐day mortality provides health systems with an array of prognostic information that may provide a common reference point for organizing the clinical activities of the many health professionals involved in the care of the patient. Journal of Hospital Medicine 2013;8:229–235 © 2012 Society of Hospital Medicine

Copyright © 2012 Society of Hospital Medicine

Favorable health outcomes are more likely to occur when the healthcare team quickly identifies and responds to patients at risk.[1, 2, 3] However, the treatment process can break down during handoffs if the clinical condition and active issues are not well communicated.[4] Patients whose decline cannot be reversed also challenge the health team. Many are referred to hospice late,[5] and some do not receive the type of end‐of‐life care matching their preferences.[6]

Progress toward the elusive goal of more effective and efficient care might be made via an industrial engineering approach, mass customization, in which bundles of services are delivered based on the anticipated needs of subsets of patients.[7, 8] An underlying rationale is the frequent finding that a small proportion of individuals experiences the majority of the events of interest, commonly referenced as the Pareto principle.[7] Clinical prediction rules can help identify these high‐risk subsets.[9] However, as more condition‐specific rules become available, the clinical team faces logistical challenges when attempting to incorporate these into practice. For example, which team member will be responsible for generating the prediction and communicating the level of risk? What actions should follow for a given level of risk? What should be done for patients with conditions not addressed by an existing rule?

In this study, we present our rationale for health systems to implement a process for generating mortality predictions at the time of admission on most, if not all, adult patients as a context for the activities of the various clinical team members. Recent studies demonstrate that in‐hospital or 30‐day mortality can be predicted with substantial accuracy using information available at the time of admission.[10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Relationships are beginning to be explored among the risk factors for mortality and other outcomes such as length of stay, unplanned transfers to intensive care units, 30‐day readmissions, and extended care facility placement.[10, 20, 21, 22] We extend this work by examining how a number of adverse events can be understood through their relationship with the risk of dying. We begin by deriving and validating a new mortality prediction rule using information feasible for our institution to use in its implementation.

METHODS

The prediction rule was derived from data on all inpatients (n = 56,003) 18 to 99 years old from St. Joseph Mercy Hospital, Ann Arbor from 2008 to 2009. This is a community‐based, tertiary‐care center. We reference derivation cases as D1, validation cases from the same hospital in the following year (2010) as V1, and data from a second hospital in 2010 as V2. The V2 hospital belonged to the same parent health corporation and shared some physician specialists with D1 and V1 but had separate medical and nursing staff.

The primary outcome predicted is 30‐day mortality from the time of admission. We chose 30‐day rather than in‐hospital mortality to address concerns of potential confounding of duration of hospital stay and likelihood of dying in the hospital.[23] Risk factors were considered for inclusion into the prediction rule based on their prevalence, conceptual, and univariable association with death (details provided in the Supporting information, Appendix I and II, in the online version of this article). The types of risk factors considered were patient diagnoses as of the time of admission obtained from hospital administrative data and grouped by the 2011 Clinical Classification Software (http://www.hcupus.ahrq.gov/toolssoftware/ccs/ccs.jsp#download, accessed June 6, 2012), administrative data from previous hospitalizations within the health system in the preceding 12 months, and the worst value of clinical laboratory blood tests obtained within 30 days prior to the time of admission. When a given patient had missing values for the laboratory tests of interest, we imputed a normal value, assuming the clinician had not ordered these tests because he/she expected the patient would have normal results. The imputed normal values were derived from available results from patients discharged alive with short hospital stays (3 days) in 2007 to 2008. The datasets were built and analyzed using SAS version 9.1, 9.2 (SAS Institute, Inc., Cary, NC) and R (R Foundation for Statistical Computing, Vienna, Austria; http://www.R‐project.org).

Prediction Rule Derivation Using D1 Dataset

Random forest procedures with a variety of variable importance measures were used with D1 data to reduce the number of potential predictor variables.[24] Model‐based recursive partitioning, a technique that combines features of multivariable logistic regression and classification and regression trees, was then used to develop the multivariable prediction model.[25, 26] Model building was done in R, employing functions provided as part of the randomForest and party packages. The final prediction rule consisted of 4 multivariable logistic regression models, each being specific to 1 of 4 possible population subgroups: females with/females without previous hospitalizations, and males with/males without previous hospitalizations. Each logistic regression model contains exactly the same predictor variables; however, the regression coefficients are subgroup specific. Therefore, the predicted probability of 30‐day mortality for a patient having a given set of predictor variables depends on the subgroup to which the patient is a member.

Validation, Discrimination, Calibration

The prediction rule was validated by generating a predicted probability of 30‐day mortality for each patient in V1 and V2, using their observed risk factor information combined with the scoring weights (ie, regression coefficients) derived from D1, then comparing predicted vs actual outcomes. Discriminatory accuracy is reported as the area under the receiver operating characteristic (ROC) curve that can range from 0.5 indicating pure chance, to 1.0 or perfect prediction.[27] Values above 0.8 are often interpreted as indicating strong predictive relationships, values between 0.7 and 0.79 as modest, and values between 0.6 and 0.69 as weak.[28] Model calibration was tested in all datasets across 20 intervals representing the spectrum of mortality risk, by assessing whether or not the 95% confidence limits for the actual proportion of patients dying encompassed the mean predicted mortality for the interval. These 20 intervals were defined using 5 percentile increments of the probability of dying for D1. The use of intervals based on percentiles ensures similarity in the level of predicted risk within an interval for V1 and V2, while allowing the proportion of patients contained within that interval to vary across hospitals.

Relationships With Other Adverse Events

We then used each patient's calculated probability of 30‐day mortality to predict the occurrence of other adverse events. We first derived scoring weights (ie, regression parameter estimates) from logistic regression models designed to relate each secondary outcome to the predicted 30‐day mortality using D1 data. These scoring weights were then respectively applied to the V1 and V2 patients' predicted 30‐day mortality rate to generate their predicted probabilities for: in‐hospital death, a stay in an intensive care unit at some point during the hospitalization, the occurrence of a condition not present on admission (a complication, see the Supporting information, Appendix I, in the online version of this article), palliative care status at the time of discharge (International Classification of Diseases, 9th Revision code V66.7), 30‐day readmission, and death within 180 days (determined for the first hospitalization of the patient in the calendar year, using hospital administrative data and the Social Security Death Index). Additionally, for V1 patients but not V2 due to unavailability of data, we predicted the occurrence of an unplanned transfer to an intensive care unit within the first 24 hours for those not admitted to the intensive care unit (ICU), and resuscitative efforts for cardiopulmonary arrests (code blue, as determined from hospital paging records and resuscitation documentation, with the realization that some resuscitations within the intensive care units might be undercaptured by this approach). Predicted vs actual outcomes were assessed using SAS version 9.2 by examining the areas under the receiver operating curves generated by the PROC LOGISTIC ROC.

Implications for Care Redesign

To illustrate how the mortality prediction provides a context for organizing the work of multiple health professionals, we created 5 risk strata[10] based on quintiles of D1 mortality risk. To display the time frame in which the peak risk of death occurs, we plotted the unadjusted hazard function per strata using SAS PROC LIFETEST.

RESULTS

Table 1 displays the risk factors used in the 30‐day mortality prediction rule, their distribution in the populations of interest, and the frequency of the outcomes of interest. The derivation (D1) and validation (V1) populations were clinically similar; the patients of hospital V2 differed in the proportion of risk factors and outcomes. The scoring weights or parameter estimates for the risk factors are given in the Appendix (see Supporting Information, Appendix I, in the online version of this article).

Demographics, Risk Factors, and Outcomes
Hospital AHospital V2
D1 Derivation, N = 56,003V1 Validation, N = 28,441V2 Validation, N = 14,867
  • NOTE: Abbreviations: ICU, intensive care unit; NA, not applicable.
The 24 risk factors used in the prediction rule
Age in years, mean (standard deviation)59.8 (19.8)60.2 (19.8)66.4 (20.2)
Female33,185 (59.3%)16,992 (59.7%)8,935 (60.1%)
Respiratory failure on admission2,235 (4.0%)1,198 (4.2%)948 (6.4%)
Previous hospitalization19,560 (34.9%)10,155 (35.7%)5,925 (39.9%)
Hospitalization billed as an emergency admission[38]30,116 (53.8%)15,445 (54.3%)11,272 (75.8%)
Admitted to medicine service29,472 (52.6%)16,260 (57.2%)11,870 (79.8%)
Heart failure at the time of admission7,558 (13.5%)4,046 (14.2%)2,492 (16.8%)
Injury such as fractures or trauma at the time of admission7,007 (12.5%)3,612 (12.7%)2,205 (14.8%)
Sepsis at the time of admission2,278 (4.1%)1,025 (3.6%)850 (5.7%)
Current or past atrial fibrillation8,329 (14.9%)4,657 (16.4%)2,533 (17.0%)
Current or past metastatic cancer2,216 (4.0%)1,109 (3.9%)428 (2.9%)
Current or past cancer without metastases5,260 (9.34%)2,668 (9.4%)1,248 (8.4%)
Current or past history of leukemia or lymphoma1,025 (1.8%)526 (1.9%)278 (1.9%)
Current or past cognitive deficiency3,708 (6.6%)1,973 (6.9%)2,728 (18.4%)
Current or past history of other neurological conditions (such as Parkinson's disease, multiple sclerosis, epilepsy, coma, stupor, brain damage)4,671 (8.3%)2,537 (8.9%)1,606 (10.8%)
Maximum serum blood urea nitrogen (mg/dL), continuous21.9 (15.1)21.8 (15.1)25.9 (18.2)
Maximum white blood count (1,000/UL), continuous2.99 (4.00)3.10 (4.12)3.15 (3.81)
Minimum platelet count (1,000/UL), continuous240.5 (85.5)228.0 (79.6)220.0 (78.6)
Minimum hemoglobin (g/dL), continuous12.3 (1.83)12.3 (1.9)12.1 (1.9)
Minimum serum albumin (g/dL) <3.14, binary indicator11,032 (19.7%)3,848 (13.53%)2,235 (15.0%)
Minimum arterial pH <7.3, binary indicator1,095 (2.0%)473 (1.7%)308 (2.1%)
Minimum arterial pO2 (mm Hg) <85, binary indicator1,827 (3.3%)747 (2.6%)471 (3.2%)
Maximum serum troponin (ng/mL) >0.4, binary indicator6,268 (11.2%)1,154 (4.1%)2,312 (15.6%)
Maximum serum lactate (mEq/L) >4.0, binary indicator533 (1.0%)372 (1.3%)106 (0.7%)
Outcomes of interest
30‐day mortalityprimary outcome of interest2,775 (5.0%)1,412 (5.0%)1,193 (8.0%)
In‐hospital mortality1,392 (2.5%)636 (2.2%)467 (3.1%)
180‐day mortality (deaths/first hospitalization for patient that year)2,928/38,995 (7.5%)1,657/21,377 (7.8%)1,180/10,447 (11.3%)
Unplanned transfer to ICU within first 24 hours/number of patients with data not admitted to ICU434/46,647 (0.9%)276/25,920 (1.1%)NA
Ever in ICU during hospitalization/those with ICU information available5,906/55,998 (10.6%)3,191/28,429 (11.2%)642/14,848 (4.32%)
Any complication6,768 (12.1%)2,447 (8.6%)868 (5.8%)
Cardiopulmonary arrest228 (0.4%)151 (0.5%)NA
Patients discharged with palliative care V code1,151 (2.1%)962 (3.4%)340 (2.3%)
30‐day rehospitalization/patients discharged alive6,616/54,606 (12.1%)3,602/27,793 (13.0%)2,002/14,381 (13.9%)

Predicting 30‐Day Mortality

The areas under the ROC (95% confidence interval [CI]) for the D1, V1, and V2 populations were 0.876 (95% CI, 0.870‐0.882), 0.885 (95% CI, 0.877‐0.893), and 0.883 (95% CI, 0.875‐0.892), respectively. The calibration curves for all 3 populations are shown in Figure 1. The overlap of symbols indicates that the level of predicted risk matched actual mortality for most intervals, with slight underprediction for those in the highest risk percentiles.

Figure 1

Calibration. The horizontal axis displays 20 intervals of risk, containing 5‐percentile increments of the predicted mortality based on the D1 population. The vertical axis displays the actual proportion of patients within the interval who died within 30 days. The cluster of 3 symbols represent the mean predicted chance of dying for the derivation and 2 validation populations, respectively. The crosshatches represent the actual proportion of patients within each interval who died, with the 95% binomial confidence limits represented by the length of the vertical bar. The 20 intervals (named for the highest percentile within the interval) with corresponding probabilities of death: 5th percentile (probability 0‐0.0008); 10th percentile (probability 0.0008‐0.0011); 15th percentile (probability 0.0011‐0.0021); 20 (0.0021‐0.0033); 25 (0.0033‐0.0049); 30 (0.0049‐0.0067); 35 (0.0067‐0.0087); 40 (0.0087‐0.0108); 45 (0.0108‐0.0134); 50 (0.0134‐0.0165); 55 (0.0165‐0.0201); 60 (0.0201‐0.0247); 65 (0.0247‐0.0308); 70 (0.0308‐0.0392); 75 (0.0392‐0.0503); 80 (0.0503‐0.0669); 85 (0.0669‐0.0916); 90 (0.0916‐0.1308); 95 (0.1308‐0.2186); 100 (0.2186‐1.0).

Example of Risk Strata

Figure 2 displays the relationship between the predicted probability of dying within 30 days and the outcomes of interest for V1, and illustrates the Pareto principle for defining high‐ and low‐risk subgroups. Most of the 30‐day deaths (74.7% of D1, 74.2% of V1, and 85.3% of V2) occurred in the small subset of patients with a predicted probability of death exceeding 0.067 (the top quintile of risk of D1, the top 18 % of V1, and the top 29.8% of V2). In contrast, the mortality rate for those with a predicted risk of 0.0033 was 0.02% for the lowest quintile of risk in D1, 0.07% for the 19.3% having the lowest risk in V1, and 0% for the 9.7% of patients with the lowest risk in V2. Figure 3 indicates that the risk for dying peaks within the first few days of the hospitalization. Moreover, those in the high‐risk group remained at elevated risk relative to the lower risk strata for at least 100 days.

Figure 2

Risk of outcomes within intervals of mortality risk (validation hospital V1). The curves for the other 2 populations (D1, V2) were similar (see the Supporting information, Appendix II, in the online version of this article). Examples of possible risk strata are indicated.

Figure 3

Instantaneous risk of death (hazard function) following hospital admission—validation hospital V1. For sake of clarity, 5 ordinal categories of predicted risk are shown. The curves for the other 2 populations (D1, V2) were similar and are shown in the Appendix II (see the Supporting information, Appendix I, in the online version of this article).

Relationships With Other Outcomes of Interest

The graphical curves of Figure 2 represent the occurrence of adverse events. The rising slopes indicate the risk for other events increases with the risk of dying within 30 days (for details and data for D1 and V2, see the Supporting Information, Appendix II, in the online version of this article). The strength of these relationships is quantified by the areas under the ROC curve (Table 2). The probability of 30‐day mortality strongly predicted the occurrence of in‐hospital death, palliative care status, and death within 180 days; modestly predicted having an unplanned transfer to an ICU within the first 24 hours of the hospitalization and undergoing resuscitative efforts for cardiopulmonary arrest; and weakly predicted intensive care unit use at some point in the hospitalization, occurrence of a condition not present on admission (complication), and being rehospitalized within 30 days

Area Under the Receiver Operating Characteristic Curve Models Predicting Secondary Outcomes of Interest
OutcomeHospital AHospital V2
D1DerivationV1ValidationV2Validation
  • NOTE: Mann‐Whitney (95% Wald confidence limits). Each outcome of interest was predicted by the patients' calculated probability of dying within 30 days and its logarithm. Details are provided in the Appendix II. Abbreviations: ICU, intensive care unit; NA, not applicable.
Unplanned transfer to an ICU within the first 24 hours (for those not admitted to an ICU)0.712 (0.690‐0.734)0.735 (0.709‐0.761)NA
Resuscitation efforts for cardiopulmonary arrest0.709 (0.678‐0.739)0.737 (0.700‐0.775)NA
ICU stay at some point during the hospitalization0.659 (0.652‐0.666)0.663 (0.654‐0.672)0.702 (0.682‐0.722)
Intrahospital complication (condition not present on admission)0.682 (0.676‐0.689)0.624 (0.613‐0.635)0.646 (0.628‐0.664)
Palliative care status0.883 (0.875‐0.891)0.887 (0.878‐0.896)0.900 (0.888‐0.912)
Death within hospitalization0.861 (0.852‐0.870)0.875 (0.862‐0.887)0.880 (0.866‐0.893)
30‐day readmission0.685 (0.679‐0.692)0.685 (0.676‐0.694)0.677 (0.665‐0.689)
Death within 180 days0.890 (0.885‐0.896)0.889 (0.882‐0.896)0.873 (0.864‐0.883)

DISCUSSION

The primary contribution of our work concerns the number and strength of associations between the probability of dying within 30 days and other events, and the implications for organizing the healthcare delivery model. We also add to the growing evidence that death within 30 days can be accurately predicted at the time of admission from demographic information, modest levels of diagnostic information, and clinical laboratory values. We developed a new prediction rule with excellent accuracy that compares well to a rule recently developed by the Kaiser Permanente system.[13, 14] Feasibility considerations are likely to be the ultimate determinant of which prediction rule a health system chooses.[13, 14, 29] An independent evaluation of the candidate rules applied to the same data is required to compare their accuracy.

These results suggest a context for the coordination of clinical care processes, although mortality risk is not the only domain health systems must address. For illustrative purposes, we will refer to the risk strata shown in Figure 2. After the decisions to admit the patient to the hospital and whether or not surgical intervention is needed, the next decision concerns the level and type of nursing care needed.[10] Recent studies continue to show challenges both with unplanned transfers to intensive care units[21] and care delivered that is consistently concordant with patient wishes.[6, 30] The level of risk for multiple adverse outcomes suggests stratum 1 patients would be the priority group for perfecting the placement and preference assessment process. Our institution is currently piloting an internal placement guideline recommending that nonpalliative patients in the top 2.5 percentile of mortality risk be placed initially in either an intensive or intermediate care unit to receive the potential benefit of higher nursing staffing levels.[31] However, mortality risk cannot be the only criterion used for placement, as demonstrated by its relatively weak association with overall ICU utilization. Our findings may reflect the role of unmeasured factors such as the need for mechanical ventilation, patient preference for comfort care, bed availability, change in patient condition after admission, and inconsistent application of admission criteria.[17, 21, 32, 33, 34]

After the placement decision, the team could decide if the usual level of monitoring, physician rounding, and care coordination would be adequate for the level of risk or whether an additional anticipatory approach is needed. The weak relationship between the risk of death and incidence of complications, although not a new finding,[35, 36] suggests routine surveillance activities need to be conducted on all patients regardless of risk to detect a complication, but that a rescue plan be developed in advance for high mortality risk patients, for example strata 1 and 2, in the event they should develop a complication.[36] Inclusion of the patient's risk strata as part of the routine hand‐off communication among hospitalists, nurses, and other team members could provide a succinct common alert for the likelihood of adverse events.

The 30‐day mortality risk also informs the transition care plan following hospitalization, given the strong association with death in 180 days and the persistent level of this risk (Figure 3). Again, communication of the risk status (stratum 1) to the team caring for the patient after the hospitalization provides a common reference for prognosis and level of attention needed. However, the prediction accuracy is not sufficient to refer high‐risk patients into hospice, but rather, to identify the high‐risk subset having the most urgent need to have their preferences for future end‐of‐life care understood and addressed. The weak relationship of mortality risk with 30‐day readmissions indicates that our rule would have a limited role in identifying readmission risk per se. Others have noted the difficulty in accurately predicting readmissions, most likely because the underlying causes are multifactorial.[37] Our results suggest that 1 dynamic for readmission is the risk of dying, and so the underlying causes of this risk should be addressed in the transition plan.

There are a number of limitations with our study. First, this rule was developed and validated on data from only 2 institutions, assembled retrospectively, with diagnostic information determined from administrative data. One cannot assume the accuracy will carry over to other institutions[29] or when there is diagnostic uncertainty at the time of admission. Second, the 30‐day mortality risk should not be used as the sole criterion for determining the service intensity for individual patients because of issues with calibration, interpretation of risk, and confounding. The calibration curves (Figure 2) show the slight underprediction of the risk of dying for high‐risk groups. Other studies have also noted problems with precise calibration in validation datasets.[13, 14] Caution is also needed in the interpretation of what it means to be at high risk. Most patients in stratum 1 were alive at 30 days; therefore, being at high risk is not a death sentence. Furthermore, the relative weights of the risk factors reflect (ie, are confounded by) the level of treatment rendered. Some deaths within the higher‐risk percentiles undoubtedly occurred in patients choosing a palliative rather than a curative approach, perhaps partially explaining the slight underprediction of deaths. Conversely, the low mortality experienced by patients within the lower‐risk strata may indicate the treatment provided was effective. Low mortality risk does not imply less care is needed.

A third limitation is that we have not defined the thresholds of risk that should trigger placement and care intensity, although we provide examples on how this could be done. Each institution will need to calibrate the thresholds and associated decision‐making processes according to its own environment.[14] Interested readers can explore the sensitivity and specificity of various thresholds\ by using the tables in the Appendix (see the Supporting information, Appendix II, in the online version of this article). Finally, we do not know if identifying the mortality risk on admission will lead to better outcomes[19, 29]

CONCLUSIONS

Death within 30 days can be predicted with information known at the time of admission, and is associated with the risk of having other adverse events. We believe the probability of death can be used to define strata of risk that provide a succinct common reference point for the multidisciplinary team to anticipate the clinical course of subsets of patients and intervene with proportional intensity.

Acknowledgments

This work benefited from multiple conversations with Patricia Posa, RN, MSA, Elizabeth Van Hoek, MHSA, and the Redesigning Care Task Force of St. Joseph Mercy Hospital, Ann Arbor, Michigan.

Disclosure: Nothing to report.

References

Online-Only Materials

AttachmentSize
File Supporting Information (1)40.69 KB
   Comments ()