Original Research

Inpatient safety outcomes following the 2011 residency work‐hour reform

Abstract

BACKGROUND

The impact of the 2011 residency work‐hour reforms on patient safety is not known.

OBJECTIVE

To evaluate the association between implementation of the 2011 reforms and patient safety outcomes at a large academic medical center.

DESIGN

Observational study using difference‐in‐differences estimation strategy to evaluate whether safety outcomes improved among patients discharged from resident and hospitalist (nonresident) services before (2008–2011) and after (2011–2012) residency work‐hour changes.

PATIENTS

All adult patients discharged from general medicine services from July 2008 through June 2012.

MEASUREMENTS

Outcomes evaluated included length of stay, 30‐day readmission, intensive care unit (ICU) admission, inpatient mortality, and presence of Maryland Hospital Acquired Conditions. Independent variables included time period (pre‐ vs postreform), resident versus hospitalist service, patient age at admission, race, gender, and case mix index.

RESULTS

Patients discharged from the resident services in the postreform period had higher likelihood of an ICU stay (5.7% vs 4.5%, difference 1.4%; 95% confidence interval [CI]: 0.5% to 2.2%), and lower likelihood of 30‐day readmission (17.2% vs 20.1%, difference 2.8%; 95 % CI: 1.3 to 4.3%) than patients discharged from the resident services in the prereform period. Comparing pre‐ and postreform periods on the resident and hospitalist services, there were no significant differences in patient safety outcomes.

CONCLUSIONS

In the first year after implementation of the 2011 work‐hour reforms relative to prior years, we found no change in patient safety outcomes in patients treated by residents compared with patients treated by hospitalists. Further study of the long‐term impact of residency work‐hour reforms is indicated to ensure improvement in patient safety. Journal of Hospital Medicine 2014;9:347–352. © 2014 Society of Hospital Medicine

© 2014 Society of Hospital Medicine

The Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements implemented in July 2011 increased supervision requirements and limited continuous work hours for first‐year residents.[1] Similar to the 2003 mandates, these requirements were introduced to improve patient safety and education at academic medical centers.[2] Work‐hour reforms have been associated with decreased resident burnout and improved sleep.[3, 4, 5] However, national observational studies and systematic reviews of the impact of the 2003 reforms on patient safety and quality of care have been varied in terms of outcome.[6, 7, 8, 9, 10] Small studies of the 2011 recommendations have shown increased sleep duration and decreased burnout, but also an increased number of handoffs and increased resident concerns about making a serious medical error.[11, 12, 13, 14] Although national surveys of residents and program directors have not indicated improvements in education or quality of life, 1 observational study did show improvement in clinical exposure and conference attendance.[15, 16, 17, 18] The impact of the 2011 reforms on patient safety remains unclear.[19, 20]

The objective of this study was to evaluate the association between implementation of the 2011 residency work‐hour mandates and patient safety outcomes at a large academic medical center.

METHODS

Study Design

This observational study used a quasi‐experimental difference‐in‐differences approach to evaluate whether residency work‐hour changes were associated with patient safety outcomes among general medicine inpatients. We compared safety outcomes among adult patients discharged from resident general medical services (referred to as resident) to safety outcomes among patients discharged by the hospitalist general medical service (referred to as hospitalist) before and after the 2011 residency work‐hour reforms at a large academic medical center. Differences in outcomes for the resident group were compared to differences observed in the hospitalist group, with adjustment for relevant demographic and case mix factors.[21] We used the hospitalist service as a control group, because ACGME changes applied only to resident services. The strength of this design is that it controls for secular trends that are correlated with patient safety, impacting both residents and hospitalists similarly.[9]

Approval for this study and a Health Insurance Portability and Accountability Act waiver were granted by the Johns Hopkins University School of Medicine institutional review board. We retrospectively examined administrative data on all patient discharges from the general medicine services at Johns Hopkins Hospital between July 1, 2008 and June 30, 2012 that were identified as pertaining to resident or hospitalist services.

Patient Allocation and Physician Scheduling

Patient admission to the resident or hospitalist service was decided by a number of factors. To maintain continuity of care, patients were preferentially admitted to the same service as for prior admissions. New patients were admitted to a service based on bed availability, nurse staffing, patient gender, isolation precautions, and cardiac monitor availability.

The inpatient resident services were staffed prior to July 2011 using a traditional 30‐hour overnight call system. Following July 2011, the inpatient resident services were staffed using a modified overnight call system, in which interns took overnight calls from 8 pm until 12 pm the following day, once every 5 nights with supervision by upper‐level residents. These interns rotated through daytime admitting and coverage roles on the intervening days. The hospitalist service was organized into a 3‐physician rotation of day shift, evening shift, and overnight shift.

Data and Outcomes

Twenty‐nine percent of patients in the sample were admitted more than once during the study period, and patients were generally admitted to the same resident team during each admission. Patients with multiple admissions were counted multiple times in the model. We categorized admissions as prereform (July 1, 2008June 30, 2011) and postreform (July 1, 2011June 30, 2012). Outcomes evaluated included hospital length of stay, 30‐day readmission, intensive care unit stay (ICU) stay, inpatient mortality, and number of Maryland Hospital Acquired Conditions (MHACs). ICU stay pertained to any ICU admission including initial admission and transfer from the inpatient floor. MHACs are a set of inpatient performance indicators derived from a list of 64 inpatient Potentially Preventable Complications developed by 3M Health Information Systems.[22] MHACs are used by the Maryland Health Services Cost Review Commission to link hospital payment to performance for costly, preventable, and clinically relevant complications. MHACs were coded in our analysis as a dichotomous variable. Independent variables included patient age at admission, race, gender, and case mix index. Case mix index (CMI) is a numeric score that measures resource utilization for a specific patient population. CMI is a weighted value assigned to patients based on resource utilization and All Patient Refined Diagnostic Related Group and was included as an indicator of patient illness severity and risk of mortality.[23] Data were obtained from administrative records from the case mix research team at Johns Hopkins Medicine.

To account for transitional differences that may have coincided with the opening of a new hospital wing in late April 2012, we conducted a sensitivity analysis, in which we excluded from analysis any visits that took place in May 2012 to June 2012.

Data Analysis

Based on historical studies, we calculated that a sample size of at least 3600 discharges would allow us to detect a difference of 5% between the pre‐ and postreform period assuming baseline 20% occurrence of dichotomous outcomes (=0.05; =0.2; r=4).[21]

The primary unit of analysis was the hospital discharge. Similar to Horwitz et al., we analyzed data using a difference‐in‐differences estimation strategy.[21] We used multivariable linear regression for length of stay measured as a continuous variable, and multivariable logistic regression for inpatient mortality, 30‐day readmission, MHACs coded as a dichotomous variable, and ICU stay coded as a dichotomous variable.[9] The difference‐in‐differences estimation was used to determine whether the postreform period relative to prereform period was associated with differences in outcomes comparing resident and hospitalist services. In the regression models, the independent variables of interest included an indicator variable for whether a patient was treated on a resident service, an indicator variable for whether a patient was discharged in the postreform period, and the interaction of these 2 variables (resident*postreform). The interaction term can be interpreted as a differential change over time comparing resident and hospitalist services. In all models, we adjusted for patient age, gender, race, and case mix index.

To determine whether prereform trends were similar among the resident and hospitalist services, we performed a test of controls as described by Volpp and colleagues.[6] Interaction terms for resident service and prereform years 2010 and 2011 were added to the model. A Wald test was then used to test for improved model fit, which would indicate differential trends among resident and hospitalist services during the prereform period. Where such trends were found, postreform results were compared only to 2011 rather than the 2009 to 2011 prereform period.[6]

To account for correlation within patients who had multiple discharges, we used a clustering approach and estimated robust variances.[24] From the regression model results, we calculated predicted probabilities adjusted for relevant covariates and prepost differences, and used linear probability models to estimate percentage‐point differences in outcomes, comparing residents and hospitalists in the pre‐ and postreform periods.[25] All analyses were performed using Stata/IC version 11 (StataCorp, College Station, TX).

RESULTS

In the 3 years before the 2011 residency work‐hour reforms were implemented (prereform), there were a total of 15,688 discharges for 8983 patients to the resident services and 4622 discharges for 3649 patients to the hospitalist services. In the year following implementation of residency work‐hour changes (postreform), there were 5253 discharges for 3805 patients to the resident services and 1767 discharges for 1454 patients to the hospitalist service. Table 1 shows the characteristics of patients discharged from the resident and hospitalist services in the pre‐ and postreform periods. Patients discharged from the resident services were more likely to be older, male, African American, and have a higher CMI.

Demographics and Case Mix Index of Patients Discharged From Resident and Hospitalist (Nonresident) General Medicine Services 20092012 at Johns Hopkins Hospital
 Resident ServicesHospitalist Service 
 20092010201120122009201020112012P Valuea
  • NOTE: Abbreviations: SD, standard deviation.

  • Comparing patients admitted to resident versus hospitalist service over the length of the study period 2009 to 2012. Case mix index range for this sample was 0.2 to 21.9 (SD 0.9). Higher case mix index indicates higher risk of mortality.

Discharges, n53455299504452531366149217641767 
Unique patients, n30822968293338051106118013631454 
Age, y, mean (SD)55.1 (17.7)55.7 (17.4)56.4 (17.9)56.7 (17.1)55.9 (17.9)56.2 (18.4)55.5 (18.8)54 (18.7)0.02
Sex male, n (%)1503 (48.8)1397 (47.1)1432 (48.8)1837 (48.3)520 (47)550 (46.6)613 (45)654 (45)<0.01
Race         
African American, n (%)2072 (67.2)1922 (64.8)1820 (62.1)2507 (65.9)500 (45.2)592 (50.2)652 (47.8)747 (51.4)<0.01
White, n (%)897 (29.1)892 (30.1)957 (32.6)1118 (29.4)534 (48.3)527 (44.7)621 (45.6)619 (42.6) 
Asian, n (%)19 (.6%)35 (1.2)28 (1)32 (.8)11 (1)7 (.6)25 (1.8)12 (.8) 
Other, n (%)94 (3.1)119 (4)128 (4.4)148 (3.9)61 (5.5)54 (4.6)65 (4.8)76 (5.2) 
Case mix index, mean (SD)1.2 (1)1.1 (0.9)1.1 (0.9)1.1 (1.2)1.2 (1)1.1 (1)1.1 (1)1 (0.7)<0.01

Differences in Outcomes Among Resident and Hospitalist Services Pre‐ and Postreform

Table 2 shows unadjusted results. Patients discharged from the resident services in the postreform period as compared to the prereform period had a higher likelihood of an ICU stay (5.9% vs 4.5%, P<0.01), and lower likelihood of 30‐day readmission (17.1% vs 20.1%, P<0.01). Patients discharged from the hospitalist service in the postreform period as compared to the prereform period had a significantly shorter mean length of stay (4.51 vs 4.88 days, P=0.03)

Unadjusted Patient Safety Outcomes by Year and Service
 Resident ServicesHospitalist Service
OutcomePrereformaPostreformP ValuePrereformaPostreformP Value
  • NOTE: Abbreviations: ICU, intensive care unit; MHACs, Maryland Hospital Acquired Conditions.

  • For the outcomes length of stay and ICU admission, the postreform period was compared to 2011 only. For MHACs, readmissions, and mortality the postreform period was compared to 2009 to 2011.

Length of stay (mean)4.55 (5.39)4.50 (5.47)0.614.88 (5.36)4.51 (4.64)0.03
Any ICU stay (%)225 (4.5%)310 (5.9%)<0.0182 (4.7%)83 (4.7%)0.95
Any MHACs (%)560 (3.6%)180 (3.4%)0.62210 (4.5%)64 (3.6%)0.09
Readmit in 30 days (%)3155 (20.1%)900 (17.1%)<0.01852 (18.4%)296 (16.8%)0.11
Inpatient mortality (%)71 (0.5%)28 (0.5%)0.4818 (0.4%)7 (0.4%)0.97

Table 3 presents the results of regression analyses examining correlates of patient safety outcomes, adjusted for age, gender, race, and CMI. As the test of controls indicated differential prereform trends for ICU admission and length of stay, the prereform period was limited to 2011 for these outcomes. After adjustment for covariates, the probability of an ICU stay remained greater, and the 30‐day readmission rate was lower among patients discharged from resident services in the postreform period than the prereform period. Among patients discharged from the hospitalist services, there were no significant differences in length of stay, readmissions, ICU admissions, MHACs, or inpatient mortality comparing the pre‐ and postreform periods.

Adjusted Changes in Patient Safety Outcomes by Year and Service
 Resident ServicesHospitalist ServiceDifference in Differences
OutcomePrereformaPostreformDifferencePrereformPostreformDifference(ResidentHospitalist)
  • NOTE: Predicted probabilities and 95% confidence intervals were obtained via margins command. Logistic regression was used for dichotomous outcomes and linear regression for continuous outcomes, adjusted for case mix index, age, race, gender, and clustering at patient level.

  • Abbreviations: ICU, intensive care unit; MHACs, Maryland Hospital Acquired Conditions.

  • For the outcomes length of stay and ICU admission, the postreform period was compared to 2011 only. For MHACs, readmissions, and mortality, the postreform period was compared to 2009 to 2011.

ICU stay4.5% (4.0% to 5.1%)5.7% (5.1% to 6.3%)1.4% (0.5% to 2.2%)4.4% (3.5% to 5.3%)5.3% (4.3% to 6.3%)1.1% (0.2 to 2.4%)0.3% (1.1% to 1.8%)
Inpatient mortality0.5% (0.4% to 0.6%)0.5% (0.3% to 0.7%)0 (0.2% to 0.2%)0.3% (0.2% to 0.6%)0.5% (0.1% to 0.8%)0.1% (0.3% to 0.5%)0.1% (0.5% to 0.3%)
MHACs3.6% (3.3% to 3.9%)3.3% (2.9% to 3.7%)0.4% (0.9 to 0.2%)4.5% (3.9% to 5.1%)4.1% (3.2% to 5.1%)0.3% (1.4% to 0.7%)0.2% (1.0% to 1.3%)
Readmit 30 days20.1% (19.1% to 21.1%)17.2% (15.9% to 18.5%)2.8% (4.3% to 1.3%)18.4% (16.5% to 20.2%)16.6% (14.7% to 18.5%)1.7% (4.1% to 0.8%)1.8% (0.2% to 3.7%)
Length of stay4.6 (4.4 to 4.7)4.4 (4.3 to 4.6)0.1 (0.3 to 0.1)4.9 (4.6 to 5.1)4.7 (4.5 to 5.0)0.1 (0.4 to 0.2)0.01 (0.37 to 0.34)

Differences in Outcomes Comparing Resident and Hospitalist Services Pre‐ and Postreform

Comparing pre‐ and postreform periods in the resident and hospitalist services, there were no significant differences in ICU admission, length of stay, MHACs, 30‐day readmissions, or inpatient mortality. In the sensitivity analysis, in which we excluded all discharges in May 2012 to June 2012, results were not significantly different for any of the outcomes examined.

DISCUSSION

Using difference‐in‐differences estimation, we evaluated whether the implementation of the 2011 residency work‐hour mandate was associated with differences in patient safety outcomes including length of stay, 30‐day readmission, inpatient mortality, MHACs, and ICU admissions comparing resident and hospitalist services at a large academic medical center. Adjusting for patient age, race, gender, and clinical complexity, we found no significant changes in any of the patient safety outcomes indicators in the postreform period comparing resident to hospitalist services.

Our quasiexperimental study design allowed us to gauge differences in patient safety outcomes, while reducing bias due to unmeasured confounders that might impact patient safety indicators.[9] We were able to examine all discharges from the resident and hospitalist general medicine services during the academic years 2009 to 2012, while adjusting for age, race, gender, and clinical complexity. Though ICU admission was higher and readmission rates were lower on the resident services post‐2011, we did not observe a significant difference in ICU admission or 30‐day readmission rates in the postreform period comparing patients discharged from the resident and hospitalist services and all patients in the prereform period.

Our neutral findings differ from some other single‐institution evaluations of reduced resident work hours, several of which have shown improved quality of life, education, and patient safety indicators.[18, 21, 26, 27, 28] It is unclear why improvements in patient safety were not identified in the current study. The 2011 reforms were more broad‐based than some of the preliminary studies of reduced work hours, and therefore additional variables may be at play. For instance, challenges related to decreased work hours, including the increased number of handoffs in care and work compression, may require specific interventions to produce sustained improvements in patient safety.[3, 14, 29, 30]

Improving patient safety requires more than changing resident work hours. Blum et al. recommended enhanced funding to increase supervision, decrease resident caseload, and incentivize achievement of quality indicators to achieve the goal of improved patient safety within work‐hour reform.[31] Schumacher et al. proposed a focus on supervision, professionalism, safe transitions of care, and optimizing workloads as a means to improve patient safety and education within the new residency training paradigm.[29]

Limitations of this study include limited follow‐up time after implementation of the work‐hour reforms. It may take more time to optimize systems of care to see benefits in patient safety indicators. This was a single‐institution study of a limited number of outcomes in a single department, which limits generalizability and may reflect local experience rather than broader trends. The call schedule on the resident service in this study differs from programs that have adopted night float schedules. [27] This may have had an effect on patient care outcomes.[32] In an attempt to conduct a timely study of inpatient safety indicators following the 2011 changes, our study was not powered to detect small changes in low‐frequency outcomes such as mortality; longer‐term studies at multiple institutions will be needed to answer these key questions. We limited the prereform period where our test of controls indicated differential prereform trends, which reduced power.

As this was an observational study rather than an experiment, there may have been both measured and unmeasured differences in patient characteristics and comorbidity between the intervention and control group. For example, CMI was lower on the hospitalist service than the resident services. Demographics varied somewhat between services; male and African American patients were more likely to be discharged from resident services than hospitalist services for unknown reasons. Although we adjusted for demographics and CMI in our model, there may be residual confounding. Limitations in data collection did not allow us to separate patients initially admitted to the ICU from patients transferred to the ICU from the inpatient floors. We attempted to overcome this limitation through use of a difference‐in‐differences model to account for secular trends, but factors other than residency work hours may have impacted the resident and hospitalist services differentially. For example, hospital quality‐improvement programs or provider‐level factors may have differentially impacted the resident versus hospitalist services during the study period.

Work‐hour limitations for residents were established to improve residency education and patient safety. As noted by the Institute of Medicine, improving patient safety will require significant investment by program directors, hospitals, and the public to keep resident caseloads manageable, ensure adequate supervision of first‐year residents, train residents on safe handoffs in care, and conduct ongoing evaluations of patient safety and any unintended consequences of the regulations.[33] In the first year after implementation of the 2011 work‐hour reforms, we found no change in ICU admission, inpatient mortality, 30‐day readmission rates, length of stay, or MHACs compared with patients treated by hospitalists. Studies of the long‐term impact of residency work‐hour reform are necessary to determine whether changes in work hours have been associated with improvement in resident education and patient safety.

Disclosure: Nothing to report.

References

Online-Only Materials

AttachmentSize
PDF icon Supplementary information (1)143.68 KB
   Comments