Original Research

Use of UpToDate and outcomes in US hospitals

Abstract

Abstract

BACKGROUND:

Computerized clinical knowledge mana‐gement systems hold enormous potential for improving quality and efficiency. However, their impact on clinical practice is not well known.

OBJECTIVE:

To examine the impact of UpToDate on outcomes of care.

DESIGN:

Retrospective study.

SETTING:

National sample of US inpatient hospitals.

PATIENTS:

Fee‐for‐service Medicare beneficiaries.

INTERVENTION:

Adoption of UpToDate in US hospitals.

MEASUREMENT:

Risk‐adjusted lengths of stay, mortality rates, and quality performance.

RESULTS:

We found that patients admitted to hospitals using UpToDate had shorter lengths of stay than patients admitted to non‐UpToDate hospitals overall (5.6 days vs 5.7 days; P < 0.001) and among 6 prespecified conditions (range, −0.1 to −0.3 days; P < 0.001 for each). Further, patients admitted to UpToDate hospitals had lower risk‐adjusted mortality rate for 3 of the 6 conditions (range, −0.1% to −0.6% mortality reduction; P < 0.05). Finally, hospitals with UpToDate had better quality performance for every condition on the Hospital Quality Alliance metrics. In subgroup analyses, we found that it was the smaller hospitals and the non‐teaching hospitals where the benefits of the UpToDate seemed most pronounced, compared to the larger, teaching institutions where the benefits of UpToDate seemed small or nonexistent.

CONCLUSIONS:

We found a very small but consistent association between use of UpToDate and reduced length of stay, lower risk‐adjusted mortality rates, and better quality performance, at least in the smaller, non‐teaching institutions. These findings may suggest that computerized tools such as UpToDate could be helpful in improving care. Journal of Hospital Medicine 2012. © 2011 Society of Hospital Medicine.

Copyright © 2011 Society of Hospital Medicine

New health information technologies hold enormous potential for improving the quality and efficiency of healthcare. One commonly used health information technology is computerized clinical knowledge management (CKM) systems, which provide clinicians with access to relevant and continually updated clinical information about major medical topics at the point of care. Studies indicate that clinicians often have questions about patient care, which go largely unanswered during patient encounters.13 The availability of answers to critical clinical questions can have a large impact on clinical decision‐making and practice.4

UpToDate is one of the most widely used computerized clinical knowledge management systems in the nation.1, 2, 5, 6 Previous studies of UpToDate and similar systems demonstrated that these systems improve acquisition of knowledge, increase the number of answered clinical questions, and change management decisions.7, 8 However, whether these changes lead to real improvements in clinical outcomes is unknown. Given the urgent need to improve both the quality and efficiency of healthcare, understanding whether UpToDate has the potential to improve outcomes is critical.

Therefore, we examined whether the use of UpToDate was associated with lower risk‐adjusted mortality rates, shorter lengths of stay, and better performance on standard quality process metrics. Further, we sought to determine whether the impact of UpToDate was particularly potent in certain subsets of hospitals. Finally, we examined whether the duration of use of UpToDate was associated with better outcomes.

METHODS

Overview

Our overall approach was to examine the relationship between UpToDate and 3 main outcomes: risk‐adjusted length of stay, risk‐adjusted mortality, and performance on standard quality process metrics in the period from 2004 to 2006. We took 4 approaches to try to reduce potential confounding (adopters of the UpToDate are likely different than non‐adopters). First, we used a longitudinal modeling approach where hospitals were allowed to serve as their own controls over time. For example, if a hospital was an adopter of UpToDate in the fourth quarter of 2005, all data for that hospital prior to that quarter of 2005 would be counted as part of the control hospitals. Second, we used multivariable models to adjust for observable differences between adopters and non‐adopters. Third, we tested for interactions to see if the effect of UpToDate was particularly concentrated in a subset of hospitals. Finally, we examined whether the duration of use, which reflects a potential dose‐response relationship, was related to the outcomes of interest.

UpToDate

The UpToDate system provides a compendium of regularly revised, evidence‐based monographs on topics in adult internal medicine, pediatrics, and obstetrics and gynecology.9 The system is available through multiple medias (ie, the Internet, handheld devices). Providers at subscribing hospitals can usually access it through any computer terminal within the facility, and often through remote access.

Data Sources and Linkage

We used 5 primary sources of data: UpToDate usage data, which was provided directly by UpToDate and includes a list of all hospitals in the United States who use the UpToDate software and the start date of usage; the American Hospital Association (AHA) Annual Survey, which contains individual hospital structural characteristics (such as size, location, teaching status); the 2007 Medicare Inpatient Impact files, which includes hospital characteristics not available in the AHA data; the 2004‐2006 Medicare Provider Analysis and Review (MEDPAR) databases, which have patient‐level discharge information about all Medicare fee‐for‐service patients hospitalized in a given year; and the 2004‐2007 Hospital Quality Alliance (HQA) database, which includes publicly available data for inpatient quality measures. We linked these 4 datasets with a database of hospitals that use UpToDate.

Outcomes

We chose, a priori, to examine 3 primary outcomes: risk‐adjusted length of stay (LOS), which is considered a measure of efficiency; risk‐adjusted mortality, a commonly used marker of quality of care; and performance outcomes on HQA quality metrics.

Risk‐Adjusted LOS and Risk‐Adjusted Mortality Rates

We examined risk‐adjusted LOS and risk‐adjusted mortality rates among all hospitalized patients and among 6 common medical and surgical conditions: acute myocardial infarction (AMI), congestive heart failure (CHF), pneumonia, gastrointestinal hemorrhage, stroke, and hip fracture. We selected these 6 conditions because they are used by the Agency for Healthcare Research and Quality (AHRQ) to measure hospital quality. We used International Classification of Diseases, Ninth Revision (ICD‐9) codes used by the AHRQ Inpatient Quality Indicators to identify patients admitted for these 6 conditions.10 We performed risk‐adjustment using the Elixhauser comorbidity adjustment scheme, which was developed by AHRQ and is commonly used to adjust for differences in severity using administrative data.

Quality Processes of Care

To examine hospital quality performance, we used the HQA process measures for 4 conditions from 2004 to 2007: AMI, CHF, pneumonia, and surgical infection prevention (SIP). We examined all HQA indicators publically available in 2004 (8 measures for AMI, 4 measures for CHF, 6 measures for pneumonia, and 2 measures for SIP). (The specific indicators are listed in Supporting Appendix Table 1 in the online version of this article.) We created summary scores for each condition, and an overall hospital summary score for the performance on all indicators, using methodology previously described by the Joint Commission.11 Each summary score represents the number of times a hospital performed the appropriate action across all measures for that condition divided by the number of opportunities the hospital had to provide appropriate care for that condition. Composite scores were only calculated if a hospital had at least 30 patients for at least one of the measures of each condition.

AnalysisCKM Users Versus Non‐Users

We chose, a priori, to perform several sets of analyses to understand the relationship between the use of UpToDate and clinical outcomes. Our primary approach was a longitudinal model where each hospital was allowed to serve as its own control. In sensitivity analyses, we used a differences‐in‐differences model where we examined whether the changes for hospitals that adopted UpToDate differed compared to changes in outcomes for non‐adopters, adjusting for temporal trends by using time as a covariate in the model.

In the first analysis using longitudinal data, we examined whether being admitted to a hospital with UpToDate was associated with shorter length of stay, lower risk‐adjusted 30‐day mortality or higher process quality. This model allowed each hospital to serve as its own control and tested to see how the outcomes changed after adoption of UpToDate, controlling for secular trends by including non‐UpToDate hospitals. The models were adjusted for hospital characteristics including size, region, location (urban vs rural), ownership (for‐profit, not‐for‐profit private, not‐for‐profit public), teaching status (member of the Council of Teaching Hospital vs not), the proportion of patients that had Medicaid, the Disproportionate Share Hospital (DSH) Index, and the presence or absence of a medical intensive care unit (ICU). We used a repeated‐measures generalized estimating equations (GEE) to account for both clustering at the hospital level and for the repeated measures nature of our analysis. We used the Elixhauser comorbidity adjustment scheme to account for patient‐level factors.12

In our first set of models, we included all patients. We subsequently built 6 condition‐specific models for each of the 6 common medical conditions: AMI, CHF, pneumonia, gastrointestinal hemorrhage, stroke, and hip fracture.

Identifying Subsets of Hospitals

Next, we postulated a priori that certain subsets of hospitalssmaller institutions and non‐teaching institutionsmight have less access to high‐quality clinical information and, thus, be mode likely to benefit from UpToDate. To determine if the potential impact of UpToDate on the outcomes varied based on these hospital characteristics, we repeated our analyses using multivariable models but tested interaction terms. We found significant interactions for 1 outcome (HQA quality performance scores), and present data with stratified analyses for this outcome.

Impact of Duration of Use

Finally, we calculated duration of UpToDate use for each hospital. For each hospital using UpToDate, we identified the date it started using the system. Based on the start date, we calculated each hospital's duration of use for each quarter. We used the midpoint of that quarter to calculate the number of days a hospital used UpToDate. For example, if a hospital started using UpToDate on January 1, 2002, we assigned 775 days for its duration of use for the first quarter of 2004 (365 days per year 2 years + 45 days for the first quarter of 2004) and 865 days for its duration of use for the second quarter of 2004. We excluded hospitals that did not use UpToDate.

Our primary analysis used a dataset that was restricted to just those hospitals using UpToDate. We ran the model for all discharges and for each of the 6 conditions for both LOS and for risk‐adjusted mortality rate, as well as for the 4 conditions encompassed in the HQA quality reporting program. In all analyses, we considered a 2‐sided P value as statistically significant.

Assessing the Overall Impact of UpToDate

To better assess the clinical significance of any change in mortality rates, we calculated the number of deaths prevented if all hospitals saw the same gain in mortality if they adopted UpToDate. To calculate this impact number, we identified the overall reduction in risk‐adjusted mortality associated with UpToDate adoption and multiplied this number by the number of elderly Medicare patients admitted to non‐UpToDate hospitals. We calculated a similar number for changes in length of stay.

RESULTS

We found that between 2004 through 2006, 1017 hospitals used UpToDate for at least 1 quarter. Users of UpToDate were more likely to be large, urban, teaching hospitals located in the Northeast, and either public or nonprofit (private) hospitals (Table 1).

Characteristics of Hospitals Using and Not Using UpToDate
CharacteristicsUsing UpToDate (N = 1017)Not Using UpToDate (N = 2305)P Value
%%
Hospital size <0.001
Small (6‐99)1336
Medium (100‐399)6455
Large (400+)238
Hospital region <0.001
Northeast2712
Midwest2623
South2747
West2018
Profit status <0.001
Profit hospitals1021
Nongovernment nonprofit7561
Government nonprofit1518
Teaching hospitals194<0.001
Urban location9684<0.001

Over the 3 years, patients admitted to hospitals with UpToDate had generally shorter lengths of stay for all hospitalizations than patients admitted to hospitals without this specific system (5.6 days vs 5.7 days, P < 0.001; Table 2), and shorter lengths of stay for each of the 6 conditions examined (0.1 to 0.3 days shorter LOS, P < 0.001; Table 2).

Risk‐Adjusted Length of Stay for Hospitals Using UpToDate Compared to Non‐Users
ConditionsUsing UpToDate (Days)Not Using UpToDate (Days)Difference (CI) (Days)P Value
  • NOTE: Quarterly data from 2004 through 2006. All analyses are adjusted for hospital characteristics including size, census region, urban vs rural location, ownership (for‐profit, not‐for‐profit private, not‐for‐profit public), teaching status (member of the Council of Teaching Hospital vs not), and the presence or absence of a medical intensive care unit (ICU). Analyses were also adjusted for patient‐level factors and comorbidites using methodology developed by Elixhauser.

  • Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; GIH, gastrointestinal hemorrhage; PN, pneumonia.

Total5.65.70.1 (0.2 to 0.0)0.001
AMI5.35.50.2 (0.3 to 0.2)<0.001
CHF5.65.70.2 (0.2 to 0.1)<0.001
PN6.36.50.2 (0.2 to 0.1)<0.001
Stroke5.96.00.1 (0.2 to 0.1)<0.001
GIH5.35.40.2 (0.3 to 0.2)<0.001
Hip fracture6.76.80.1 (0.2 to 0.1)<0.001

Similarly, we found that hospitals with UpToDate had lower risk‐adjusted 30‐day mortality rates, although the effects here were less consistent (Table 3). When we examined individual conditions, we found that patients admitted to UpToDate hospitals had lower risk‐adjusted 30‐day mortality rate for 4 of the 6 conditions, although only 3 of those 4 differences were statistically significant (Table 3). For example, patients in UpToDate hospitals had a lower likelihood of mortality for AMI (18.4% vs 18.9%, P = 0.03).

Risk‐Adjusted 30‐Day Mortality Rates Among Hospitals Using UpToDate Compared to Non‐Users
ConditionsUsing UpToDate (%)Not Using UpToDate (%)% Difference (CI)P Value
  • NOTE: Rates from 2004 through 2006. All analyses are adjusted for hospital characteristics and patient characteristics.

  • Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; GIH, gastrointestinal hemorrhage; PN, pneumonia.

Total9.09.10.1 (0.2 to 0.0)0.04
AMI18.419.00.7 (1.2 to 0.2)0.03
CHF11.111.30.2 (0.4 to 0.1)0.21
PN12.112.60.5 (0.7 to 0.2)<0.001
Stroke19.919.90.02 (0.5 to 0.5)0.91
GIH6.97.30.4 (0.7 to 0.2)0.001
Hip fracture8.88.60.2 (0.2 to 0.5)0.41

We found a more consistent association between the adoption of UpToDate and quality performance. Hospitals that had UpToDate had higher performance for each of the 4 conditions examined. For example, hospitals with UpToDate had higher quality performance for AMI compared to hospitals that did not adopt UpToDate (93.2% vs 90.4%, P < 0.001; Table 4). The results for CHF, pneumonia, and surgical complication prevention were qualitatively similar, and each reached statistical significance (Table 4).

UpToDate Use and Performance on the Standard Quality Indicators
ConditionsUsing UpToDate (%)Not Using UpToDate (%)% Difference (CI)P Value
  • NOTE: All analyses are adjusted for hospital characteristics and patient characteristics. Data are based on performance on the Hospital Quality Alliance (HQA) indicators; UpToDate use and HQA scores among all hospitals, 2004 through 2007.

  • Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; PN, pneumonia, SIP, surgical infection prevention.

AMI summary score93.490.23.2 (2.6, 3.6)< 0.001
CHF summary score81.075.15.9 (5.0, 6.8)< 0.001
PN summary score83.783.10.6 (0.3, 0.9)0.003
SIP summary score80.078.11.9 (1.0, 2.9)0.002

In analyses that test for interaction, we found that the relationship between UpToDate use and quality performance was modified by hospital size and teaching status. Specifically, much of the benefit of UpToDate seemed limited to small and medium‐sized hospitals, as well as non‐teaching hospitals. For example, among small hospitals, those with UpToDate had, on average, 3‐7 point greater performance on the HQA scores, but almost no effect was found among large hospitals. Similarly, we found that non‐teaching hospitals were likely to have better performance in each of these areas if they had UpToDate, but this relationship was not consistent among major teaching hospitals (Table 5).

UpToDate Use and HQA Scores, Stratified by Size and Teaching Status, 2004‐2007
Using UpToDate (%)Not Using UpToDate (%)% Difference (CI)P Value
  • NOTE: All analyses are adjusted for hospital characteristics and patient characteristics.

  • Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; HQA, Hospital Quality Alliance; PN, pneumonia.

Hospital sizeSmall (6‐99 beds)
AMI summary score90.387.92.35 (0.91, 3.78)<0.001
CHF summary score75.769.66.11 (4.04, 8.18)<0.001
PN summary score84.983.21.67 (0.83, 2.51)<0.001
Medium (100‐399 beds)
AMI summary score93.090.52.52 (2.04, 3.00)<0.001
CHF summary score81.178.92.22 (1.31, 3.12)<0.001
PN summary score84.083.20.84 (0.31, 1.36)<0.001
Large (>399 beds)
AMI summary score94.793.80.82 (0.19, 1.46)<0.001
CHF summary score82.982.70.21 (1.27, 1.69)0.83
PN summary score82.482.70.34 (1.49, 0.81)0.39
Teaching statusMajor teaching
AMI summary score94.994.80.15 (0.62, 0.92)0.60
CHF summary score83.383.00.26 (1.73, 2.25)0.83
PN summary score81.781.70.00 (1.67, 1.67)0.95
Not major teaching
AMI summary score92.790.12.59 (2.15, 3.02)<0.001
CHF summary score80.175.05.04 (4.18, 5.90)<0.001
PN summary score84.383.21.05 (0.64, 1.47)<0.001

In our analyses of duration of use of UpToDate, we found a consistent relationship with shorter lengths of stay (see Supporting Appendix Table 2 in the online version of this article). Each 1000 days of UpToDate use was associated with a 0.08 day shorter length of stay (P < 0.001). A similar relationship was present and statistically significant in each of the 6 conditions examined. When we examined the impact of duration of use of UpToDate on risk‐adjusted mortality rate, we found a comparable relationship: Duration was associated with a lower mortality rate overall and for 5 of the 6 conditions examined, although only significant for 3 of the conditions (see Supporting Appendix Table 2 in the online version of this article). Similarly, greater duration was associated with better quality performance for each of the 4 conditions in the HQA program (data not shown).

When we quantified the overall impact of UpToDate, we found that if non‐adopters had a similar benefit in mortality (0.1%) seen in hospitals that adopted this system, it would lead, overall, to approximately 5550 fewer deaths annually (95% confidence interval 2601 fewer deaths to 7529 fewer deaths) and the 0.1 days shorter length of stay would lead to approximately 523,000 fewer patient days (95% confidence interval 160,000 to 799,000 fewer days) out of the approximately 30 million patient‐days that occurred in 2006 among non‐UpToDate hospitals.

DISCUSSION/CONCLUSION

We found that use of a commonly used computerized clinical knowledge management system (UpToDate) was associated with consistent, although small, reductions in lengths of stay, lower risk‐adjusted mortality rates, and higher quality performance. Much of the quality performance benefit seemed to be limited to small and medium‐sized, non‐teaching hospitals, while larger teaching hospitals realized little benefit. We found a stronger relationship between duration of use and better outcomes among UpToDate hospitals. Our findings suggest that hospitals using UpToDate had modestly better care that was also somewhat more efficient.

Prior studies have demonstrated that clinical decision support tools (ie, drug‐drug alerts and electronic reminders) can improve processes of care and enhance quality.13 Computerized clinical knowledge management systems, such as UpToDate, have unique advantages over other computerized clinical decision support tools. For example, computerized clinical knowledge management systems generally do not require electronic health records and can provide guidance to clinicians over a broader spectrum of diseases and clinical scenarios. UpToDate has previously shown to help providers answer questions rapidly, which can lead to changes in decision‐making that can improve management and efficiency.1, 7, 14

Ours is the first national study, to our knowledge, that has directly examined the relationship between UpToDate and key outcome metrics. Bonis et al. have previously examined the use of UpToDate and its relationship to risk‐adjusted LOS and mortality in a limited set of hospitals using a proprietary risk‐adjustment scheme.15 They found similar results among the Thompson 100 hospitals that were clinical knowledge management system users (compared to non‐users), including modestly shorter lengths of stay and a trend towards lower mortality. Our findings build on this work, but use publicly available data, a national sample, a well‐validated risk‐adjustment approach, and a much longer time period. The consistency of the findings across the 2 studies, despite differences in the approaches, help lend confidence that the results are unlikely to be due to chance alone.

There are likely to be important considerations surrounding the costs of UpToDate systems and whether those who purchase the system are wealthier than hospitals that chose not to. The typical annual subscription costs for a 100‐bed hospital between 2006 and 2010 was $10,578, which likely represents less than 0.01% of the annual operating costs for a 100‐bed institution (and is approximately the amount Medicare reimbursed for a single case of pneumonia without complications). Whether this cost would prohibit the adoption of UpToDate for most hospitals is unclear, and it is possible that a hospital that spent nearly $10,600 a year in such a system might therefore forego other quality improvement efforts. We suspect that these effects likely vary from hospital to hospital.

The primary limitation of the study is our inability to address whether or not the associations we found between UpToDate and outcomes are causally related. This is a fundamental limitation of all nonrandomized data. Four factors should lend some confidence to the interpretation that these findings may not be due to confounding alone. First, the effects were consistent across a series of measures (mortality, efficiency, and processes) that are not, themselves, highly correlated with each other1618. Our findings, that hospitals with UpToDate were somewhat better across all measures examined, point to the potential benefit of having high‐quality clinical information readily available for clinicians. Second, we found that the benefits persisted even after controlling for other hospital characteristics that were associated with adoption, including measures of hospital financial health (as measured by proportion of Medicaid patients and the DSH Index). Of course, this does not negate the possibility that other factors, such as the presence of medical libraries or a culture of quality and continuous learning, may be associated both with the use of UpToDate and with the outcomes. Third, the effects, at least for quality performance, were prominent among smaller, non‐teaching hospitals (which, one would surmise, a priori, to be most likely to benefit from UpToDate). Finally, the dose‐response relationship of duration of use, which was an analysis limited to only those hospitals with UpToDate, provides more evidence that the system itself may have some impact.

Another limitation of our work is that we used administrative data for risk‐adjustment, which has inherent challenges.1925 Given that users of UpToDate, such as teaching hospitals and larger institutions, generally have a much sicker patient population, inadequate risk‐adjustment may have lead us to underestimate the true effect. It is also possible that many of the hospitals designated as non‐users had other clinical knowledge management systems of which we were unaware. However, this would likely have made it harder to find an effect, biasing our study towards a null finding. We conducted a series of analyses and, yet, did not adjust for multiple testing. We had chosen these analyses a priori and, although the association we found may have been due to selection bias, it is unlikely that all of the associations in our analyses were due to random chance. Next, although we examined the potential impact of UpToDate, we suspect that any high‐quality clinical knowledge management system should allow clinicians to deliver higher quality, more efficient care. Finally, we are unsure whether or not the magnitude of effect we found is clinically significant. While the added advantage of having UpToDate appeared to be a reduction in mortality of only 0.1% (over all conditions), such a difference would be associated with approximately 5550 fewer deaths each year among Medicare fee‐for‐service beneficiaries. Whether such a benefit would be worth the cost of implementing systems like UpToDate needs to be further explored.

In conclusion, we found consistent association between use of a widely deployed computerized clinical knowledge management system, UpToDate, and reduced length of stay, lower risk‐adjusted mortality rates, and higher quality performance. Whether use of UpToDate led to better care is not definitive, but our findings suggest that these types of management software may play an important role as our nation endeavors to improve the quality and efficiency of the healthcare system.

References

Online-Only Materials

AttachmentSize
PDF icon Supporting Information (1)758.19 KB
PDF icon Supporting Tables (2)583.57 KB
   Comments ()