The use of clinical decision support, despite great promise to improve health care, remains preliminary.1 The broad scope of quality and safety challenges facing clinicians2, 3 requires this situation to change. There is an urgent need to develop decision support tools and strategies that are effective, address many quality issues simultaneously, and are easy to implement in both academic and community settings.
One decision support tool that could help to meet this challenge is the order set. An order set is a group of orders with a common functional purpose that is used directly by a physician to create orders for a specific patient. Order sets can be used with either paper‐based or computerized provider order entry (CPOE) systems. Several studies have investigated the delivery of focused evidence‐based treatments to patients admitted using disease‐specific order sets compared with either historical or concurrent controls and have demonstrated increased use of therapies such as aspirin for acute myocardial infarction admissions,4 systemic corticosteroids, metered‐dose inhalers and pulse oximetry for pediatric asthma admissions,5 and venous thromboembolism prophylaxis for adult emergency department admissions.6 However, the ability of order sets to improve multiple quality measures in a diverse patient population has not been evaluated previously.
This study examined the effect of paper‐based order sets on the quality of admission orders for general medical patients in a community hospital. The primary hypothesis was that order set use would increase the proportion of general medical patients ordered deep venous thrombosis (DVT) prophylaxis. We chose this primary endpoint because DVT prophylaxis continues to be significantly underused in hospitalized patients.7, 8 Secondary hypotheses were that order sets would improve other admission order quality of care measures. We studied paper‐based order sets because the study hospital, along with the vast majority of North American hospitals, uses paper for order entry.9
PATIENTS AND METHODS
The study took place in a 750‐bed community hospital in Mississauga, Ontario, Canada. The study included only general medical patients and excluded cardiology, neurology, and intensive care unit patients. Approximately 30 different internists admitted patients during the study period from April 1, 2003 to March 31, 2005. The internists were not aware that this study was being conducted. Order sets were implemented as an option for writing admission orders in December 2003. Prior to the implementation of order sets, physicians wrote all admission orders using traditional free‐text handwritten orders on blank paper order sheets. Essentially all general medical patients are admitted through the emergency room. The hospital's Research Ethics Board approved this study.
Order Set Development
Local specialists developed order set content (evidence‐based where possible) using informal consensus methods, without explicitly grading evidence. This process created a general admission order set and six diagnosis‐specific order sets (community acquired pneumonia, chronic obstructive pulmonary disease [COPD], febrile neutropenia, soft tissue infection, upper gastrointestinal [GI] bleeding, and urinary tract infection [UTI]). All order sets contained the same orders pertaining to the primary and secondary outcomes, except for the GI bleed admission order set, which did not contain a DVT prophylaxis section.
Order sets were paper‐based and consisted of a menu of orders typically required for a medical admission. These included admitting service, admitting physician, allergies, resuscitation status, diet, activity level, frequency of vital sign measurement, laboratory investigations, diagnostic imaging, intravenous fluid therapy, and medications. The orders were either optional (requiring the physician to check a box to be performed) or default (enacted unless specifically crossed out by the physician). Both order types could consist of a single order (for example, heparin for DVT prophylaxis) or several orders simultaneously (for example, measurement of serum sodium, potassium, and creatinine). All order sets included space for additional free‐text handwritten orders to meet individual patient needs.
The DVT prophylaxis section contained optional orders for 5,000 units of heparin subcutaneously (sc) twice daily (BID) and compression stockings. The ordering physician could select 1, both, or neither of these options. Initiating other forms of DVT prophylaxis or therapeutic anticoagulation required additional free‐text handwritten orders.
Informal clinician feedback led to improved order set content and formatting in August 2004. Orders pertaining to study outcomes were unchanged in this upgrade.
In December 2003, we placed the order sets near the stacks of blank paper order sheets used by internists admitting patients in the Emergency Department. We notified physicians by e‐mail when order sets became available but provided no formal education about order sets, DVT prophylaxis, or other study outcomes. The use of order sets was voluntary. We developed a website to facilitate timely reordering of depleted order sets from the hospital's print shop and trained all emergency room clerks regarding website access and storage of the order sets in convenient locations for physicians. Although order set availability was not formally assessed, there were no reports by physicians or observations by study investigators of order sets being unavailable at any time.
To assess the effect of order sets on the ordering of DVT prophylaxis, we retrospectively and randomly selected patient admissions and reviewed these patients' charts from 3 time periods during the study period: OctoberNovember 2003 (period 1, immediately prior to availability of order sets; 113 charts available of 120 discharged patients randomly selected from a total of 1,169 discharges); AprilDecember 2004 (period 2, 412 months after order set availability; 291 charts available of 300 discharged patients randomly selected from a total of 4,620 discharges); and FebruaryMarch 2005 (period 3, 1415 months after order set availability; 283 charts available of 290 randomly selected discharges out of a total of 1,057 discharges). We conducted an additional chart audit just prior to final submission of the manuscript (108 charts available of 120 discharged patients randomly selected from a total of 1,060 discharges in OctoberNovember 2007) to determine the sustainability of the improvements. The same patient could be selected in different time periods. One author (C.O. or K.D.) reviewed each chart using a jointly developed data collection form.
We assessed the admission orders of each chart for the use of an order set and the ordering of DVT prophylaxis, defined as 5,000 units of heparin sc BID or compression stockings (no patient received sc heparin 3 times daily, heparin sc BID in doses greater than 5,000 units, prophylactic doses of low molecular‐weight heparin, or low dose warfarin). We recorded the ordering of therapeutic anticoagulation, defined as intravenous heparin, full‐dose low molecular‐weight heparin, or warfarin with a target international normalized ratio 2.
Independent from the chart review, we examined the overall administration of heparin doses for DVT prophylaxis to all medical inpatients using the hospital pharmacy database. We estimated the overall administration of heparin for DVT prophylaxis in medical inpatients (136 medical beds, 4 wards) on a monthly basis from April 1, 2003 (8 months prior to order set availability) to March 30, 2005 (15 months after order set availability). We calculated monthly utilization as the proportion of patient‐days for which DVT prophylaxis was administered, as follows: (number of doses of subcutaneous heparin dispensed by the hospital pharmacy to the 4 wards)/(2 [since there are 2 doses per patient‐day] number of patient‐days).
We collected additional data from the charts selected during period 2 (AprilDecember 2004) to evaluate the effect of order sets on the following secondary outcomes: (1) the documentation of admission diagnosis, allergies, and code status; (2) general care orders (electrocardiogram [ECG] and notification of physician for chest pain, allied health consultation, standard hospital potassium replacement protocol [already available in the hospital], standard hospital diabetic diet and standard hospital insulin sliding scale [for patients with diabetes], night time sedation diet or nil per os, activity level and vital sign frequency); (3) blood urea nitrogen (BUN), a laboratory test often inappropriately ordered according to local guidelines10; (4) order formatting (numbering, dating, timing, and signing of all order pages); and (5) organization of orders in the standardized arrangement used in the order sets. This standardized arrangement of content was as follows: attending physician, admitting diagnosis, requests for consultation, diet, activity, vital signs, oxygen, nasogastric tube, urinary catheter, investigations, intravenous fluids, and medications. Free‐text admission orders and order set orders that maintained this arrangement were recorded as standardized. We did not assess order appropriateness.
We recorded the characteristics of all medical patients admitted to the hospital in two 1‐year periods during the study (April 1, 2003 to March 30, 2004 and April 1, 2004 to March 30, 2005), including age, gender, length of stay, diagnosis (defined by case management group [CMG]), and resource intensity weight (RIW). CMG defines groups of patients who are similar in diagnosis or procedure and RIW is a measure of resources used during a patient's hospital stay.11 The definitions of CMG and RIW did not change during the study.
Baseline characteristics were compared using Student t‐test for normally distributed continuous variables (patient age) and the Mann‐Whitney U test for skewed continuous variables (length of stay and RIW). Chi square or Fisher's exact tests were used to compare categorical variables. Relative risks (RR) and 95% confidence intervals (CI) were calculated and compared using a z‐test. A 2‐sided P value <0.05 was taken to be statistically significant. All calculations were carried out using SAS Version 8.2 (SAS Institute, Cary, NC).
As shown in Table 1, there were no clinically important differences in demographic or clinical characteristics of medical patients between the 2 years of the study. There were small but statistically significant increases in patient illness complexity (as reflected in median RIW) (P = 0.003) and length of stay (P = 0.0002).
|Patient Characteristic||April 1, 2003 to March 30, 2004 (n = 4,415)||April 1, 2004 to March 30, 2005 (n = 4,287)|
|Age, mean SD||67.2 17.7||67.6 17.5|
|Length of stay, median days (IQR)||6 (3‐12)||6 (3‐13)|
|RIW,11 median (IQR)||0.96 (0.68‐1.73)||1.03 (0.72‐1.88)|
|Females, number (% of total)||2,276 (52)||2,223 (52)|
|Case mix group,11 number of patients (% of total)|
|Chronic obstructive pulmonary disease||357 (8.1)||385 (9.0)|
|Simple pneumonia and pleurisy||322 (7.3)||217 (5.1)|
|Esophageal, gastrointestinal, and miscellaneous digestive disease||223 (5.1)||239 (5.6)|
|Gastrointestinal hemorrhage||185 (4.2)||198 (4.6)|
|Respiratory neoplasm||127 (2.9)||144 (3.4)|
|Total||1,214 (27.5)||1,183 (27.7)|
Clinicians used order sets in 32.3% of admissions during period 2 (AprilDecember 2004, 412 months after order set availability), increasing to 51.6% in period 3 (FebruaryMarch 2005, 1415 months after order set availability). The results of the chart audit assessing the impact of order set use on DVT prophylaxis are shown in Figure 1. Prior to order set introduction, 10.9% of patients received orders for DVT prophylaxis. Subsequently, ordering of DVT prophylaxis in patients admitted with order sets increased (period 2: 35.6%; P < 0.001; RR, 3.27; 95% CI, 1.806.12 and period 3: 44.0%; P < 0.001; RR, 4.04; 95% CI, 2.327.31). In contrast, DVT prophylaxis ordering in the nonorder set group was initially unchanged (period 2: 10.6%; P = 0.93; RR, 0.97; 95% CI, 0.491.95), although later it increased to a smaller extent (period 3: 20.6%; P = 0.049; RR, 1.90; 95% CI, 1.013.65). As a result of this differential increase, patients admitted with order sets were more likely to be ordered DVT prophylaxis in both study periods (period 2: 35.6% versus 10.6%; P < 0.0001; RR, 3.38; 95% CI, 2.035.62 and period 3: 44.0% versus 20.6%; P < 0.0001; RR, 2.13; 95% CI, 1.443.16). The use of therapeutic anticoagulation was similar in patients admitted with and without order sets and did not change between time periods.
The hospital‐wide monthly utilization of heparin for DVT prophylaxis in medical inpatients increased from an average of 12.8% (range, 9.7%16.1%) of patient‐days before order set implementation (AprilNovember 2003) to 18.5% (range, 16.4%20.0%) of patient‐days in the 8 months after order sets were first implemented (DecemberJuly 2004, P < 0.0001 compared to the preorder set time period). After August 2004, when upgraded order sets were introduced, DVT prophylaxis utilization increased further in the last 7 months of the study to 25.8% (range, 22.4%32.2%; P < 0.0001 compared to preorder set time period; Figure 2).
Table 2 shows the impact of order sets on secondary outcomes. Admissions completed with order sets had statistically significant increases in general care orders (ECG and notification of physician for chest pain, allied health consultations and standard hospital diabetic diet, insulin scale, and potassium replacement protocol orders), documentation of allergies and code status, numbering of pages, and use of a standardized arrangement for orders. Ordering of BUN decreased significantly.
|Outcome||Optional or Default||Order Set [n = 94 (%)]||No Order Set [n = 197 (%)]||P Value|
|Admitting diagnosis||Optional||91 (96.8)||187 (94.9)||0.47|
|Allergies||Optional||51 (54.3)||19 (9.6)||<0.0001|
|Resuscitation status||Optional||54 (57.4)||20 (10.2)||<0.0001|
|General care orders|
|ECG and call MD for chest pain||Default*||80 (85.1)||0 (0.0)||<0.0001|
|Allied health consult||59 (62.8)||25 (12.7)||<0.0001|
|Diet||Optional||90 (95.7)||188 (95.4)||0.90|
|Activity||Optional||80 (85.1)||150 (76.1)||0.08|
|Vitals signs and frequency||Optional||91 (96.8)||178 (90.4)||0.052|
|Standard hospital diabetic diet||Optional||16 (17.0)||10 (5.1)||0.0008|
|Standard hospital insulin sliding scale||Optional||18 (19.1)||15 (7.6)||0.004|
|Standard hospital potassium protocol||Optional||60 (63.8)||1 (0.51)||<0.0001|
|Zopiclone as needed||Optional||43 (45.7)||2 (1.0)||< 0.0001|
|Lorazepam as needed||Optional||12 (12.8)||15 (7.6)||0.16|
|Laboratory test order|
|Blood urea nitrogen||Optional||37 (39.4)||117 (59.0)||0.0014|
|Numbering of pages||Default||94 (100)||4 (2.0)||<0.0001|
|Dating of orders||Optional||79 (84.0)||185 (93.9)||0.0067|
|Timing of orders||Optional||14 (14.9)||29 (14.7)||0.97|
|Signing of orders||Optional||93 (98.9)||196 (99.5)||0.54|
|Standard arrangement of orders||81 (86.2)||66 (33.5)||<0.0001|
Order sets were not associated with changes in diet, activity, or vital sign orders, documentation of admission diagnosis, or the signing and timing of orders. Apart from order timing, these orders were present in >75% of admissions completed without order sets. The only negative effect of order sets was a reduction in the dating of orders (84.0% of order set admissions versus 93.9% of nonorder set admissions, P = 0.007). Finally, order sets had both an intended and unintended effect on nighttime sedation orders. Relative frequency of ordering of zopiclone compared to lorazepam increased (43/55 in the order set group vs. 2/17 in the no order set group [P < 0.0001], the intended effect), and increased overall frequency of ordering of nighttime sedation (55/94 vs. 17/197 [P < 0.0001], an unintended effect).
The additional chart audit in OctoberNovember 2007, just prior to final submission of the manuscript, determined that clinician use of order sets had increased to 92.6% of admitted medical patients, and that ordering of DVT prophylaxis in patients admitted with order sets had been sustained at 43.2% (P = 0.90 compared to period 3) (Figure 1).
We found that paper‐based order sets were associated with markedly increased use of DVT prophylaxis and made physician ordering more consistent with hospital consensus guidelines in multiple other areas, including laboratory test utilization and general care, while also increasing completeness of documentation. Given the difficulties and limited resources frequently associated with guideline development, dissemination, and implementation,12 it is worth noting that our improvements were achieved in a community hospital with voluntary physician adoption and no dedicated project funding, care process redesign, or healthcare worker education. The broad impact of order sets combined with minimal organizational resources required for implementation in this study suggests that this clinical decision support tool may have wide applicability.
The study hospital used paper‐based orders rather than CPOE, similar to 90% of U.S. hospitals at the time of the study.9 Order sets can be deployed in either paper‐based or computerized ordering systems. By providing a mechanism for entering large blocks of orders in an efficient manner, paper‐based order sets may be a necessary first step to facilitate the paper to CPOE transition, making them well suited to the current care delivery environment. Successful use of paper‐based order sets may help accelerate adoption of CPOE, which appears to be many years away from full implementation in the majority of U.S. hospitals.13
The most clinically important outcome in our study was a more than 4‐fold increase in ordering of DVT prophylaxis (last study period compared with baseline) in medical patients admitted with order sets, compared to a smaller increase in patients admitted without order sets. Our result is particularly significant as this study was performed in a community hospital, a setting with a lower adherence to DVT prophylaxis guidelines compared to academic centers.8, 14 The increase in DVT prophylaxis in patients admitted without order sets could be the result of a secular trend or a passive educational effect of order sets on physicians who only used order sets intermittently. The study was not publicized and thus was unlikely in itself to contribute to the increased performance.
We did not assess clinical outcomes of DVT or pulmonary embolism, but the clinical efficacy of improving adherence to DVT prophylaxis has been previously established.15 We also did not assess the appropriateness of DVT prophylaxis (or any other order). However, a recent multicenter Canadian observational study, using the American College of Chest Physician's Consensus Guidelines on Antithrombotic Therapy16 as a reference standard, found that 90% of medical patients admitted to hospital meeting study criteria had indications for thromboprophylaxis, but only 16% of eligible patients actually received it.8 In addition, multivariable regression analysis demonstrated even lower utilization in community hospitals compared to academic hospitals. These data suggest that the study hospital is typical of Canadian hospitals, and that the low overall utilization of DVT prophylaxis (13% of hospital patient‐days) prior to the availability of order sets in the study hospital is a significant gap between optimal and actual practice.
In addition, order sets had an impact on many secondary outcomes, such as standardization and completeness of orders (for example documentation of allergies and resuscitation status). While these effects appear to be beneficial in terms of quality of care and patient safety, the relationship of our secondary outcomes to patient‐important outcomes has not been established.
Furthermore, our before‐after design does not exclude the possibility of unknown confounding effects as explanations of improved performance in the order set group. For example, the change could have been driven by a small number of admitting physicians, since it is likely that order sets were adopted more readily by some physicians than others, and this group could have been responsible for a greater proportion of the admissions at different times. Unfortunately, we did not record the identity of the admitting physician. However, data from OctoberNovember 2007 show that >90% of medical patients were admitted using order sets, suggesting that voluntary clinician adoption of order sets has become nearly universal. Nevertheless, there still appear to be a few physicians who rarely or never used orders sets. Motivating these physicians to prescribe appropriate DVT prophylaxis remains a challenge.
Although this study was conducted in 1 center, other hospitals have similarly low rates of thromboprophylaxis,8 and our order set implementation strategy consumed few resources, improving the generalizability of our results. While most changes were beneficial, order set use was associated with decreased dating of orders and with an unintended effect or overall increase ordering of nightime sedation. Although the reasons for this are unclear, it highlights the importance of systematically evaluating the impact of order sets to identify unintended consequences and areas in which the order set may need to be redesigned to address these issues.
The study of order sets is preliminary despite their role as a key enabler for CPOE17 and their suggested usefulness to reduce medical error.18 For example, order sets were not considered in recent analyses of factors predicting success of computerized decision support systems19, 20 and have not been reviewed by the Cochrane Effective Practice and Organisation of Care Group.21 As discussed in the introduction, several studies have demonstrated that disease‐specific order sets can increase the use of evidence‐based treatments.46 Our study extends this work by demonstrating that admission order sets can improve performance hospital‐wide for a broad range of outcomes simultaneously, including DVT prophylaxis. Although most studies have demonstrated increased utilization of evidence‐based therapies, at least 1 study found no increased use of aspirin, heparin, or beta‐blockers in acute coronary syndrome admissions with the introduction of order sets.22 This suggests that the way order sets are structured or introduced is important to ensure that they achieve the desired changes in practice. Finally, our study suggests that improved outcomes using order sets can still be achieved with minimal organizational resources.
Order sets may potentially complement other decision support tools such as alerts and reminders. Alerts are an effective decision support tool12, 23, 24 but risk disrupting clinician workflow. Moreover, excessive alerts can lead to alert fatigue, resulting in many alerts being ignored.25 This phenomenon reduces alert effectiveness and limits the number of issues that alerts can address simultaneously. In contrast, order sets are broad in scope due to integration with clinical workflow, but lack the ability of alerts to apply rules to a specific patient's data. A potentially effective 2‐staged decision support strategy would use order sets as the primary admission decision support tool and selective alerts for remaining issues. This approach may increase the overall scope, physician adoption, and effectiveness of clinical decision support, and should be evaluated.
Our postintervention rate of DVT prophylaxis, while substantially improved from baseline, is still below ideal practice. Order sets were simply made available to clinicians admitting medical patients, who had the option to select DVT prophylaxis. Given limited resources, we did not develop and implement education programs regarding the appropriate use of DVT prophylaxis or make available any DVT risk assessment evaluations (available in Ref.26). Our study methodology thus provides a realistic assessment of improvements attainable in other hospitals with similarly limited resources. Additional increases in DVT prophylaxis rates would likely require a more comprehensive and resource‐intensive multifaceted quality improvement initiative. Detailed guidelines and supporting references for implementing such an initiative are available from the Society of Hospital Medicine.26 As described in their Venous Thromboembolism (VTE) Resource Room,26 such an initiative should include a standardized DVT risk assessment to guide the need for DVT prophylaxis integrated into admission order sets; prompts to order DVT prophylaxis when completing admission orders; and a system to audit adverse events and variations from best practice and return this information to clinicians.26
This is the largest and most comprehensive evaluation of the effectiveness of order sets as a clinical decision support tool. We found that order sets improved the quality of multiple patient orders and improved hospital‐wide DVT prophylaxis rates. These improvements were achieved in a community hospital with voluntary physician adoption and no dedicated project funding, care process redesign, or healthcare worker education. Although used in a paper‐based format in this study, order sets can also be employed in a computerized ordering environment. By providing a mechanism for entering large blocks of orders in an efficient manner, paper‐based order sets may be a necessary first step to facilitate the paper‐to‐CPOE transition. These attributes make order sets an attractive quality improvement tool in community and academic settings. More research is needed on the optimal design and use of this promising decision support tool.