Brief Reports

The effect of simulation‐based mastery learning on thoracentesis referral patterns

Abstract

Internal medicine (IM) residents and hospitalist physicians commonly perform thoracenteses. National data show that thoracenteses are also frequently referred to other services such as interventional radiology (IR), increasing healthcare costs. Simulation‐based mastery learning (SBML) is an effective method to boost physicians' procedural skills and self‐confidence. This study aimed to (1) assess the effect of SBML on IM residents' simulated thoracentesis skills and (2) compare thoracentesis referral patterns, self‐confidence, and reasons for referral between traditionally trained residents (non–SBML‐trained), SBML‐trained residents, and hospitalist physicians. A random sample of 112 IM residents at an academic medical center completed thoracentesis SBML from December 2012 to May 2015. We surveyed physicians caring for hospitalized patients with thoracenteses during the same time period and compared referral patterns, self‐confidence, and reasons for referral. SBML‐trained resident thoracentesis skills improved from a median of 57.6% (interquartile range [IQR] 43.3–76.9) at pretest to 96.2% (IQR 96.2–100.0) at post‐test (P < 0.001). Surveys demonstrated that traditionally trained residents were more likely to refer to IR and cited lower confidence as reasons. SBML‐trained residents were more likely to perform bedside thoracenteses. Hospitalist physicians were most likely to refer to pulmonary medicine and cited lack of time to perform the procedure as the main reason. SBML‐trained residents were most confident about their thoracentesis skills, despite hospitalist physicians having more experience. This study identifies confidence and time as reasons physicians refer thoracenteses rather than perform them at the bedside. Thoracentesis SBML boosts skills and promotes bedside procedures that are safe and less expensive than referrals. Journal of Hospital Medicine 2016;11:792–795. © 2016 Society of Hospital Medicine

© 2016 Society of Hospital Medicine

Internal medicine (IM) residents and hospitalist physicians commonly conduct bedside thoracenteses for both diagnostic and therapeutic purposes.[1] The American Board of Internal Medicine only requires that certification candidates understand the indications, complications, and management of thoracenteses.[2] A disconnect between clinical practice patterns and board requirements may increase patient risk because poorly trained physicians are more likely to cause complications.[3] National practice patterns show that many thoracenteses are referred to interventional radiology (IR).[4] However, research links performance of bedside procedures to reduced hospital length of stay and lower costs, without increasing risk of complications.[1, 5, 6]

Simulation‐based education offers a controlled environment where trainees improve procedural knowledge and skills without patient harm.[7] Simulation‐based mastery learning (SBML) is a rigorous form of competency‐based education that improves clinical skills and reduces iatrogenic complications and healthcare costs.[5, 6, 8] SBML also is an effective method to boost thoracentesis skills among IM residents.[9] However, there are no data to show that thoracentesis skills acquired in the simulation laboratory transfer to clinical environments and affect referral patterns.

We hypothesized that a thoracentesis SBML intervention would improve skills and increase procedural self‐confidence while reducing procedure referrals. This study aimed to (1) assess the effect of thoracentesis SBML on a cohort of IM residents' simulated skills and (2) compare traditionally trained (nonSBML‐trained) residents, SBML‐trained residents, and hospitalist physicians regarding procedure referral patterns, self‐confidence, procedure experience, and reasons for referral.

METHODS AND MATERIALS

Study Design

We surveyed physicians about thoracenteses performed on patients cared for by postgraduate year (PGY)‐2 and PGY‐3 IM residents and hospitalist physicians at Northwestern Memorial Hospital (NMH) from December 2012 to May 2015. NMH is an 896‐bed, tertiary academic medical center, located in Chicago, Illinois. A random sample of IM residents participated in a thoracentesis SBML intervention, whereas hospitalist physicians did not. We compared referral patterns, self‐confidence, procedure experience, and reasons for referral between traditionally trained residents, SBML‐trained residents, and hospitalist physicians. The Northwestern University Institutional Review Board approved this study, and all study participants provided informed consent.

At NMH, resident‐staffed services include general IM and nonintensive care subspecialty medical services. There are also 2 nonteaching floors staffed by hospitalist attending physicians without residents. Thoracenteses performed on these services can either be done at the bedside or referred to pulmonary medicine or IR. The majority of thoracenteses performed by pulmonary medicine occur at the patients' bedside, and the patients also receive a clinical consultation. IR procedures are done in the IR suite without additional clinical consultation.

Procedure

One hundred sixty residents were available for training over the study period. We randomly selected 20% of the approximately 20 PGY‐2 and PGY‐3 IM residents assigned to the NMH medicine services each month to participate in SBML thoracentesis training before their rotation. Randomly selected residents were required to undergo SBML training but were not required to participate in the study. This selection process was repeated before every rotation during the study period. This randomized wait‐list control method allowed residents to serve as controls if not initially selected for training and remain eligible for SBML training in subsequent rotations.

Intervention

The SBML intervention used a pretest/post‐test design, as described elsewhere.[9] Residents completed a clinical skills pretest on a thoracentesis simulator using a previously published 26‐item checklist.[9] Following the pretest, residents participated in 2, 1‐hour training sessions including a lecture, video, and deliberate practice on the simulator with feedback from an expert instructor. Finally, residents completed a clinical skills post‐test using the checklist within 1 week from training (but on a different day) and were required to meet or exceed an 84.3% minimum passing score (MPS). The entire training, including pre‐ and post‐tests, took approximately 3 hours to complete, and residents were given an additional 1 hour refresher training every 6 months for up to a year after original training. We compared pre‐ and post‐test checklist scores to evaluate skills improvement.

Thoracentesis Patient Identification

The NMH electronic health record (EHR) was used to identify medical service inpatients who underwent a thoracentesis during the study period. NMH clinicians must place an EHR order for procedure kits, consults, and laboratory analysis of thoracentesis fluid. We developed a real‐time query of NMH's EHR that identified all patients with electronic orders for thoracenteses and monitored this daily.

Physician Surveys

After each thoracentesis, we surveyed the PGY‐2 or PGY‐3 resident or hospitalist caring for the patient about the procedure. A research coordinator, blind to whether the resident received SBML, performed the surveys face‐to‐face on Monday to Friday during normal business hours. Residents were not considered SBML‐trained until they met or exceeded the MPS on the simulated skills checklist at post‐test. Surveys occurred on Monday for procedures performed on Friday evening through Sunday. Survey questions asked physicians about who performed the procedure, their procedural self‐confidence, and total number of thoracenteses performed in their career. For referred procedures, physicians were asked about reasons for referral including lack of confidence, work hour restrictions (residents only), and low reimbursement rates.[10] There was also an option to add other reasons.

Measurement

The thoracentesis skills checklist documented all required steps for an evidence‐based thoracentesis. Each task received equal weight (0 = done incorrectly/not done, 1 = done correctly).[9] For physician surveys, self‐confidence about performing the procedure was rated on a scale of 0 = not confident to 100 = very confident. Reasons for referral were scored on a Likert scale 1 to 5 (1 = not at all important, 5 = very important). Other reasons for referral were categorized.

Statistical Analysis

The clinical skills pre‐ and post‐test checklist scores were compared using a Wilcoxon matched pairs rank test. Physician survey data were compared between different procedure performers using the 2 test, independent t test, analysis of variance (ANOVA), or Kruskal‐Wallis test depending on data properties. Referral patterns measured by the Likert scale were averaged, and differences between physician groups were evaluated using ANOVA. Counts of other reasons for referral were compared using the 2 test. We performed all statistical analyses using IBM SPSS Statistics version 23 (IBM Corp., Armonk, NY).

RESULTS

Thoracentesis Clinical Skills

One hundred twelve (70%) residents were randomized to SBML, and all completed the protocol. Median pretest scores were 57.6% (interquartile range [IQR] 43.376.9), and final post‐test mastery scores were 96.2 (IQR 96.2100.0; P < 0.001). Twenty‐three residents (21.0%) failed to meet the MPS at initial post‐test, but met the MPS on retest after <1 hour of additional training.

Physician Surveys

The EHR query identified 474 procedures eligible for physician surveys. One hundred twenty‐two residents and 51 hospitalist physicians completed surveys for 472 procedures (99.6%); 182 patients by traditionally trained residents, 145 by SBML‐trained residents, and 145 by hospitalist physicians. As shown in Table 1, 413 (88%) of all procedures were referred to another service. Traditionally trained residents were more likely to refer to IR compared to SBML‐trained residents or hospitalist physicians. SBML‐trained residents were more likely to perform bedside procedures, whereas hospitalist physicians were most likely to refer to pulmonary medicine. SBML‐trained residents were most confident in their procedural skills, despite hospitalist physicians performing more actual procedures.

Characteristics of 472 Thoracentesis Procedures Described on Surveys of Traditionally Trained Residents, SBML‐Trained Residents, and Hospitalist Physicians
Traditionally Trained Resident Surveys, n = 182 SBML‐Trained Resident Surveys, n = 145 Hospitalist Physician Surveys, n = 145 P Value
  • NOTE: Abbreviations: IQR, interquartile range; IR, interventional radiology; SBML, simulation‐based mastery learning; SD, standard deviation. *Scale of 0 = not at all confident to 100 = very confident.

Bedside procedures, no. (%) 26 (14.3%) 32 (22.1%) 1 (0.7%) <0.001
IR procedures, no. (%) 119 (65.4%) 74 (51.0%) 82 (56.6%) 0.029
Pulmonary procedures, no. (%) 37 (20.3%) 39 (26.9%) 62 (42.8%) <0.001
Procedure self‐confidence, mean (SD)* 43.6 (28.66) 68.2 (25.17) 55.7 (31.17) <0.001
Experience performing actual procedures, median (IQR) 1 (13) 2 (13.5) 10 (425) <0.001

Traditionally trained residents were most likely to rate low confidence as reasons why they referred thoracenteses (Table 2). Hospitalist physicians were more likely to cite lack of time to perform the procedure themselves. Other reasons were different across groups. SBML‐trained residents were more likely to refer because of attending preference, whereas traditionally trained residents were mostly like to refer because of high risk/technically difficult cases.

Reasons Provided for Referral of 413 Thoracentesis Procedures Between Traditionally Trained Residents, SBML‐Trained Residents, and Hospitalist Physicians
Traditionally Trained Residents, n = 156 SBML‐Trained Residents, n = 113 Hospitalist Physicians, n = 144 P Value
  • NOTE: Abbreviations: IR, interventional radiology; SBML, simulation‐based mastery learning; SD, standard deviation. *Mean score on a 5‐point Likert scale (1 = not at all important, 5 = very important). Some expected counts are less than 5; 2 test may be invalid.

Lack of confidence to perform procedure, mean (SD)* 3.46 (1.32) 2.52 (1.45) 2.89 (1.60) <0.001
Work hour restrictions, mean (SD) * 2.05 (1.37) 1.50 (1.11) n/a 0.001
Low reimbursement, mean (SD)* 1.02 (0.12) 1.0 (0) 1.22 (0.69) <0.001
Other reasons for referral, no. (%)
Attending preference 8 (5.1%) 11 (9.7%) 3 (2.1%) 0.025
Don't know how 6 (3.8%) 0 0 0.007
Failed bedside 0 2 (1.8%) 0 0.07
High risk/technically difficult case 24 (15.4%) 12 (10.6%) 5 (3.5%) 0.003
IR or pulmonary patient 5 (3.2%) 2 (1.8%) 4 (2.8%) 0.77
Other IR procedure taking place 11 (7.1%) 9 (8.0%) 4 (2.8%) 0.13
Patient preference 2 (1.3%) 7 (6.2%) 2 (3.5%) 0.024
Time 9 (5.8%) 7 (6.2%) 29 (20.1%) <0.001

DISCUSSION

This study confirms earlier research showing that thoracentesis SBML improves residents' clinical skills, but is the first to use a randomized study design.[9] Use of the mastery model in health professions education ensures that all learners are competent to provide patient care including performing invasive procedures. Such rigorous education yields downstream translational outcomes including safety profiles comparable to experts.[1, 6]

This study also shows that SBML‐trained residents displayed higher self‐confidence and performed significantly more bedside procedures than traditionally trained residents and more experienced hospitalist physicians. Although the Society of Hospital Medicine considers thoracentesis skills a core competency for hospitalist physicians,[11] we speculate that some hospitalist physicians had not performed a thoracentesis in years. A recent national survey showed that only 44% of hospitalist physicians performed at least 1 thoracentesis within the past year.[10] Research also shows a shift in medical culture to refer procedures to specialty services, such as IR, by over 900% in the past 2 decades.[4] Our results provide novel information about procedure referrals because we show that SBML provides translational outcomes by improving skills and self‐confidence that influence referral patterns. SBML‐trained residents performed almost a quarter of procedures at the bedside. Although this only represents an 8% absolute difference in bedside procedures compared to traditionally trained residents, if a large number of residents are trained using SBML this results in a meaningful number of procedures shifted to the patient bedside. According to University HealthSystem Consortium data, in US teaching hospitals, approximately 35,325 thoracenteses are performed yearly.[1] Shifting even 8% of these procedures to the bedside would result in significant clinical benefit and cost savings. Reduced referrals increase additional bedside procedures that are safe, cost‐effective, and highly satisfying to patients.[1, 12, 13] Further study is required to determine the impact on referral patterns after providing SMBL training to attending physicians.

Our study also provides information about the rationale for procedure referrals. Earlier work speculates that financial incentive, training and time may explain high procedure referral rates.[10] One report on IM residents noted an 87% IR referral rate for thoracentesis, and confirmed that both training and time were major reasons.[14] Hospitalist physicians reported lack of time as the major factor leading to procedural referrals, which is problematic because bedside procedures yield similar clinical outcomes at lower costs.[1, 12] Attending preference also prevented 11 additional bedside procedures in the SBML‐trained group. Schedule adjustments and SBML training of hospitalist physicians should be considered, because bundled payments in the Affordable Care Act may favor shifting to the higher‐value approach of bedside thoracenteses.[15]

Our study has several limitations. First, we only performed surveys at 1 institution and the results may not be generalizable. Second, we relied on an electronic query to alert us to thoracenteses. Our query may have missed procedures that were unsuccessful or did not have EHR orders entered. Third, physicians may have been surveyed more than once for different or the same patient(s), but opinions may have shifted over time. Fourth, some items such as time needed to be written in the survey and were not specifically asked. This could have resulted in under‐reporting. Finally, we did not assess the clinical outcomes of thoracenteses in this study, although earlier work shows that residents who complete SBML have safety outcomes similar to IR.[1, 6]

In summary, IM residents who complete thoracentesis SBML demonstrate improved clinical skills and are more likely to perform bedside procedures. In an era of bundled payments, rethinking current care models to promote cost‐effective care is necessary. We believe providing additional education, training, and support to hospitalist physicians to promote bedside procedures is a promising strategy that warrants further study.

Acknowledgements

The authors acknowledge Drs. Douglas Vaughan and Kevin O'Leary for their support and encouragement of this work. The authors also thank the internal medicine residents at Northwestern for their dedication to patient care.

Disclosures: This project was supported by grant R18HS021202‐01 from the Agency for Healthcare Research and Quality (AHRQ). AHRQ had no role in the preparation, review, or approval of the manuscript. Trial Registration: ClinicalTrials.gov NCT01898247 (https://clinicaltrials.gov/ct2/show/NCT01898247?term=thoracentesis+and+s... rank=1). The authors report no conflicts of interest.

References

Online-Only Materials

   Comments ()