Original Research

Use of simulation‐based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit

Abstract

Abstract

BACKGROUND:

Central venous catheter (CVC) insertions are performed frequently by internal medicine residents. Complications, including arterial puncture and pneumothorax, decrease when operators use fewer needle passes to insert the CVC. In this study, we evaluated the effect of simulation‐based mastery learning on CVC insertion skill.

DESIGN:

This was a cohort study of internal jugular (IJ) and subclavian (SC) CVC insertions by 41 internal medicine residents rotating through the medical intensive care unit (MICU) over a five‐month period. Thirteen traditionally‐trained residents were surveyed about the number of needle passes, complications, and procedural self‐confidence on CVCs inserted in the MICU. Concurrently, 28 residents completed simulation‐based training in IJ and SC CVC insertions. Simulator‐trained residents were expected to perform CVC insertions to mastery standards on a central line simulator. Simulator‐trained residents then rotated through the MICU and were surveyed regarding CVC placement. The impact of simulation training was assessed by comparing group survey results.

RESULTS:

No resident met the minimum passing score (MPS) (79.1%) for CVC insertion at baseline: mean (M) (IJ) = 48.4%, standard deviation (SD) = 23.1, M(SC) = 45.2%, SD = 26.3. All residents met or exceeded the MPS at testing after simulation training: M(IJ) = 94.8%, SD = 10.0, M(SC) = 91.1%, SD = 17.8 (p < 0.001). In the MICU, simulator‐trained residents required fewer needle passes to insert a CVC than traditionally‐trained residents: M = 1.79, SD = 1.0 versus M = 2.78, SD = 1.77 (p = 0.04). Simulator‐trained residents displayed more self‐confidence about their procedural skills: (M = 81, SD = 11 versus M = 68, SD = 20, p = 0.02).

CONCLUSIONS:

Simulation‐based mastery learning increased residents' skills in simulated CVC insertion, decreased the number of needle passes when performing actual procedures, and increased resident self‐confidence. Journal of Hospital Medicine 2009;4:397–403. © 2009 Society of Hospital Medicine.

Copyright © 2009 Society of Hospital Medicine

Central venous catheter (CVC) insertions are commonly performed at the bedside in medical intensive care unit (MICU) settings. Internal medicine residents are required to demonstrate knowledge regarding CVC indications, complications, and sterile technique,1 and often perform the procedure during training. Education in CVC insertion is needed because many internal medicine residents are uncomfortable performing this procedure.2 CVC insertion also carries the risk of potentially life‐threatening complications including infection, pneumothorax, arterial puncture, deep vein thrombosis, and bleeding. Education and training may also contribute to improved patient care because increased physician experience with CVC insertion reduces complication risk.3, 4 Similarly, a higher number of needle passes or attempts during CVC insertion correlates with mechanical complications such as pneumothorax or arterial punctures.48 Pneumothorax rates for internal jugular (IJ) CVCs have been reported to range from 0% to 0.2% and for subclavian (SC) CVCs from 1.5% to 3.1%.4, 5 The arterial puncture rate for IJ CVCs ranges from 5.0% to 9.4% and for SC CVCs from 3.1% to 4.9%.4, 5 Proper use of ultrasound to assist with IJ CVC insertion has been shown to decrease these mechanical complications.4, 5 However, studies of ultrasound use with SC CVC insertion have mixed results.4

Simulation‐based training has been used in medical education to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.9, 10 We previously used simulation‐based mastery learning to improve the thoracentesis and advanced cardiac life support (ACLS) skills of internal medicine residents.11, 12 Although a few small studies have linked simulation‐based interventions to improved quality of care,1319 more work is needed to show that results from a simulated environment transfer to actual patient care.

This study had 2 aims. The first was to expand our simulation‐based mastery learning to CVC insertion using a CVC simulator and ultrasound device. The second was to assess quality indicators (number of needle passes, pneumothorax, arterial punctures, and need for catheter adjustment) and resident confidence related to actual CVC insertions in the MICU before and after an educational intervention.

Materials and Methods

Design

This was a cohort study20 of IJ and SC CVC insertions by 41 second‐ and third‐year internal medicine residents rotating through the MICU in a university‐affiliated program from October 2006 to February 2007. The Northwestern University Institutional Review Board approved the study. All study participants were required to give informed consent prior to participation.

Thirteen residents rotated through the MICU during a 6‐week preintervention phase. These residents served as a traditionally trained group that did not receive CVC insertion simulator training. Simultaneously, 28 residents who rotated through the MICU later in the study period received simulation‐based training in CVC insertion and served as the simulator‐trained group (Figure 1). Demographic data were obtained from the participants including age, gender, ethnicity, year of training, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2.

Figure 1
Timeline of CVC training and clinical rotations.

Simulator‐trained residents underwent baseline skill assessment (pretest) using a 27‐item checklist in IJ and SC CVC insertions (see Appendix). Checklists were developed by one author (J.H.B.) using appropriate references4, 5 and a step‐by‐step process,21 and reviewed for completeness by another author with expertise in checklist development (D.B.W.). Each skill or other action was listed in order and given equal weight. A dichotomous scoring scale of 1 = done correctly and 0 = done incorrectly/not done was imposed for each item. Assessments were performed using Simulab's CentralLineMan. This model features realistic tissue with ultrasound compatibility, an arterial pulse, and self‐sealing veins and skins. Needles, dilators, and guidewires can be inserted and realistic venous and arterial pressures demonstrated (Figure 2).

Figure 2
Resident training on the CVC simulator.

Residents in the simulator‐trained group received two, 2‐hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator, and feedback.22 Education sessions contained standardized didactic material on CVC indications and complications, as well as a stepwise demonstration of IJ and SC CVC insertions using ultrasound and landmark techniques. These sessions were supervised by a senior hospitalist faculty member with expertise in CVC insertions (J.H.B.). Residents were expected to use the ultrasound device for all IJ CVC insertions. However, its use was optional for SC CVC insertion. After training, residents were retested (posttest) and required to meet or exceed a minimum passing score (MPS) set by an expert panel for both IJ and SC procedures.23 This 11 member expert panel provided item‐based (Angoff) and group‐based (Hofstee) judgments on the 27‐item checklists as described previously.23

Residents who did not achieve the MPS had more deliberate practice and were retested until the MPS was reached; the key feature of mastery learning.24 After completing simulation‐based mastery learning in CVC insertion, the 28 simulator‐trained residents rotated through the MICU.

Data Collection

All pretests and posttests (using the 27‐item checklist) were graded by a single unblinded instructor (J.H.B.) and were videotaped. Another faculty instructor with expertise in scoring clinical skills examinations and blind to pre‐post status (D.B.W.) rescored a random 50% sample of the tests to assess interrater reliability.

Data regarding actual CVC insertions in the MICU were collected by contacting all MICU residents daily during the study period. This allowed for CVC insertions to be identified within 24 hours. All survey data were collected anonymously. The primary inserter of each CVC was questioned about quality indicators and procedural self‐confidence concerning CVC placement. CVCs primarily inserted by nonstudy subjects (first‐year residents, emergency medicine residents, pulmonary‐critical care medicine faculty members, and subspecialty fellows) or CVC placements that were supervised, but not directly placed by study participants, were excluded.

Outcome Measures

Pretest and posttest checklist scores from simulator‐trained residents were compared to measure the impact of training sessions. Residents rotating through the MICU were asked about several quality indicators related to actual CVC insertions. Quality indicators include: (1) number of needle passes required during the procedure (skin punctures); (2) presence of complications including pneumothorax and arterial puncture; and (3) need for CVC adjustment after chest x‐ray. Participants were also questioned regarding their confidence in CVC insertion using a 100 point scale (0 = not confident and 100 = very confident). Survey results from the 28 simulator‐trained residents were compared to results from the 13 traditionally‐trained residents.

Data Analysis

Checklist score reliability was estimated by calculating interrater reliability, the preferred method for assessments that depend on human judges, using the kappa () coefficient adjusted25, 26 using the formula of Brennan and Prediger.27 Within‐group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t‐tests.

MICU survey results were compared using t‐tests. Traditionally‐trained and simulator‐trained groups were assessed for demographic differences using t‐tests and the chi‐square statistic. Spearman's rank correlation coefficient was used to assess for relationships between resident self‐confidence and quality indicators. All analyses were preformed using SPSS statistical software, version 16.0 (SPSS, Inc., Chicago, IL).

Results

All eligible residents participated in the study and completed the entire protocol. There was no significant difference in age, gender, ethnicity, year of training, or USMLE Step 1 and 2 scores between the traditionally‐trained and simulator‐trained groups.

Interrater reliability measured by the mean kappa coefficient was very high (n = 0.94) across the 27 IJ and SC checklist items. No resident met the MPS (79.1%) for CVC insertion at baseline testing. In the simulator‐trained group, 25 of 28 (89%) residents achieved SC skill mastery and 27 of 28 (96%) achieved IJ skill mastery within the standard four hour curriculum. All residents subsequently reached the MPS with less than one hour of additional practice time. A graphic portrait of the residents' pretest and posttest performance on the simulated CVC clinical skills examination with descriptive statistics is shown in Figure 3. After the educational intervention, posttest scores significantly improved (p < 0.001), to meet or exceed the MPS.

Figure 3
Mean scores and standard deviations on the simulator‐based skills exam before and after the educational intervention. MPS = 79.1%.

Traditionally trained and simulator‐trained residents independently inserted 46 CVCs during the study period. Simulator‐trained residents required significantly fewer needle passes to insert all actual CVCs in the MICU compared to traditionally trained residents: mean (M) = 1.79, standard deviation (SD) = 1.03 versus M = 2.78, SD = 1.77 (p = 0.04). As shown in Table 1, the groups did not differ in pneumothorax, arterial puncture, or mean number of CVC adjustments. In addition, the groups did not differ in use of ultrasound for IJ or SC CVC insertions. One IJ CVC was inserted without ultrasound in the traditionally‐trained group; 2 were inserted without ultrasound in the simulator‐trained group. Ultrasound was not used during any SC CVC insertions in the traditionally‐trained group and was used for 1 SC CVC insertion in the simulator‐trained group.

Comparison of Traditionally Trained Residents vs. Simulator trained Residents in Self‐confidence and CVC Quality Indicators During Actual CVC Insertions in the MICU
 Internal Jugular and Subclavian CVCs
Traditionally Trained ResidentsSimulator Trained ResidentsP value
  • p < 0.05.

  • Abbreviations: CVC, central venous catheter; MICU, medical intensive care unit; n/a, not applicable.

Number of attempts during insertion [mean (SD)]2.78 (1.77)1.79 (1.03)0.04*
Pneumothorax (number)00n/a
Arterial puncture (%)1170.65
CVC adjustment (%)1580.52
Confidence (%) [mean (SD)]68 (20)81 (11)0.02*

Simulator‐trained residents displayed more self‐confidence about their procedural skills than traditionally‐trained residents (M = 81, SD = 11 versus M = 68, SD = 20, p = 0.02). Spearman correlations showed no practical association between resident self‐confidence and performance on CVC insertion quality indicators.

Discussion

This study demonstrates the use of a mastery learning model to develop CVC insertion skills to a high achievement level among internal medicine residents. Our data support prior work showing that procedural skills that are poor at baseline can be increased significantly using simulation‐based training and deliberate practice.1118, 28 This report on CVC insertion adds to the growing body of literature showing that simulation training complements standard medical education,1119, 28 and expands the clinical application of the mastery model beyond thoracentesis and ACLS.11, 12 Use of the mastery model described in this study also has important implications for patients. In our training program, residents are required to demonstrate procedural mastery in a simulated environment before independently performing a CVC insertion on an actual patient. This is in sharp contrast to the traditional clinical model of procedural training at the bedside, and may be used in other training programs and with other invasive procedures.

The second aim of our study was to determine the impact of simulation‐based training on actual clinical practice by residents in the MICU. To our knowledge, no prior study has demonstrated that simulation‐based training in CVC insertion improves patient outcomes. We believe our results advance what is known about the impact of simulation‐based training because simulator‐trained residents in this study performed actual CVC insertions in the MICU using significantly fewer needle passes. Needle passes have been used by other investigators as a surrogate measure for reduced CVC‐associated complications because mechanical complications rise exponentially with more than two insertion attempts.47, 29 We believe this finding demonstrates transfer of skill acquired from simulation‐based training to the actual clinical environment. It is possible that ultrasound training accounts for the improvement in the simulator‐trained group. However, we do not believe that ultrasound training is entirely responsible as prior work has shown that deliberate practice using mastery learning without ultrasound significantly improved resident performance of thoracentesis11 and ACLS12, 19 procedures. We did not show a significant reduction in complications such as pneumothorax or arterial puncture. This is likely due to the small sample size and the low number of procedures and complications during the study period.

Our results also show that resident self‐confidence regarding actual CVC insertions improved after simulation training. These findings are similar to prior reports linking improved confidence among trainees after simulation‐based training in CVC insertion.29, 30 Our results did not reveal a correlation between improved self‐confidence and clinical skill acquisition. Linking improved self‐confidence to improved clinical skill is important because self‐assessment does not always correlate with performance ability.31, 32

More study is needed to evaluate the impact of simulation‐based training on the quality of CVC insertions by trainees. Mechanisms shown to decrease complications of CVC placement include use of ultrasound,4, 7, 3336 full sterile barrier technique,3739 chlorhexidine skin preparations,4042 and nurse‐physician education.43 Our simulation‐training program incorporates each of these elements. We plan to expand our simulation‐based training intervention to a larger sample size to determine its impact on mechanical and infectious complication rates linked to CVC insertion.

This study has several limitations. It was performed at a single institution over a short time period. However, demonstration of significantly fewer needle passes and improved resident self‐confidence after simulator training are important findings that warrant further study. It was impossible to blind raters during the skills assessment examination about whether the resident was performing a pretest or posttest. This was accounted for by using a second rater, who was blind to the pretest and posttest status of the examinee. The arterial puncture rate of 7% among simulator‐trained residents was higher than expected, although it remains within published ranges.4, 5 Also, a low total number of CVCs were evaluated during the study. This is likely due to strict exclusion criteria employed in order to study the impact of simulation training. For example, CVC insertions were only evaluated if they were actually performed by study residents (supervised insertions were excluded) and femoral catheters were not evaluated. We did not track clinical experience with CVC insertion by residents before the study. Residents who were simulator‐trained may have had more clinical experience with CVC insertion and this may have impacted their performance. However, residents did not differ in year of training or clinical rotations, and there is clear evidence that clinical training is not a proxy for skill acquisition.44 Finally, outcome data were measured via resident questionnaires that relied on resident recall about CVC insertion rather than observer ratings. This method was selected because observer ratings could not be standardized given the large number of clinical supervisors in the MICU over the study period. Information about needle passes and arterial puncture also may not be documented in procedural notes and could not be obtained by medical record review. We attempted to minimize recall bias by surveying residents within 24 hours of CVC placement.

In conclusion, this study demonstrates that simulation‐based training and deliberate practice in a mastery learning setting improves performance of both simulated and actual CVC insertions by internal medicine residents. Procedural training remains an important component of internal medicine training, although internists are performing fewer invasive procedures now than in years past.45, 46 Use of a mastery model of CVC insertion requires that trainees demonstrate skill in a simulated environment before independently performing this invasive procedure on patients. Further study is needed to assess clinical outcomes such as reduced CVC‐related infections and mechanical complications after simulation‐based training.

Acknowledgements

The authors thank the Northwestern University internal medicine residents for their dedication to education and patient care. They acknowledge Drs. J. Larry Jameson and Charles Watts for their support and encouragement of this work.

Appendix

Central Venous Catheter Insertion Checklists for Simulation‐based Education 0, 0

Central Venous Catheter Placement (IJ)
  • Skill Key: A = Done Correctly B = Done Incorrectly C = Not Done.

Informed consent obtained: must do allABC
Benefits
Risks
Consent given
Place the patient in slight Trendelenburg positionABC
Flush the ports on the catheter with sterile salineABC
Clamp each port (ok to keep brown port open)ABC
Remove brown port from end of catheter to accommodate wireABC
Area is cleaned with chlorhexadineABC
Resident gets in sterile gown, gloves, hat and maskABC
Area is draped in usual sterile fashion (must be full body drape)ABC
The ultrasound (US) probe is properly set up with sterile sheath and sonographic gelABC
The vein is localized using anatomical landmarks with the US machineABC
If no US is used this is wrong
The skin is anesthetized with 1% lidocaine in a small whealABC
The deeper structures are anesthetizedABC
Localize the vein with this needle (optional)ABC
Using the large needle or catheter‐ syringe complex, cannulate the vein while aspirating (must be done with US)ABC
Remove the syringe from the needle or advance the catheter into the vein removing both the syringe and needleABC
Advance the guidewire into the vein no more than approximately 1215 cmABC
Knick the skin with the scalpel to advance the dilatorABC
Advance the dilator over the guidewire and dilate the veinABC
Advance the triple lumen over the guidewireABC
Never let go of the guidewireABC
Once the catheter is inserted remove the guidewire in its entiretyABC
Advance the catheter to approx to 1416cm on the right side, 1618 cm on the left sideABC
Ensure there is blood flow/flush each portABC
Secure the catheter in place (suture or staple)ABC
Place dressing over catheterABC
Get a chest x‐rayABC
Notify that the catheter is ok to useABC
Maintain sterile techniqueABC
Central Venous Catheter Placement (Subclavian)
  • Skill Key: A = Done Correctly B = Done Incorrectly C = Not Done

Informed consent obtained: must do allABC
Benefits
Risks
Consent given
Place the patient in slight Trendelenburg positionABC
Flush the ports on the catheter with sterile salineABC
Clamp each port (ok to leave brown port open)ABC
Remove brown port from end of catheter to accommodate wireABC
Area is cleaned with chlorhexadineABC
Resident gets in sterile gown, gloves, hat and maskABC
Area is draped in usual sterile fashion (must be full body drape)ABC
**The US probe is properly set up with sterile sheath and sonographic gel . (MUST DO if use US)ABC
The vein is localized using US machine or anatomical landmarks are verbalizedABC
The skin is anesthetized with 1% lidocaine in a small whealABC
The deeper structures are anesthetized using a larger needle (must verbalize they anesthetize the clavicle)ABC
Localize the vein with this needle (optional)ABC
Using the large needle or catheter syringe complex cannulate the vein while aspirating (optional confirmed by US)ABC
If US was not used then expected to state they are directing the needle to the sternal notchABC
Remove the syringe from the needle or advance the catheter into the vein removing both the syringe and needleABC
Advance the guidewire into the vein no more than approximately 1215 cmABC
Knick the skin with the scalpel to advance the dilatorABC
Advance the dilator over the guidewire and dilate the veinABC
Advance the triple lumen over the guidewireABC
Never let go of the guidewireABC
Once the catheter is inserted remove the guidewire in its entiretyABC
Advance the catheter to approx to 1416cm on the right side, 1618 cm on the left sideABC
Ensure there is blood flow/flush each portABC
Secure the catheter in place (suture or staple)ABC
Place dressing over catheterABC
Get a chest x‐rayABC
Notify that the catheter is ok to useABC
Maintain sterile techniqueABC

References

   Comments