Original Research

Improving internal medicine residents' performance, knowledge, and confidence in central venous catheterization using simulators

Abstract

Abstract

BACKGROUND:

Efficacy of simulators in teaching central venous catheterization (CVC) in an internal medicine residency program is unknown.

OBJECTIVE:

To determine whether or not learning CVC on simulators is associated with improvement in performance of CVC, knowledge about the procedure, and self‐reported confidence.

METHODS:

All consenting first‐year internal medicine residents who completed training in CVC on simulators were included. Participants were evaluated pre‐ and post‐training by video‐recorded CVC insertion and multiple‐choice knowledge assessments. Procedural technique was rated in a blinded fashion by two independent adjudicators. Knowledge retention and self‐reported confidence were reassessed at 18 months.

MEASUREMENTS:

Primary outcome of CVC performance was assessed based on global rating score (minimum 1, maximum 5). Secondary measures include checklist score (out of ten), knowledge score and self‐reported confidence (6‐point Likert scale ranging from “none” to “complete”).

RESULTS:

Median global rating scores in 30 participants increased from 3.5 (IQR = 3‐4) to 4.5 (IQR = 4‐4.5) (P < 0.001). Checklist score increased from 9 (IQR = 6‐9.5) to 9.5 (IQR = 9‐9.5) (P < 0.001). Knowledge score increased from 65.7 ± 11.9% to 81.2 ± 10.7% (P < 0.001). Confidence increased from 3 (“moderate”, IQR = 2‐3) to 4 (“good”, IQR=3‐4) (P < 0.001). Sixteen participants completed the retention tests. Improvement in knowledge score and confidence at 18 months was retained compared with baseline (P = 0.002 and P < 0.0001 respectively).

CONCLUSIONS:

Use of simulators in teaching CVC in an internal medicine residency program results in improved procedural performance, knowledge, and self‐reported confidence. Improvement in knowledge and confidence was retained at 18 months. Journal of Hospital Medicine 2009;4:410–416. © 2009 Society of Hospital Medicine.

Copyright © 2009 Society of Hospital Medicine

The Accreditation Council for Graduate Medical Education mandates that internal medicine residents develop technical proficiency in performing central venous catheterization (CVC).1 Likewise in Canada, technical expertise in performing CVC is a specific competency requirement in the training objectives in internal medicine by the Royal College of Physicians and Surgeons of Canada.2 For certification in Internal Medicine, the American Board of Internal Medicine expects its candidates to be competent with respect to knowledge and understanding of the indications, contraindications, recognition, and management of complications of CVC.3 Despite the importance of procedural teaching in medical education, most internal medicine residency programs do not have formal procedural teaching.4 A number of potential barriers to teaching CVC exist. A recent study by the American College of Physicians found that fewer general internists perform procedures, compared with internists in the 1980s.5 With fewer internists performing procedures, there may be fewer educators with adequate confidence in teaching these procedures.6 Second, with increasing recognition of patient safety, there is a growing reluctance of patients to be used as teaching subjects for those who are learning CVC.7 Not surprisingly, residents report low comfort in performing CVC.8

Teaching procedures on simulators offers the benefits of allowing both teaching and evaluation of procedures without jeopardizing patient safety9 and is a welcomed variation from the traditional see 1, do 1, teach 1 approach.10 Despite the theoretical benefits of the use of simulators in medical education, the efficacy of their use in teaching internal medicine residents remains understudied.

The purpose of our study was to evaluate the benefits of using simulator models in teaching CVC to internal medicine residents. We hypothesized that participation in a simulator‐based teaching session on CVC would result in improved knowledge and performance of the procedure and confidence.

Methods

Participants

The study was approved by our university and hospital ethics review board. All first‐year internal medicine residents in a single academic institution, who provided written informed consent, were included and enrolled in a simulator curriculum on CVC.

Intervention

All participants completed a baseline multiple choice knowledge test on anatomy, procedural technique, and complications of CVC, as well as a self‐assessment questionnaire on confidence and prior experience of CVC. Upon completion of these, participants were provided access to Internet links of multimedia educational materials on CVC, including references for suggested reading.11, 12 One week later, participants, in small groups of 3 to 5 subjects, attended a 2‐hour simulation session on CVC. For each participant, the baseline performance of an internal jugular CVC on a simulator (Laerdal IV Torso; Laerdal Medical Corp, Wappingers Falls, NY) was videotaped in a blinded fashion. Blinding was done by assigning participants study numbers. No personal identifying information was recorded, as only gowned and gloved hands, arms, and parts of the simulator were recorded. A faculty member and a senior medical resident then demonstrated to each participant the proper CVC techniques on the simulator, which was followed by an 1‐hour practice session with the simulator. Feedback was provided by the supervising faculty member and senior resident during the practice session. Ultrasound‐guided CVC was demonstrated during the session, but not included for evaluation. postsession, each participant's repeat performance of an internal jugular CVC was videotaped in a blinded fashion, similar to that previously described. All participants then repeated the multiple‐choice knowledge assessment 1 week after the simulation session. For assessment of knowledge retention and confidence, all participants were invited to complete the multiple‐choice knowledge assessment at 18 months after training on simulation.

Measurements and Outcomes

We assessed knowledge on CVC based on performance on a 20‐item multiple‐choice test (scored out of 20, presented as a percentage). Questions were constructed based on information from published literature,11 covering areas of anatomy, procedural technique, and complications of CVC. Procedural skills were assessed by 2 raters using a previously validated modified global rating scale (5‐point Likert scale design)13, 14 during review of videotaped performances (Appendix A). Our primary outcome was an overall global rating score. Review was done by 2 independent, blinded raters who were not aware of whether the performances were recorded presimulation or postsimulation training sessions. Weighted Kappa scores for interobserver agreement between video reviewers ranged between 0.35 (95% confidence interval, 0.250.60) for the global average rating score to 0.43 (95% confidence interval, 0.170.53) for the procedural checklist. These Kappa scores indicate fair to moderate agreement, respectively.15

Secondary outcomes included other domains on the global rating scale such as: time and motion, instrument handling, flow of operation, knowledge of instruments,13, 14 checklist score, number of attempts to locate vein and insert catheter, time to complete the procedure, and self‐reported confidence. The criteria of this checklist score were based on a previously published 14‐item checklist,16 assessing for objective completion of specific tasks (Appendix B). Participants were scored based on 10 items, as the remaining 4 items on the original checklist were not applicable to our simulation session (that is, the items on site selection, catheter selection, Trendelenburg positioning, and sterile sealing of site). In our study, all participants were asked to complete an internal jugular catheterization with a standard kit provided, without the need to dress the site after line insertion. Confidence was assessed by self‐report on a 6‐point Likert scale ranging from none to complete.

Statistical Analysis

Comparisons between pretraining and posttraining were made with the use of McNemar's test for paired data and Wilcoxon signed‐rank test where appropriate. Usual descriptive statistics using mean standard deviation (SD), median and interquartile range (IQR) are reported. Comparisons between presession, postsession, and 18‐month data were analyzed using 1‐way analysis of variance (ANOVA), repeated measures design. Homogeneity of covariance assumption test indicated no evidence of nonsphericity in data (Mauchly's criterion; P = 0.53). All reported P values are 2‐sided. Analyses were conducted using the SAS version 9.1 (SAS Institute, Cary, NC).

Results

Of the 33 residents in the first‐year internal medicine program, all consented to participate. Thirty participants (91%) completed the study protocol between January and June 2007. The remaining three residents completed the presession assessments and the simulation session, but did not complete the postsession assessments. These were excluded from our analysis. At baseline, 20 participants had rotated through a previous 1‐month intensive care unit (ICU) block, 8 residents had no prior ICU training, and 2 residents had 2 months of ICU training. Table 1 summarizes the residents' baseline CVC experience prior to the simulation training. In general, participants had limited experience with CVC at baseline, having placed very few internal jugular CVCs (median = 4).

Baseline Median Number of Central Venous Catheterizations Performed or Observed
Femoral LinesInternal Jugular LinesSubclavian Lines
  • NOTE: n = 30.

  • Abbreviation: IQR, interquartile range.

Number observed (IQR)4 (25)5 (37)2 (13)
Number attempted or performed (IQR)3 (15)4 (28)0 (02)
Number performed successfully without assistance (IQR)2 (03)4 (15)0 (01)

Simulation training was associated with an increase in knowledge. Mean scores on multiple‐choice tests increased from 65.7 11.9% to 81.2 10.7 (P < 0.001). Furthermore, simulation training was associated with improvement in CVC performance. Median global average rating score increased from 3.5 (IQR = 34) to 4.5 (IQR = 44.5) (P < 0.001).

The procedural checklist median score increased from 9 (IQR = 69.5) to 9.5 (IQR = 99.5) (P <.001), median time required for completion of CVC decreased from 8 minutes 22 seconds (IQR = 7 minutes 37 seconds to 12 minutes 1 second) to 6 minutes 43 seconds (IQR = 6 minutes to 7 minutes 25 seconds) (P = 0.002). Other performance measures are summarized in Table 2. Last, simulation training was associated with an increase in self‐rated confidence. The median confidence rating (0 lowest, 5 highest) increased from 3 (moderate, IQR = 23) to 4 (good, IQR = 34) (P < 0.001). The overall satisfaction with the CVC simulation course was good (Table 3) and all 30 participants felt this course should continue to be offered.

Central Venous Catheterization Performance Scores Presimulation Training and Postsimulation Training
PrePostP Value
  • NOTE: n = 30.

  • Abbreviation: IQR, interquartile range.

Number of attempts to locate vein; n (%) 0.01
11626
2144
Number of attempts to insert catheter; n (%) 0.32
12729
>131
Median time and motion score (IQR)4 (34)5 (45)<0.001
Median instrument handling score (IQR)4 (34)5 (45)<0.001
Median flow of operation score (IQR)4 (35)5 (45)<0.001
Median knowledge of instruments score (IQR)4 (35)5 (45)0.004
Subjective Rating of Simulation Training by Internal Medicine Residents
Agree StronglyAgreeNeutralDisagreeDisagree Strongly
  • NOTE: n = 30.

Course improved technical ability to insert central lines321600
Course decreased anxiety related to line placement622200
Course improved ability to deliver patient care622200

Knowledge Retention

Sixteen participants completed the knowledge‐retention multiple‐choice tests. No significant differences in baseline characteristics were found between those who completed the knowledge retention tests (n = 16) vs. those who did not (n = 17) in baseline knowledge scores, previous ICU training, gender, and baseline self‐rated confidence.

Mean baseline multiple‐choice test score in the 16 participants who completed the knowledge retention tests was 65.3% 10.6%. Mean score 1 week posttraining was 80.3 9.4%. Mean score at 18 months was 78.4 9.6%. Results from 1‐way ANOVA, repeated‐measures design showed a significant improvement in knowledge scores, F(2, 30) = 14.83, P < 0.0001. Contrasts showed that scores at 1 week were significantly higher than baseline scores (P < 0.0001), and improvement in scores at 18 months was retained, remaining significantly higher than baseline scores (P = 0.002). Median confidence level increased from 3 (moderate) at baseline, to 4 (good) at 1‐week posttraining. Confidence level remained 4 (good) at 18 months (P < 0.0001).

Discussion

CVC is an important procedural competency for internists to master. Indications for CVC range from measuring of central venous pressure to administration of specific medications or solutions such as inotropic agents and total parental nutrition. Its use is important in the care of those who are acutely or critically ill and those who are chronically ill. CVC is associated with a number of potentially serious mechanical complications, including arterial puncture, pneumothorax, and death.11, 17 Proficiency in CVC insertion is found to be closely related to operator experience: catheter insertion by physicians who have performed 50 or more CVC are one‐half as likely to sustain complications as catheter insertion by less experienced physicians.18 Furthermore, training has been found to be associated with a decrease in risk for catheter associated infection19 and pneumothorax.20 Therefore, cumulative experience from repeated clinical encounters and education on CVC play an important role in optimizing patient safety and reducing potential errors associated with CVC. It is well‐documented that patients are generally reluctant to allow medical trainees to learn or practice procedures on them,7 especially in the early phases of training. Indeed, close to 50% of internal medicine residents in 1 study reported poor comfort in performing CVCs.8

Deficiencies in procedural teaching in internal medicine residency curriculum have long been recognized.4 Over the past 20 years, there has been a decrease in the percentage of internists who place central lines in practice.5 In the United States, more than 5 million central venous catheters are inserted every year.21 Presumably, concurrent with this trend is an increasing reliance on other specialists such as radiologists or surgical colleagues in the placement of CVCs. Without a concerted effort for medical educators in improving the quality of procedural teaching, this alarming trend is unlikely to be halted.

With the advance of modern technology and medical education, patient simulation offers a new platform for procedural teaching to take place: a way for trainees to practice without jeopardizing patient safety, as well as an opportunity for formal teaching and supervision to occur,9, 22 serving as a welcomed departure from the traditional see 1, do 1, teach 1 model to procedural teaching. While there is data to support the role of simulation teaching in the surgical training programs,16, 23 less objective evidence exists to support its use in internal medicine residency training programs.

Our study supports the use of simulation in training first‐year internal medicine residents in CVC. Our formal curriculum on CVC involving simulation was associated with an increase in knowledge of the procedure, observed procedural performance, and subjective confidence. Furthermore, this improvement in knowledge and confidence was maintained at 18 months. Our study has several limitations, such as those inherent to all studies using the prestudy and poststudy design. Improvement in performance under study may be a result of the Hawthorne effect. We are not able to isolate the effect of self‐directed learning from the benefits from training using simulators. Furthermore, improvement in knowledge and confidence may be a function of time rather than training. However, by not comparing our results to that from a historical cohort, we hope to minimize confounders by having participants serve as their own control. Results from our study would be strengthened by the introduction of a control group and a randomized design. Future study should include a randomized, controlled crossover educational trial. The second limitation of our study is the low interobserver agreement between video raters, indicating only fair to moderate agreement. Although our video raters were trained senior medical residents and faculty members who frequently supervise and evaluate procedures, we did not specifically train them for the purposes of scoring videotaped performances. Further faculty development is needed. Overall, the magnitudes of the interobserver differences were small (mean global rating scale between raters differed by 1 0.9), and may not translate into clinically important differences. Third, competence on performing CVC on simulators has not been directly correlated with clinical competence on real‐life patients, although a previous study on surgical interns suggested that simulator training was associated with clinical competence on real patients.16 Future studies are warranted to investigate the association between educational benefits from simulation and actual error reductions from CVC during patient encounters. Fourth, our study did not examine residents in the use of ultrasound during simulator training. Use of ultrasound devices has been shown to decrease complication rates of CVC insertion24 and increase rates of success,25 and is recommended as the preferred method for insertion of elective internal jugular CVCs.26 However, because our course was introduced during first‐year residency, when majority of the residents have performed fewer than 5 internal jugular CVCs prior to their simulation training session, the objective of our introductory course was to first ensure technical competency by landmark techniques. Once learners have mastered the technical aspect of line insertion, ultrasound‐guided techniques were introduced later in the residency training curriculum. Fifth, only 48% of participants completed the knowledge assessment at 18 months. Selection bias may be present in our long‐term data. However, baseline comparisons between those who completed the 18 month assessment vs. those who did not demonstrated no obvious differences between the 2 groups. Performance was not examined in our long‐term data. Demonstration of retention of psychomotor skills would be reassuring. However, by 18 months, many of our learners had completed a number of procedurally‐intensive rotations. Therefore, because retention of skills could not be reliably attributed to our initial training in the simulator laboratory, the participants were not retested in their performance of the CVC. Future study should include a control group that undergoes delayed training at 18 months to best assess the effect of simulation training on performance. Sixth, only overall confidence in performing CVC was captured. Confidence in each venous access site pretraining and posttraining was not assessed and warrants further study. Last, our study did not document overall costs of delivering the program, including cost of the simulator, facility, supplies, and stipends for teaching faculty. One full‐time equivalent (FTE) staff has previously been estimated to be a requirement for the success of the program.9 Our program required 3 medical educators, all of whom are clinical faculty members, as well as the help of 4 chief medical residents.

Despite our study's limitations, our study has a number of strengths. To our knowledge, our study is the first of its kind examining the use of simulators on CVC procedural performance, with the use of video‐recording, in an independent and blinded manner. Blinding should minimize bias associated with assessments of resident competency by evaluators who may be aware of residents' previous experience. Second, we were able to recruit all of our first‐year residents for our pretests and posttests, although 3 did not complete the study in its entirety due to on‐call or post‐call schedules. We were able to obtain detailed assessment of knowledge, performance, and subjective confidence before and after the simulation training on our cohort of residents and were able to demonstrate an improvement in knowledge, skills, and confidence. Furthermore, we were able to demonstrate that benefits in knowledge and confidence were retained at 18 months. Last, we readily implemented our formal CVC curriculum into the existing residency training schedule. It requires a single, 2‐hour session in addition to brief pretest and posttest surveys. It requires minimal equipment, using only 2 CVC mannequins, 4 to 6 central line kits, and an ultrasound machine. The simulation training adheres to and incorporates key elements that have been previously identified as effective teaching tools for simulators: feedback, repetitive practice, and curriculum integration.27 With increasing interest in the use of medical simulation for training and competency assessment in medical education,10 our study provides evidence in support of its use in improving knowledge, skills, and confidence. One of the most important goals of using simulators as an educational tool is to improve patient safety. While assessing for whether or not training on simulators lead to improved patient outcomes is beyond the scope of our study, this question merits further study and should be the focus of future studies.

Conclusions

Training on CVC simulators was associated with an improvement in performance, and increase in knowledge assessment scores and self‐reported confidence in internal medicine residents. Improvement in knowledge and confidence was maintained at 18 months.

Acknowledgements

The authors acknowledge the help of Dr. Valentyna Koval and the staff at the Center of Excellence in Surgical Education and Innovations, as well as Drs. Matt Bernard, Raheem B. Kherani, John Staples, Hin Hin Ko, and Dan Renouf for their help on reviewing of videotapes and simulation teaching.

Appendix

Modified Global Rating Scale

Appendix

Procedural Checklist

References

   Comments ()