Original Research

Does personalized vascular access training on a nonhuman tissue model allow for learning and retention of central line placement skills? Phase II of the procedural patient safety initiative (PPSI‐II)

Copyright © 2009 Society of Hospital Medicine

The Accreditation Counsel for Graduate Medical Education (ACGME) states in its Program Requirements for Residency Education in Internal Medicine that all residents must develop technical proficiency in several procedures, including central venous line placement.1 Developing competency in common procedural skills has long been a part of medical training. The philosophy of see‐1, do‐1, teach‐1 is still the most common means by which most residents seek to obtain this proficiency, even though serious concerns have been raised about this approach.2 A typical first experience in central line placement usually involves an eager (or terrified) trainee making several clumsy attempts on an actual patient, under the hurried guidance of a senior resident who themselves received an unknown degree of training. In this scenario, rarely does standardized instruction, formal evaluation, or structured follow‐up occur.

A revitalized emphasis is now being placed on patient safety in healthcare, including an industry‐wide commitment to minimizing procedural complications. The most common complications associated to central line placement include vascular damage and catheter‐related bloodstream infections. A number of creative approaches are being developed to improve the quality of instruction on proper procedural techniques, all varying considerably in sophistication, scope, and rigor. Examples include the use of computer‐assisted methods for training ultrasound‐guided needle insertion techniques and ureteroscopy training, hands‐on training with synthetic models for thoracentesis training, video training, and uterine aspiration using papayas.311 Implicit in this trend is recognition that we, as educators, healthcare providers, and patient advocates, must design more cost effective and efficient ways to teach medical and surgical procedural techniques to clinicians.

Our approach was previously described in phase I of the Procedural Patient Safety Initiative (PPSI).12 In PPSI‐I, we introduced a nonhuman tissue model (NHTM; Figure 1) as the basis for teaching physicians a more rigorous curriculum of essential central line placement skills. By way of brief review, the NHTMs were constructed by tunneling 0.2‐mm‐thick rubber tubing (vessels) lengthwise through raw, whole chickens purchased at the grocery store. The vessels were filled with colored water to simulate blood. The NHTM has several unique features, including: (1) realistic‐appearing vessels when viewed under ultrasound, which mimic the appearance of human internal jugular veins and carotid arteries (Figure 2); (2) tissue turgor and vessel composition that produce realistic pops and flashes upon puncture and allow for multiple cannulations; (3) the ability to perform a complete central line placement (including wire advancement, dilation, line insertion, suturing, and sterile dressing placement); (4) cost effectiveness relative to other commercially‐available products (each NHTM costs $120 and can withstand multiple cannulations over 2 days).1317 During the training sessions of Phase I, participants were oriented to the ultrasound machines, shown the contents of our central line kit, and taught the principles of wide sterile barriers (WSBs), sharps safety, and vascular access under real‐time ultrasound guidance. A self‐completed survey tool was filled out by the participants before and after the session that contained questions about their precourse baseline procedural experience, and their subjective comfort level with specific skills after the course. The results of our intervention, as measured by the responses to the survey, were significantly positive. We recognized the limitations of these results based on using subjective criteria to measure efficacy, a lack of follow‐up on participants' skill retention, and with no ultimate evaluation of procedural competency evaluations on actual patients (compared to an untrained control group).

Figure 1

Nonhuman tissue model (NHTM).

Figure 2

Ultrasound images of the NHTM vessels (left) vs. internal jugular vein and carotid artery on a human volunteer (right). Abbreviation: NHTM, nonhuman tissue model.

Our ultimate goal is to validate a curriculum that will give trainees the necessary education and skills that enables them to make a smooth, competent, and complication‐free transition to live patient procedures. Phase II of PPSI is our next step toward this goal. In this study, we sought to measure the impact of intensive, 1:1 central line placement training with a proceduralist, objectively validate the efficacy of the NHTM and our training curriculum using a standardized 6‐point scoring scale and skilled evaluators, and to evaluate the degree of skill retention over time (decay). Our hypothesis was that the depth of skills' imprinting from a single, standardized training session would result in a significant improvement in measured procedure skills immediately after the trainee is taught the skills, and that the retention of these skills would be demonstrable when participants were reevaluated at a future date.

Methods

PPSI‐II was an observational, prospective study conducted by The Procedure Center at Cedars‐Sinai Medical Center, a 900+‐bed, community‐based teaching hospital. The Procedure Center is staffed by dedicated Proceduralists who perform a number of common medical procedures on a daily basis and are facile with both real‐time ultrasound guidance and proper procedural techniques.18, 19 Our target population was the incoming Internal Medicine residents. Subjects were recruited by email prior to orientation week and were offered the option of participating in our study. Our only exclusion criterion was the prior observation or placement of 10 or more central lines. The study was approved by our hospital's Institutional Review Board prior to initiating recruitment. Those who chose not to participate underwent the standard orientation training required by our institution, which included a brief overview lecture on the topic of central lines and ultrasound‐guidance, a group viewing of the New England Journal of Medicine (NEJM) video on central line placement,20 and small‐group (4 participants/group) hands‐on practice sessions lasting 45 minutes with NHTMs and a trained Proceduralist.

All of the evaluations for Phase II were done using the Central Line Placement Skill Assessment Tool depicted in Figure 3. This tool, which was developed by Cedars‐Sinai Proceduralists solely for the purposes of this study, is a comprehensive step‐by‐step checklist delineating the specific steps necessary to place a sterile, ultrasound‐guided central venous catheter. It was closely derived from a central line insertion checklist that was created by the Procedure Center 3 years ago to help guide novice clinicians through the procedure, and has since been widely used throughout the institution during the placement of hundreds of central lines. The scoring system, also devised by Cedars‐Sinai Proceduralists, was based on over 15 years of experience supervising and teaching hundreds of residents on proper central line insertion techniques. It consists of clear definitions for each score that were agreed upon via consensus amongst study coordinators. Prior to any evaluations being conducted, we put our 2 evaluators (both senior medicine residents) through identical and simultaneous scoring training with the Proceduralist trainer to standardize procedural knowledge and scoring methodology.

Figure 3

Central line placement skills assessment tool (essential elements indicated in bold).

A total of 20 incoming interns (trainees) out of a possible 54 invitations (37%) volunteered to participate. Each trainee was randomly assigned a number from 1 to 20. The study began for each trainee with a brief, 5‐question survey to determine their prior procedural experience (Table 1). Next, each trainee watched the NEJM online training video on central line placement.20 They were then brought into a training room that contained an NHTM sitting on a Mayo stand, an ultrasound machine, and all the materials required to place a central line insertion under ultrasound‐guidance. The trainee's baseline central line insertion skills were evaluated on 22 unique procedure steps, with each score being given by 1 of the 2 evaluators (initial evaluation). The trainee did not receive any guidance or suggestions during this initial evaluation unless the trainee reached an impasse. In these cases, the evaluator completed that single step on the trainee's behalf and then allowed the session to resume. The identity of the evaluator was indicated on each evaluation form, and after each of the evaluations the completed assessment tool was given to our blinded assistant for data recording.

Trainee Characteristics
Number of trainees20
How many prior central lines have you inserted independently? (exclusion criteria: >10)20 answered 0
How many prior central line insertions have you assisted with? (exclusion criteria: >10)13 answered 0; 7 had assisted between 1 and 4 lines
How many prior central line insertions have you observed? (exclusion criteria: >10)3 answered 0; 17 had observed between 1 and 5 lines
Have you had any prior exposure to the use of ultrasound for central line insertion?13 no; 7 yes
Have you had any prior exposure to the use of wide sterile barriers for procedures?11 no; 9 yes

Each trainee was then given a personalized, hands‐on training session by a proceduralist, using the checklist as a guide to take them through all the steps of a central line insertion. The trainee was allowed to observe and practice each skill for an unlimited period of time with the proceduralist present, until he or she demonstrated competency and felt confident enough with their independent skills (in both trainee's and proceduralist's judgment) to move forward. The entire session ended only when all steps had been taught and practiced to the proceduralist's satisfaction, the trainee felt comfortable independently performing each step (and in proper sequence), and all questions had been answered. At no time was there an imposition of time constraints or external pressure from study coordinators.

Immediately following this training session the proceduralist and trainee left the room, the procedure room was reset by an evaluator (taking approximately 5‐10 minutes), and then the trainee submitted themselves to an immediate posttraining evaluation (immediate evaluation). As with the initial assessment, the evaluator did not interfere or make any comments or suggestions during the evaluation periods, unless the trainee reached an impasse at any step. In that case, the trainee would receive a 0 for that step, the evaluator would assist them to complete that step only, and then the session would continue. No time limits were imposed.

The final part of the study required each trainee to return for follow‐up assessment (delayed evaluation), a process that was identical to the immediate posttraining evaluation. This delayed evaluation was intended to occur between 3 to 4 weeks after the immediate posttraining session, based on trainees' schedules and availability. No refresher or practice time was permitted prior to the delayed evaluation: upon arrival, trainees wrote down on a separate piece of paper (not seen by the evaluator) the number of interim line experiences they had experienced, then they were brought directly into a fully‐prepared room, and instructed to begin. The evaluator was also blind to the trainee's scores from the 2 previous evaluation sessions.

The primary endpoints were the degree of changes in overall average scores (from the 22 steps on the assessment tool) from the initial to the immediate evaluations and from the immediate to delayed evaluations. The secondary endpoints were also based on changes in average scores from the initial to immediate and immediate to delayed evaluations, and looked at 5 essential elements (steps in the assessment tool that we deemed critical to the safe and successful placement of a central line). These essential elements included (1) hand washing; (2) creation of a WSB; (3) ultrasound‐guided vessel cannulation; (4) proper catheter placement; and (5) sharps safety. Of note, the creation of a WSB element consisted of 4 steps, each of which was analyzed separately. The average scores are reported as means standard deviations (SDs).

To determine the type of analysis that would be performed, we started by assessing the changes using paired t tests. The Kolmogorov‐Smirnov and Anderson‐Darling normality tests revealed no evidence of violations of the normality assumption, confirming that using paired t tests was valid.

To address potential contamination from residents' real experiences on the rate of their knowledge decay between the immediate evaluation and delayed follow‐up, each participant completed a brief survey before their delayed evaluation asking about interim experiences. All calculations were performed including and excluding from participants' scores with affirmative answers to control for this contamination. Last, a post‐hoc analysis was performed on participants' scores using a scatterplot and statistical analyses to control for the varying time‐to‐follow‐up.

Results

All 20 individuals completed the study, for a total of 60 evaluations (20 each of initial, immediate, and delayed). The actual training time (not including the viewing of the video) ranged between 45 to 120 minutes, depending on the trainee. Our primary endpoints are depicted in Table 2. The mean overall score on the initial evaluation was 1.0 0.8. The mean overall score for the immediate posttraining evaluation was 4.4 0.3. This improvement of 3.4 points was significant (P < 0.001; 95% CI, 3.0‐3.7). The delayed evaluations took place an average of 22 days after the training session (range, 5‐101 days), and produced an overall mean score of 4.2 0.3. This decay of 0.2 was not significant (P = 0.14; 95% CI, 0.31 to 0.05). With regard to the amount of skills decay, additional calculations were performed from the scatterplot that depicted scores and the variability in time‐to‐follow‐up. We found that even after controlling this variable, the amount of decay for the overall score remained insignificant.

Overall Results
  • Abbreviations: CI, confidence interval; SD, standard deviation.

Mean (SD) score of initial (baseline) evaluation1.0 (0.80)
Mean (SD) score of immediate posttraining (baseline) evaluation4.4 (0.30)
Average change between initial and immediate posttraining scores+3.4P < 0.001; CI, 3.0‐3.7
Mean (SD) score of delayed posttraining evaluation4.2 (0.32)
Change between immediate and delayed posttraining scores0.2P = 0.144; CI, 0.31‐0.05

The results of the secondary endpoint calculations (essential elements) are depicted in Table 3. Ultrasound‐guided vessel cannulations improved from an initial average score of 0.9 1.0 to an immediate average score of 4.2 0.5 (P < 0.001; 95% CI, 3.0‐3.7); the delayed score of 4.3 0.6 was statistically unchanged from immediate (P = 0.77; 95% CI, 0.4 to 0.3). Catheter placement skills improved from 1.1 1.1 to 4.2 0.5 (P < 0.001; 95% CI, 2.6‐3.7), and the delayed score of 4.3 0.7 was unchanged from immediate (P < 0.58; 95% CI, 0.5 to 0.3). Sharps safety also improved significantly from initial (2.0 2.3) to immediate (4.9 0.5) (P < 0.0001; 95% CI, 1.9‐3.9), and the delayed scores dropped insignificantly to 4.6 0.8 (P = 0.08; 95% CI, 0.0‐0.6). Hand washing improved significantly from an initial score of 0.9 1.9 to an immediate score of 3.5 2.2 (P < 0.001; 95% CI, 1.4‐3.7), and decayed insignificantly on the delayed evaluation to 3.0 2.3 (P = 0.53; 95% CI, 0.9 to 1.7). WSB skills consisted of 4 individual steps, all of which all improved significantly from initial to immediate scores, and had insignificant decays on the delayed evaluations (see Table 3 WSB for details).

Essential Elements Results
Initial EvaluationImmediate Follow‐UpP Value (Initial to Immediate)Delayed Follow‐UpP Value (Immediate to Delayed)
  • Abbreviations: CI, confidence interval; WSB, wide sterile barrier.

Ultrasound‐guided insertion of needle into vein (step 15)0.9 (1.0)4.2 (0.5)P < 0.001; CI, 3.0‐3.74.3 ( 0.6)P = 0.77; CI, 0.4 to 0.3
Catheter placement (step 18)1.1 (1.1)4.2 (0.5)P < 0.0001; CI, 2.6‐3.74.3 ( 0.7)P = 0.58; CI, 0.5 to 0.3
Sharps safety (step 20)2.0 (2.3)4.9 (0.5)P < 0.0001; CI, 1.9‐3.94.6 ( 0.8)P = 0.08; CI = 0 to 0.6
Hand washing (step 2)0.9 (1.9)3.5 (2.2)P < 0.001; CI, 1.4‐3.73.0 ( 2.3)P = 0.53; CI, 0.9 to 1.7
WSBs
MD prep (step 3)1.8 (1.5)4.3 (0.7)P < 0.0001; CI, 1.7‐3.34.2 ( 0.6)P = 0.30; CI, 0.2 to 0.6
Site sterilization (step 7)1.1 (1.1)4.3 (0.9)P < 0.0001; CI, 2.7‐3.74.5 ( 0.5)P = 0.45; CI, 0.6 to 0.3
WSB creation (step 8)0.6 ( 0.6)4.1 ( 0.9)P < 0.0001; CI, 3.0‐4.04.4 ( 0.6)P = 0.26; CI, 0.7 to 0.2
Ultrasound probe cover application (step 9)0.4 ( 0.9)4.1 ( 0.8)P < 0.0001; CI, 3.2‐4.14.4 ( 0.8)P = 0.23; CI, 0.8 to 0.2

We performed validation exercises to determine the degree of interrater agreement. Of the 60 total evaluations that were eventually performed, 11 evaluations had been performed simultaneously and independently by evaluators A and B. An analysis of the scores assigned by each evaluator to these 11 trainees revealed a high level of interrater agreement (96%). Further, we performed independent analyses of the trainees' scores as assessed by evaluator A (22 sessions) or evaluator B (27 sessions) across the initial, immediate, and delayed sessions, and we detected no statistical differences in the changes in scores (which mirrored the overall results above).

With regards to real‐life contamination between immediate scores and delayed scores, we identified 3 trainees who had placed central lines on actual patients during the interim period (2 trainees placed 1 line each, and 1 trainee placed 2 lines). We repeated all of the calculations without these participants' delayed scores and determined that the removal of their scores did not change the statistical significance of any of the study results. With regard to knowledge decay, the scatterplot comparing delayed scores to varying time‐to‐follow up revealed no correlation.

Discussion

Our study was designed to determine whether novice trainees could learn and retain proper central line placement skills on the NHTM by receiving personalized training in a relaxed, 1‐on‐1 learning environment. Success was measured by trained evaluators using a detailed evaluation tool with a 6‐point scoring scale. The results of our primary endpoints (changes in overall average scores across the 3 evaluation periods) confirmed that this type of training could quickly improve novice practitioners' skill levels from very low (initial evaluation) to significantly higher (immediate posttraining). The dropoff (decay) in skill levels was found to not be statistically significant over a period of several weeks, although we recognize that further study should be performed to establish the degree of skill decay over a longer period of time.

Because some steps in a central line insertion are more critical to the procedure's success than others (ie, a skin nick with a scalpel is less critical than vessel cannulation under ultrasound‐guidance), we analyzed 5 essential elements individually as secondary endpoints. This secondary analysis was designed to unmask any critical skill deficiencies that might otherwise have been lost in the overall analysis. For each individual essential elements step, this subanalysis similarly revealed a significant improvement from initial to immediate posttraining, and an insignificant score decay on the delayed evaluation.

We recognize a number of limitations to this study. First, the n is relatively small. A larger sample size would have allowed for greater statistical power. In addition, the scoring system used for this study was created by our Procedure Center staff and had never been truly validated elsewhere. The scoring system was transparent and logical, but we recognize that any attempt to use an interval scoring system to quantify procedural skills will be inherently imperfect; the difference between 1 and 2 is not necessarily the same as a difference between 4 and 5. Great efforts were taken to mitigate the impact of this limitation: explicit definitions were established for each score, and we put our evaluators through a rigorous scoring orientation at the outset to standardize their interpretation and use of the scoring system and assessment tool.

The variability in the amount of training time spent in each session could be considered to be a confounder. Our prior experiences training interns in small groups, however, suggested that individuals learn these skills at different paces and in different ways, and so we consider our customized approach to be an essential part of this training experience. We do recognize the practical limitations inherent in rolling out such an open‐ended approach, and program directors may face time and/or resource limitations if attempting to replicate this training strategy.

We were also aware of potential interrater variability between the evaluators. Our approach to addressing this was multifactorial: we went to great lengths to standardize evaluators' understanding of the intended scoring methodology prior to the initiation of the study. We also assessed the degree of interrater reliability once all data was collected. This analysis reinforced that both evaluators were scoring trainees in a virtually identical fashion. We attribute this consistency to the quality of the scoring system, the effectiveness of the prestudy evaluator orientation with a proceduralist, and the high degree of teamwork between the 2 evaluators that kept them closely in sync with one another throughout the study.

Evaluator bias was also a concern. While each evaluator was blinded to the trainees' prior scores, the setup associated with the different training sessions, as well as the obvious differences in performance between the trainees' initial and immediate/delayed performances, made full blinding of the evaluators difficult. The theoretical risk of evaluator bias in this study would have led to evaluators rating trainees higher in the immediate and delayed performances in order to demonstrate more dramatic results. We believe that, since the evaluators themselves did not perform the actual training, and since they did not know the previous scores for the trainee, they were less inclined to skew the scores. Video recording each performance and submitting this recording to a fully‐blinded, third‐party evaluator would have more rigorously ensured blinding than we were able to accomplish. This approach could be considered in future studies of this type.

An addition limitation involved the time‐to‐follow‐up. While a longer time interval between the immediate and delayed evaluations may have better evaluated the impact of the training and potential decay, we sought to balance this with the growing risk of contamination from real central line placement experiences as more time passed. With this issue in mind, the removal of the delayed scores from the 3 trainees who had placed central lines on actual patients in between the immediate and delayed evaluations (2 trainees placed 1 line each, and 1 trainee placed 2 lines) did not change the statistical significance of any of the study results.

One practical concern has to do with the reproducibility of this approach at other institutions. Each trainee received up to 2 hours of individualized attention, and each session consumed fresh supplies and required a proceduralist's and an evaluator's time. This represents a significant commitment of materials and manpower. A careful cost/benefit analysis is therefore warranted before implementing this kind of rigorous training program. As mentioned, the cost of the NHTM is approximately $120 and can withstand several cannulations over a 2‐day period; the sterile supplies and central line add up to approximately another $75/evaluation. Depending on the number of interns and residents at a given institution, these costs could prove prohibitive to cash‐poor residency training programs. In the larger picture, however, catheter‐related bloodstream infections have been estimated to result in a mortality rate of 4% to 20%, and a single catheter‐related bloodstream infection can cost up to $45,000.2124 In addition, new Medicare reimbursement policies are now beginning to limit hospital reimbursement for these types of iatrogenic events; hence, narrowing the margin of error and putting even greater financial pressures on hospitals.25 It is our belief, therefore, that an up‐front investment in NHTMs (or an alternative simulator), basic supplies, and the necessary trainer time will prove to be cost‐effective and enlightened investments from forward‐thinking leadership.

Last, we are also aware that our study did not look at whether our trainees' improved performance on the NHTM actually translated into better patient outcomes. Since patient safety is our ultimate goal, and this phase of PPSI limited all of our training and evaluations to the NHTMs, future studies must ultimately evaluate how well these learned skills translate into procedure performance on actual patients. This controlled study (possibly with a see‐1 do‐1 teach‐1 control group) will be logistically challenging, but will be the most definitive manner with which to demonstrate the true value of personalized training sessions using the NHTM (or another nonhuman simulator).

PPSI‐II demonstrated that using the NHTM as the basis for training novice practitioners in a personalized, 1‐on‐1 training session led to significant improvements in measured procedural skills. Further, these skills were retained over time. This positive study contributes to the growing body of literature pointing towards the role of intensive 1‐on‐1 training with simulators to advance procedural education for clinicians. Ultimately, we aim to demonstrate that providing trainees this type of training prior to having them perform procedures on actual patients will translate into superior patient care, greater success rates, fewer minor and major complications, and lower overall patient care costs. Rather than clinging to the classic but never‐validated see‐1, do‐1, teach‐1 approach, we believe that procedural training must adapt new curricula and technologies that will help us achieve the goals of maximizing the safety and quality of care for our patients.

Acknowledgements

The authors recognize and appreciate the entire staff of The Procedure Center at Cedars‐Sinai Medical Center for their support of this research project. The authors give special thanks to Obed Martinez for his tireless assistance with the scheduling and coordination of training activities, and to Jim Mirocha for statistical analysis and editorial contributions.

References

   Comments ()