In its most diverse forms, simulation refers to the re-creation of something real by imitation. In medical education, simulation can refer to a number of modalities used to re-create some component of the clinical encounter for the purposes of training or assessment, including part-task trainers, virtual reality simulators, standardized patients, virtual patients, and computerized full-body mannequins.1-5 Simulation in the form of standardized patients and early full-body mannequin simulators has been described in the healthcare literature since the late 1960s.6-8 However, its broader acceptance into medical education can be dated closer to the turn of the 21st century with the formation of the Association of Standardized Patients in 1991,9 the first International Meeting on Simulation in Healthcare in 1995,10 the establishment of the Society for Simulation in Healthcare in 2004,10 and the publication of Simulation in Healthcare beginning in 2006.11

The growing acceptance of simulation in healthcare training has been attributed to the decreased availability and acceptance of practising skills on patients, the growth in technology, which has fuelled the development of increasingly sophisticated simulation modalities, as well as the development of a culture of safety, which has resulted in decreased tolerance for errors.3,5,12-15 Together, these forces have led to greater interest and expertise in the development of simulation-based training modalities to re-create teaching and assessment opportunities where practice or assessment on real patients is either not feasible or undesirable.

In anesthesia, the first systematic use of simulation for training consisted of full-body mannequin simulators placed in simulated clinical environments to train students, residents, and staff physicians in managing operative and perioperative acute events. The development of the full-body mannequins that are currently used for training and assessment date back to the mid-1980s with the CAE-Link simulator developed at Stanford University by Gaba et al. 16 and the Gainesville Anesthesia Simulator developed by Good and Gravenstein at the University of Florida in Gainesville.5,14,17 Although the two groups had different approaches for simulation-based training, they both targeted the recognition of anesthetic critical events and their management.5,18 Gaba’s work on team training, Simulation-Based Training in Anesthesia Crisis Resource Management (ACRM), has had a marked impact on simulation-based training in anesthesia. Patterned after the Crew Resource Management system from the field of aviation, ACRM is currently the predominant model for training anesthesiologists and trainees to manage operative and perioperative crises.19

The ACRM curriculum developed by Gaba et al. consisted primarily of highly realistic simulation scenarios in which participants managed acute events. These events were followed by detailed video-assisted debriefing sessions during which the medical and technical elements and the principles of crisis management (leadership, teamwork, workload distribution, resource utilization, re-evaluation, and communication) were covered. By the mid-1990s, ACRM training had spread to other anesthesia simulation programs across the United States and Canada.16,20 Crisis management training and related approaches, generally described as team or non-technical skills training for high acuity events, spread rapidly across North America and Europe and now represent the bulk of anesthesia simulation-based training.8,19,21,22

More recently, the use of simulation in anesthesia has broadened to include the acquisition of technical skills (e.g., fibreoptic oral intubations and cricothyroidotomies),23-26 the study of performance-shaping factors and performance gaps,27-34 the evaluation of new equipment,35 and modelling patient flow in clinical settings.36,37

Simulation for training: state of the science

Changes in learner perceptions

Early research on the impact of simulation-based training targeted the perceptions of participants. The results of these studies showed that participants were generally very positive about their training, and they perceived their training as contributing to safe practice.38-44 However, a few studies also showed that this form of education was intimidating and stressful for participants,41 and only a minority (~30%) believed it had influenced their clinical practice.45,46 Interestingly, there is also a growing body of evidence showing that self-reports of participants do not predict their actual levels of performance.47,48 While simulation is generally well received by trainees, these results together indicate that this acceptance is not universal. Therefore, educators seeking to use simulation should carefully construct sessions that create safe learning experiences for the trainee. Also, given the growing body of work showing that perceptions of learning do not always predict actual learning, research aimed at assessing the impact of learning modalities cannot rely simply on the learners’ perceptions of learning. While learner perceptions can be useful to determine how a simulation session was received and experienced, they are not sufficient to determine whether the session actually enhanced, impaired, or had no effect on learning.

Evidence for effectiveness of simulation-based learning

Beyond showing that crisis management training is generally well received and perceived as beneficial for training, it is essential to demonstrate that this resource intensive training can also improve learning and clinical performance compared with more traditional forms of instruction. A number of studies on the effectiveness of simulation-based training have used performance during simulated critical events as outcome measures, in large part due to the challenges of measuring performance in clinical settings. Chopra et al.49 showed that anesthesiologists trained to manage malignant hyperthermia with a full-body mannequin simulator responded more quickly, deviated less from accepted guidelines, and performed better in handling a subsequent simulated malignant hyperthermia than residents who did not receive the training. Yee et al.50 showed that a single simulation session constructed around an ACRM-type course improved non-technical skills (i.e., decision-making, situation awareness, and interpersonal skills) of residents during a simulated anesthesia crisis. In a study with practicing anesthesiologists, Morgan et al.51 showed that mannequin-based simulation with debriefing led to improvements in some aspects of the clinical management of simulated critical events up to nine months after the training. In a comparison with baseline, Kuduvalli et al.52 demonstrated that anesthesiologists with simulation-based training subsequently demonstrated a more structured approach and reduced equipment misuse during simulated difficult airway scenarios. Schwid et al.53 demonstrated that residents who had received screen-based anesthesia training subsequently managed mannequin-based anesthetic emergencies better than residents who studied a handout covering the same content.

Although studies have shown enhanced learning following simulation-based sessions, a number of studies have failed to show improvements. Olympio et al.54 did not show improvement in anesthesia residents’ management of esophageal intubation following simulation-based training. Borges et al. 55 did not observe significant changes in practicing anesthesiologists’ airway management of a “cannot intubate, cannot ventilate” simulated scenario following simulation training. Wenk et al. 56 compared simulation-based learning with problem-based learning on anesthesia students’ ability to perform a rapid sequence intubation on a full-body mannequin. Following the training session, students in both groups performed equally well on the rapid sequence intubation task. However, the students who received the simulation-based training were significantly more confident regarding their knowledge of rapid sequence intubations.

A troublesome finding from the research is that participation in a simulation-based session can increase trainees’ confidence and perceived abilities without necessarily enhancing their true abilities. This overinflated sense of confidence can be counterproductive if it leads trainees to stop practising because they mistakenly believe they have reached an acceptable level of competency. It can be dangerous if this overconfidence leads trainees to take on clinical challenges for which they, in fact, do not have the required skills. This finding raises the issue regarding the degree of responsibility that educators must bear in terms of providing trainees accurate information as to their levels of competency following simulation sessions.

Transfer of learning to the clinical setting

In addition to using simulated scenarios to study the outcomes of simulation-based training on simulated tasks, researchers have recently studied the transfer of learning to the clinical setting. There is accumulating evidence that well-designed simulation-based training can translate to improved performance in the clinical setting for both technical tasks23,57,58 and management of high-acuity events.59,60 In a prospective single-blinded randomized controlled trial on weaning from cardiopulmonary bypass, Bruppacher et al.61 observed that simulation-based training led to improved performance in a real clinical setting compared with interactive seminars. In a study of central venous line insertions in intensive care units, Barsuk et al.62,63 observed that mastery-level simulation training led to higher success rates as well as reduced rates of infections with real patients. Wayne et al. 59 observed that simulation-trained residents adhered more closely to Advanced Cardiac Life Support (ACLS) guidelines during actual cardiac events than their traditionally trained counterparts.

Key elements in simulation-based learning

While learning can be enhanced with simulation sessions, the research on simulation-based learning shows that there are cases in which this does not occur. As such, it is not possible to state that simulation, as a broad approach, is effective or ineffective for learning. Rather, simulation sessions can be conducted in a number of different ways; some simulation sessions will be more effective than other methods of learning, and other simulation sessions will not be more effective. Those elements that lead to enhanced learning are not necessarily inherent to simulation itself. Therefore, it is important to understand the elements in simulation-based sessions that facilitate learning, as well as how to optimize learning using this form of practice and instruction. There has been a growing body of research aimed at a better understanding of the mechanisms that optimize simulation-based training.

Debriefing

Debriefing has been shown to be a critical element in the observation of improved performance following simulation-based training.64,65 However, the format used for the instruction or debriefing following the simulation scenario does not appear to have a significant effect on learning. Anesthesia students showed similar improvements in post-scenario debriefing sessions whether with a simulator or with a video session facilitated by faculty.66 In simulated ACLS resuscitation scenarios, Welke et al. 67 showed that multimedia instruction and faculty-led video-assisted debriefing sessions led to similar improvements. Boet et al. 68 reported similar results when comparing self-debriefing with instructor debriefing. These early studies suggest that the format in which debriefing is delivered may have a minimal impact on subsequent learning. However, more research is needed to understand more clearly the contributions of format and delivery mechanisms on the effectiveness of debriefing.

In addition to studying the format in which debriefing is delivered, researchers have explored whether the content and structure of debriefing has an impact on learning. Park et al. 69 have shown that the improvements observed following simulation-based training appear to be content specific. In their study with anesthesia residents, they demonstrated that event-specific simulation training resulted in subsequent improved performance compared with simulation-based training in an alternate event. Residents who had received training on hypoxemia subsequently performed better during simulated hypoxemic events but not during scenarios related to hypotension. In contrast, those residents who received training on hypotension subsequently performed better during hypotension-related scenarios but not on scenarios related to hypoxemia. Looking at the structure of debriefing, Johnson et al.70 compared different teaching approaches with simulation-based training. Over a 12-month period, anesthesia residents were assigned to either a control group that received standard didactic and simulation-based training or to an experimental group that received similar training but with an emphasis on part-task training (dividing tasks into components) and variable priority training (focus on optimal distribution of attention when performing multiple tasks simultaneously). The group receiving the part-task and variable priority training showed more improvements in performance when managing adverse airway and respiratory events.

While debriefing (or feedback) serves an important role in the effectiveness of simulation-based training, these results together suggest that it is a complex process. According to the research to date, the format in which debriefing is delivered (simulation-based, instructor-led, multimedia, or self-led instruction) does not seem to impact learning. However, the work of researchers, such as Johnson et al. 70 and Park et al.,69 suggests that the content and the structure of the debriefing or feedback may play an important role in learning.

Fidelity

Another important area of inquiry regarding the key elements of simulation is fidelity. In light of the high monetary and personnel resources that are invested into full-body mannequin simulations, some researchers have investigated whether lower-fidelity lower-cost simulations could be as effective as the higher-cost higher fidelity simulations. Nyssen et al.71 compared the effectiveness of a computer screen-based simulator with a mannequin-based simulator for training novice and experienced anesthesia residents in the management of simulated anaphylactic shock. They found that the two types of simulators did not result in significant differences in learning. High-fidelity and low-fidelity simulators can have equally positive impacts on learning for novice students.24,72-74 Hence, the purchase of high-cost high-fidelity simulators must be considered thoughtfully, especially for use early in the learning curve. Moreover, rather than comparing low-fidelity with high-fidelity simulators, new studies have suggested that a better approach may be to structure the simulation experience as a progressive training program.75 In addition, recent work regarding the concepts of fidelity and realism76-78 suggests that these are complex concepts that extend beyond the physical realism of the mannequin, and more work is needed to understand fully what we mean by fidelity and the role it plays in simulation-based training.

Simulation for team training

Although the bulk of the research has focused on teaching non-technical or crisis management skills to individuals, there is a small but growing body of work targeted towards team training.22 Teamwork dysfunction has been associated with decreased quality of care, such as increased adverse events and poor patient outcomes.79,80 This has led to a growing interest in collective competency, moving beyond teaching individuals alone towards also teaching team coordination and communication skills to interprofessional teams. Most of the literature to date has focused on the development and deployment of such training69,81 based on approaches that have been successful in other high-risk domains such as aviation and the military.82 Although self-reports from participants indicate that they credit the training for increasing their teamwork skills,83-85 thus far, there is little research in anesthesia investigating the effectiveness of this form of training on the behaviours and clinical practice of teams.

Simulation for assessment: state of the science

In addition to interest and research in the use of simulation for training, there is continued interest in the use of simulation modalities for assessment and certification.86 For more than ten years, simulation-based scenarios have been incorporated into the Israel Board Examination in Anesthesia.87,88 More recently, the American Board of Anesthesiology has incorporated mandatory simulation-based activities in the ten-year maintenance of certification cycle,89,90 and the Royal College of Physicians and Surgeons of Canada has introduced a simulation-assisted oral station in the 2010 anesthesia examination.91 Although the use of simulation for formative assessments is widely accepted, the use of simulation for summative assessments remains somewhat contentious, and developments are slower than those in the use of simulation for training purposes.

One reason for this interest in simulation for assessment and certification is that they are viewed as being authentic assessments of the cognitive and behavioural components of competency.92 While workplace-based assessments are highly desirable for the assessment of competency, concerns have been raised about the psychometric properties of this form of assessment.93,94 In contrast, while methods such as written examinations and oral examinations have solid psychometric properties, they are critiqued for lacking ecological validity, i.e., for not closely re-creating the practice conditions under which we want to assess competency.95,96 Simulation-based assessments have been proposed as complementary means to assess performance and behaviour in an authentic and reliable context.65,86

Assessing individual performance

The interest in the use of simulation for the assessment of performance has fostered significant research. Several review papers have presented overviews of the literature and have included recommendations regarding the appropriateness of using simulation in high-stakes examinations.97,98

Recently, Boulet and Murray99 wrote a broad narrative review on the use of simulation for assessment and the implications of such assessments on anesthesiology. They also included a thoughtful discussion of important considerations for educators looking to develop valid and reliable simulation-based assessments.

Boulet and Murray99 report that, thus far, most of the work examining the reliability of simulation-based assessments had focused on inter-rater agreement as well as the consistency of examinee scores across multiple stations or scenarios. Different tasks and assessment contexts were associated with varying levels of inter-rater agreement, with ratings of teams and non-technical skills often having lower inter-rater agreement than assessments of individual technical skills or clinical management. As for the consistency of scores across cases or scenarios, these were generally low due to the content specificity of knowledge and skills. These findings are consistent with the vast literature on assessments of performance, which show that clinical competence is very content-specific.95,100 Strong performance in one aspect of competency does not imply that a candidate will perform equally well in other aspects of competency.86 As such, multiple stations (8-15) were recommended in order to ensure that the scores obtained are reliably precise enough for an examiner to make decisions regarding an examinee’s level of competence.99,101-104

Boulet and Murray99 also discussed research into the validity of simulation-based assessments. To date, most of the work towards making inferences regarding the validity of simulation-based examinations has been directed towards content validity, i.e., seeking to ensure that simulation scenarios are modelled and scripted based on actual practice characteristics. However, for simulation-based assessments to become fully integrated into summative evaluations of performance, significantly more work is needed in terms of establishing standards and demonstrating that simulation-based performance is predictive of future performance in clinical settings.

Assessing team performance

More recently, there has been growing interest in assessing team performance. This development is in response to studies and reports showing that a high percentage of errors in the operating room could be attributed to gaps in team coordination and communication. One initial challenge in team assessment was the absence of valid and reliable tools to evaluate group performance. There has been significant work in recent years aimed at the development and evaluation of such tools.105-110 To date, there have been contradictory results regarding the validity of these tools, suggesting that more refinements are needed before the field is ready for high-stake assessments of teams.109 The implications relating to summative assessments of team performance present a second challenge. If a team were to fail on a simulated summative assessment, what implications would there be for the individual team members, for the team itself, and for the institution? Before the field of anesthesia implements summative assessments of teams, it will need to grapple with such questions.

Looking forward: advancing the field and optimizing the use of simulation

Looking at the current state of simulation, the question is no longer whether simulation will have a lasting presence in the education of the health professions. Licensing bodies in the United States and Canada now mandate the use of simulation in certain specialties,3 and accreditation standards for simulation programs are being developed and rolled out internationally.111-113 However, this does not mean that the work is done. The following section deals with aspects of simulation that need further refinement or attention if we are to use simulation modalities optimally to enhance current health professional training and practice.

Research

Anesthesia is one of the specialties in which a significant amount of research has been conducted on the use of simulation for training and assessment. Much of the research has focused on the outcomes of simulation-based training by asking the question, Does it work? The focus of research has been moving gradually from participant reactions, to behaviours and skills, and now towards transfer to the clinical setting and patient outcomes. In addition to this outcomes-based focus, the field also needs research targeted towards gaining a better understanding of simulation by asking the questions: How and why does it work? For whom does it work? and In what context does it work? This type of research is conducted best when grounded in theoretical foundations. For example, although there is growing evidence about the importance of feedback or debriefing, important unanswered questions remain about the most effective method of debriefing. To understand how to provide optimal debriefing and feedback, research is needed that looks at the structure and content of debriefing and is based on decades of inquiry from cognitive and motor learning sciences.22,114,115

Research into the effectiveness of simulation has also been directed towards the individual. More recently, however, there has been greater interest in team training and assessment of team performance. In accordance with this greater interest, the field needs additional research that not only explores the effectiveness of team training but also considers the unique challenges that emerge when attempting team training or assessment.116

Knowledge translation: uptake of evidence and best practices

Although a significant amount of research has been conducted into the use of simulation in anesthesia and other health professions, only a limited amount of the knowledge acquired through this research is transferred to clinical performance. For example, one of the central tenets attributed to simulation is that it allows for deliberate practice.14,117 Deliberate practice is defined as practice undertaken over an extended period of time to attain excellence, and it entails the ongoing efforts required to maintain it.118-120 It consists of practising a well-defined task at an appropriate level of difficulty for the individual, informative feedback, and opportunities for repetition and correction of errors. One of the main caveats of deliberate practice is that it consists of repeated practice that occurs over a long period of time. However, with the way that simulation is currently integrated into the curriculum, trainees might participate in simulation-based activities for only a few hours each year.121 As such, it is questionable whether this could qualify as deliberate practice.

There is also accumulating evidence of content-specificity of learning and performance. Strong performance in one domain or content area does not predict performance in other domains or content areas. As such, performance is likely to improve only in the specific content domain covered in the simulation activity. To have a true impact on clinical performance, simulation programs would do well to map out the tasks and content of the curricula so as to enhance current simulation-based training and provide instruction across the breadth of these content areas. In view of the research on content-specificity, more study is needed to determine what skills and knowledge are “truly” transferable across content domains and clinical situations.

The uptake of research findings into practice will necessitate a stronger focus on the development and training of simulation instructors and facilitators. Faculty development is one of the least developed aspects of simulation-based training. However, it is a crucial component given that simulation requires a different form of instruction than more traditional didactic-based methods. Faculty development based on evidence and proven theories of learning22 will be essential to ensure that we are optimizing a potentially resource intensive and expensive teaching modality.

Broaden the use of simulation

The focus of simulation education activities appears to be modality driven, and the adoption of any particular simulation modality is strongly associated with the various specialties. For example, the use of standardized patients has been adopted primarily by medicine and family medicine, while the use of task trainers for technical skills remains primarily the domain of surgery. Full-scale mannequin simulations have been the primary form of simulation adopted by anesthesia programs. As a result, the education activities in anesthesia have been mainly those that are well adapted to mannequin-based simulation. It is as though the field has found a hammer that works well and has gone forward to look for protruding nails. However, non-technical and crisis management skills, whether at the individual or team level, are not the only important skills of a practicing anesthesiologist. Competent anesthesiologists must also develop excellent technical skills (e.g., central line insertion, endotracheal intubation), patient interaction skills (e.g., preoperative assessments, ambulatory clinics), and clinical reasoning skills. These aspects of performance are not particularly well adapted to the use of full-body mannequins. Rather, technical skills can be accomplished with the use of part-task trainers, patient interaction skills can be learned quite effectively with the use of standardized patients,122,123 and clinical reasoning skills can be acquired with virtual patients.124 Significant advancements have been made in other disciplines to advance the effective use of simulation for these skills, and the integration and building upon this knowledge in anesthesia could provide a well-rounded use of simulation to enhance skills that are difficult to practise in the actual clinical setting.

Conclusion

In the past 20 years, there has been significant scholarship devoted to the study of simulation for the purposes of training and assessment. The field of anesthesia has led the way in the use of mannequin-based simulation for training and assessment of non-technical and crisis management skills. The research to date shows that simulation-based training is generally well received by participants. It can lead to improved performance in subsequent simulation events, and some transfer of learning to the clinical setting is evident. There has also been significant interest in the psychometric properties of simulation-based assessments, and there is some early evidence that well-designed performance assessments could have the reliability and validity to support high-stakes examinations. However, further work is needed towards setting standards and establishing the predictive validity to support such assessments. For simulation to reach its potential impact moving forward, further research is needed to understand how to optimize this modality of learning more effectively, how to transfer knowledge from research findings to practice, and how to broaden the simulation modalities used in anesthesia. The question is no longer whether simulation will be used for assessment and training in anesthesia, but how simulation will be implemented and used to its potential. This optimal use will depend on a clear understanding of what can and cannot be accomplished with simulation and its various modalities.

Key points

  • Anesthesia has led the way in the use of mannequin-based simulation for training and assessment of non-technical and crisis management skills.

  • Simulation-based training can lead to improved performance in subsequent simulation events, and some transfer of learning to the clinical setting is evident.

  • Further work is needed in the areas of research to understand more clearly how to optimize simulation-based learning.

  • Greater focus is needed towards knowledge transfer of research findings to practice.