Len Tanaka, Mark Ogino; Abdulla Alsalemi, Faycal Bensaali, Abbes Amira, Guillaume 1 Alinier, Yahya Mohd Osama Alhomsi, Mohammed Al Disi. 2019. A Skills Acquisition Study 2 on ECMOjo: A Screen-Based Simulator for Extracorporeal Membrane Oxygenation 3 (ECMO). Perfusion (online first) 4 5 A Skills Acquisition Study on ECMOjo: A Screen-Based 6 Simulator for Extracorporeal Membrane Oxygenation 7 (ECMO) 8 Abstract 9 Background: Extracorporeal membrane oxygenation (ECMO) relies heavily on didactic 10 teaching, emphasizing on essential cognitive skills, but overlooking core behavioral 11 skills such as leadership and communication. Therefore, simulation-based training 12 (SBT) has been adopted to instill clinical knowledge through immersive experiences. 13 Despite SBT's effectiveness, training opportunities are lessened due to high costs. 14 This is where screen-based simulators come into the scene as affordable and 15 realistic alternatives. 16 Aim: This article evaluates the educational efficacy of ECMOjo, an open-source screen-17 based ECMO simulator that aims to replace ECMO didactic instruction in an 18 interactive and cost-effective manner. 19 2 Method: A prospective cohort skills acquisition study was carried out. Forty-four 20 participants were pre-assessed, divided into two groups, where the first group 21 received traditional didactic teaching, and the second used ECMOjo. Participants 22 were then evaluated through a wet lab assessment and two questionnaires. 23 Results: The obtained results indicate that the two assessed groups show no 24 statistically significant differences in knowledge and efficacy. Hence, ECMOjo is 25 considered an alternative to didactic teaching as per the learning outcomes. 26 Conclusion: The present findings show no significant dissimilarities between ECMOjo 27 and didactic classroom-based teaching. Both methods are very comparable in terms 28 of the learner’s reported self-efficacy and complementary to mannequin-based 29 simulations. 30 Keywords 31 ECMOjo, ECMO simulation, screen-based simulation, virtual patient, computer 32 simulation, skills acquisition study. 33 34 3 Introduction 35 The rise of medical education technology is perpetually shaping new methods to 36 educate and evaluate medical professionals 1. It integrates with education policy with a 37 focus on effectiveness, efficiency, and learner-instructor interaction to forge 38 breakthroughs towards better patient care. A powerful example is simulation-based 39 training (SBT), a learning method where learners interact with people, simulators, and 40 computers to achieve learning goals in a virtual learning environment that resembles the 41 real-world 2. 42 One of the modalities of SBT is screen-based simulation, which relies on computer 43 programs that have a graphical user interface (GUI) with interactive text and images 2,3. 44 The learner has to make decisions as in real-life clinical scenarios and the simulator 45 provides corresponding evaluative feedback. The whole simulation experience can be 46 independently operated by learners without the presence of an instructor, reducing the 47 number of needed human resources especially in training centers with large numbers of 48 students 4. Furthermore, screen-based simulators have limited reoccurring costs (e.g. 49 computer and software upgrades). 50 A medical procedure relevant to both SBT and screen-based simulation is 51 extracorporeal membrane oxygenation (ECMO). It is a highly-technical 52 respiratory/circulatory support technique that uses a modified heart-lung machine to 53 provide short-term support for critically-ill patients 5,6. While on ECMO, blood is 54 continuously drained from the patient, oxygenated, and then returned via specialized 55 circuitry. Due to the inherent complexity and the multi-factorial nature of ECMO, it 56 4 demands the ECMO practitioners (nurses, perfusionists, respiratory therapists, and 57 physicians) to be attentive to every subtle change in the various parameters monitored, 58 detecting and solving potential issues to avoid further complications 5. However, ECMO 59 education practices do not catch up with its increasing international adoption 7 and 60 technological advances. Didactic lectures, multiple-choice questions, water-drills, and 61 animal laboratory testing are prevalent 8,9. Educational activities emphasize on building 62 the essential cognitive skills, however, they demand a significant human resources 63 investment, dealing with the mundane logistics of scheduling training sessions when it 64 comes to facilitating team-based immersive SBT. 65 Given those points, we introduce ECMOjo, a free VV ECMO screen-based simulator 66 for pediatric patients that mitigates the aforementioned issues of didactic training. It is 67 built on an empirical model with anatomical, physiological, and pharmacological fidelity. 68 Furthermore, it features a virtual circuit that simulates circuit data (e.g. flows, SvO2, and 69 membrane pressures), ECMO circuit components (e.g. oxygenator, pump, and gas 70 blender), and other related monitors (e.g. clinical documentation improvement (CDI) 71 monitor and vital signs screen). Figure 1 depicts the main screen of ECMOjo. The virtual 72 circuit connects to a virtual pediatric patient, modeled to react to circuit adjustments and 73 emerging issues, and hence, ECMOjo can simulate a multitude of ECMO scenarios with 74 different levels of severity. Examples include normal situations involving circuit check 75 and temperature control and emergency scenarios such as power failure, pump failure, 76 and accidental arterial decannulation. The program is open source and freely available 77 online on various operating systems 10. Cost-wise, ECMOjo is considered cost effective 78 compared to a comprehensive didactic ECMO course. The cost of a 3.5-day didactic 79 5 course in 2010 was 2,700 USD (including airfare and accommodation). However, the 80 introductory ECMO material covered lasts 4 hours, which is estimated to 285 USD, 81 which is significantly less than the former alternative. 82 [insert Figure 1] The purpose of this article is to assess the efficacy of ECMOjo in clinical training 83 through a skills acquisition study, which was carried out to answer the following 84 question: can ECMOjo replace didactic instruction? The study is based on the following 85 hypotheses. First, the use of ECMOjo generally improves the acquisition of ECMO skills 86 over conventional classroom learning. Second, ECMOjo will result in better learning 87 outcomes than didactic classroom teaching. 88 Methods 89 Sample Size 90 A total of 51 medical professionals were recruited for the skills acquisition study from 91 four hospitals (Kapiolani Women and Children’s Center (Honolulu HI), University of 92 Pittsburgh Medical Center (Pittsburgh PA), Phoenix Children’s Hospital (Phoenix AZ), 93 and Lutheran General (Chicago IL)) that host ECMO centers in 2008-2010. From the 51 94 datasets collected, 7 have been excluded (5 from the ECMOjo group and 2 from the 95 didactic group) because of (a) non-medical personnel, (b) missing data, and (c) data 96 recording issues, and so 44 datasets have been analyzed. The participants did not 97 receive any incentive for enrolling in the study and they were randomly assigned to 98 either the ECMOjo or didactic classroom group. 99 6 Table 1 summarizes sample size demographics. Twenty-five out of the 44 100 participants were experienced ECMO practitioners which we define as a nurse, 101 respiratory therapist, perfusionists, or physician with at least 5 years of ECMO 102 experience. Conversely, participants with less than 5 years of ECMO practice were 103 considered novice ECMO practitioners. Among the 44 analyzed datasets, the ECMOjo 104 comprised of 13 ECMO experienced ECMO practitioners and 7 novice ECMO 105 practitioners, and the didactic classroom group included 12 experienced ECMO 106 practitioners and 12 novice ECMO practitioners. 107 [insert Table 1] 108 IRB Approval 109 This study has been IRB-approved from the Department of the Army and University of 110 Hawaii (Award Number W81XWH-06-2-0061). 111 112 Study Design 113 The study design is illustrated in Figure 2. We chose a prospective cohort scheme since 114 the impact of ECMOjo is presently examined on the participants. The study proceeded 115 as follows. First, participants filled in a demographic questionnaire, went through the 116 pre-training wet lab test. Second, subjects were randomized into one of the two groups 117 and commenced the training sessions–whether using ECMOjo or an already existing 118 didactic classroom-based course in one of four training centers. The training scenarios 119 used were identical in both groups and lasted for an hour, and hence ECMOjo can be 120 isolated as the learning variable. Third, participants were assessed through an 121 7 evaluation post-training wet lab and two questionnaires (described in Assessment). 122 Appendix A includes the wet lab assessment cases utilized during the study. We 123 assumed that participants had existing learning resources (e.g. The ELSO Red Book or 124 ELSO Specialist Training Manual) to complement the provided course material 11,12. 125 [insert Figure 2] 126 127 Debriefing 128 The Gather, Analyze, Summarize (GAS) model was the debriefing method employed in 129 the study, which is a structured format for post-simulation debriefing, relying heavily on 130 the debriefer's ability to intently listen to learners 13. It was applied after the critical 131 training process (i.e. the didactic and ECMOjo sessions) to answer questions that the 132 subject may have had regarding the carried out scenario. It consists of three stages: 133 firstly, asking for clarifications to obtain additional information (Gather), then interpreting 134 responses (Analyze), and finally encapsulating the key lessons learned from the training 135 session's (Summarize). Debriefing was employed as means to enhance learning 136 outside of the study and not as an assessment tool. It was optional and thus not all 137 participants took part. 138 139 Assessment 140 Preceding the training sessions, a simple wet lab assessment was conducted. It 141 consisted of an ECMO circuit check exercise where the learner examined the ECMO 142 circuit at different locations (e.g. blender, roller pump, and oxygenator) and was 143 8 assessed objectively according to a checklist developed and tested prior to conducting 144 the study (available in supplementary materials). After completing the training, three wet 145 lab assessment tasks were randomly assigned to each participant (out of nine). Wet lab 146 cases included for example gas failure, heater failure, pump failure, and air in the circuit. 147 For the three wet lab cases participants were assessed by one examiner, Dr. Mark 148 Ogino (medical director) according to an evaluation checklist (available in 149 supplementary materials) corresponding to their expected interventions and the time 150 elapsed to complete the case, then both groups completed two questionnaires. The 151 questionnaires were part of the didactic course evaluation material used for feedback 152 for course organizers. The first is the reaction questionnaire (RQ), which reports the 153 participants’ own evaluation of the material presented in the training sessions based on 154 Likert scale responses (i.e. to determine how did the participants felt about the course). 155 Example questions are “the material covered was relevant to my duties as an ECMO 156 specialist” and “how was the level of difficulty of the module?”. Second, the learning 157 environment questionnaire (LEQ), which assessed the self-efficacy obtained from the 158 employed learning method (didactic classroom or ECMOjo). Sample questions are “I 159 feel confident in my ability to adequately manage a patient on ECMO” and “I feel 160 confident in my ability to manage identified abnormalities in the ECMO patient and 161 circuit”. Both questionnaires were given before and after the training sessions. 162 163 Statistical Analysis 164 To analyze participants’ responses, a nonparametric test for correlation on paired data 165 was used. Fisher’s Exact Test was chosen since it is a commonly used test of 166 9 independence for small sample sizes. A p-value of 0.05 was selected as justification for 167 rejecting the null hypothesis, which is defined as the following: the two groups (i.e. 168 ECMOjo and didactic classroom) assessed using the RQs and LEQs show no 169 conclusive statistically significant difference in the wet lab assessment performance and 170 efficacy. To further analyze the relationship between the learning method and wet lab 171 assessment performance, RQ and LEQ responses respectively, an average score for 172 each of these assessments was calculated for each participant and was tested 173 accordingly against the learning method employed. For each of the two questionnaires, 174 itemized response averages were calculated and then a cumulative average was 175 computed from these averages collectively. 176 177 Results 178 Prior the training sessions, the participants were pre-assessed. On average, the didactic 179 classroom group scored an average of 6.1 (out of 7) and 4.1 (out of 5) in the RQ and 180 the LEQ respectively. On the other hand, the ECMOjo group tallied an average of 5.6 181 (out of 7) and 4.1 (out of 5) in the RQ and the LEQ respectively. 182 Following the training sessions, participants were collectively assessed through a 183 wet lab. Figure 3 compares wet lab post-training scores between the two groups. The 184 maximum possible score per case was 1.0 (=100%) and each participant went through 185 three cases (of out nine). It was found that there was no statistically significant 186 difference between the two groups. It is noteworthy to mention that no statistically 187 significant difference in performance has been observed between experienced ECMO 188 10 practitioners and novice ECMO practitioners in both the ECMOjo and the didactic group. 189 [insert Figure 3] 190 After the wet lab assessment, post-training questionnaires were distributed. Didactic 191 classroom scored an average of 6.2 (out of 7) and 4.3 (out of 5) in the RQ and the LEQ 192 respectively and the ECMOjo participants scored an average of 6.1 and 4.4 in the RQ 193 and the LEQ respectively. Figures 4 and 5 depict the pre and post-training RQ and LEQ 194 scores for didactic classroom teaching and ECMOjo groups respectively. They 195 represent how participants scores varied before and after training exposures based on 196 their corresponding groups. 197 [insert Figure 4] 198 [insert Figure 5] 199 Discussion 200 Conventional didactic classroom teaching is prevalent in ECMO 8,14 as other educational 201 approaches present issues of human and physical resources allocation that 202 considerably limit training opportunities 15. This is where SBT and screen-based 203 simulation come into play to tackle these drawbacks in a cost-effective manner. 204 Thereupon we introduced ECMOjo that is a free screen-based simulator that relies on a 205 sophisticated empirical model that can help instill important cognitive skills by simulating 206 various pediatric ECMO scenarios–without the presence of a permanent instructor. The 207 aim of this article is to evaluate the educational effectiveness of ECMOjo through the 208 presented skills acquisition study to determine whether ECMOjo can replace didactic 209 11 instruction. Initially, we have used the assumption that ECMOjo would generally improve 210 users’ acquisition of ECMO skills and learning outcomes more than didactic classroom 211 learning. 212 Analyzing the data of the wet lab assessment, RQ scores, and LEQ scores, it is 213 evident that there is a similarity between the ECMOjo and didactic teaching groups’ 214 performance level. The resultant p-values are greater than 0.05 (i.e. no statistical 215 significance) and we therefore accept the null hypothesis (See Table 2). There is an 216 ample similarity between the responses (of RQ and LEQ) of the ECMOjo group and the 217 didactic classroom group. 218 [insert Table 2] 219 Consequently, we conclude that our hypothesis could not be satisfied since the null 220 hypothesis is fulfilled. This skills acquisition study indicated that ECMOjo is probably 221 equivalent to conventional didactic classroom learning in terms of learning outcomes, 222 self-efficacy, and learner performance, but it could not be statistically proven. ECMOjo 223 can still be considered a complementary education tool to wet labs/mannequin-based 224 simulation that enriches the ECMO learning experience. 225 ECMOjo allows learners to explore ECMO concepts wherever they prefer (e.g. at 226 home) and at their own pace thanks to the low-cost setup of the simulator. On the other 227 side, this will free up educators for other potential learner-centered, hands-on activities. 228 Furthermore, learning institutions may achieve savings by using ECMOjo in the 229 classroom, following an emerging trend in medical education 16–18. 230 The study has revealed that ECMOjo has several limitations. First, training using 231 12 ECMOjo does not instill hands-on and teamwork skills, which necessitate a more 232 immersive simulation-based approach with a patient simulator and the ECMO 233 equipment. Second, the small sample size limited the validity of the results obtained. 234 Third, post-training wet lab assessment were varying in difficulty, which has led to 235 discrepancies in assessments scores. Fourth, it is difficult to compare experience/a role 236 of physician and perfusionist/nurse during ECMO application. Assuming that the 237 learning curve for physician might differ be longer than for nurse/perfusionist. It is quite 238 true that around a decade ago and ECMO technology have advanced throughout those 239 years, notwithstanding, we believe that the educational value of teaching the core 240 ECMO concepts in an interactive manner will withstand technological changes and 241 provide meaningful educational value. Also, the simulator has been updated several 242 times since the study, though still maintaining the same look and core functionality. 243 Further required developments of the simulator comprise incorporating recent 244 ECMO technological advancements, adaptation to adult patient simulation, more ECMO 245 configurations (e.g. VA and VVA), and supporting more advanced training scenarios and 246 compounded scenarios (e.g. multiple circuit complications occurring simultaneously). 247 In the grand scheme, the study did not provide strong evidence to support our 248 hypothesis that ECMOjo generally improves the acquisition of ECMO skills over 249 conventional classroom learning. However, the data show that it is an alternative 250 training approach to consider while keeping in mind the aforementioned limitations. 251 252 13 Conclusion 253 This article evaluated the educational efficacy of ECMOjo through a prospective cohort 254 study, concluding no statistically significant dissimilarity between training through 255 ECMOjo or classroom-based teaching in terms of learning outcomes, self-efficacy, and 256 learner performance. Future developments include adding compatibility to recent ECMO 257 setups. In the grand scheme, ECMOjo can be regarded as a case study in the path 258 towards more high-fidelity, cost-effective screen-based simulators that employ the ever-259 growing power of computers. 260 14 References 261 1. McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. A critical review of 262 simulation-based medical education research: 2003–2009. Med Educ. 2010 Jan 263 1;44(1):50–63. 264 2. Lopreiato JO, Downing, D., Gammon, W., Lioce, L., Sittner, B., Slot, V., et al. 265 Screen-based Simulation. In: Healthcare Simulation Dictionary [Internet]. Society of 266 Simulation in Healthcare; 2016 [cited 2017 Jun 9]. Available from: 267 http://www.ssih.org/Dictionary 268 3. Alinier G. A typology of educationally focused medical simulation tools. Med Teach. 269 2007 Jan 1;29(8):e243–50. 270 4. Bonnetain E, Boucheix J-M, Hamet M, Freysz M. Benefits of computer screen-based 271 simulation in learning cardiac arrest procedures. Med Educ. 2010 Jul 1;44(7):716–272 22. 273 5. Frenckner B. Extracorporeal membrane oxygenation: a breakthrough for respiratory 274 failure. J Intern Med. 2015 Dec 1;278(6):586–98. 275 6. MacLaren G, Combes A, Bartlett RH. Contemporary extracorporeal membrane 276 oxygenation for adult respiratory failure: life support in the new era. Intensive Care 277 Med. 2012 Feb;38(2):210–20. 278 7. Extracorporeal Life Support Organization. ECMO and ECLS Statistics - International 279 Summary [Internet]. Extracorporeal Life Support Organization. 2017 [cited 2018 280 15 Jan 29]. Available from: 281 https://www.elso.org/Registry/Statistics/InternationalSummary.aspx 282 8. Anderson JM, Boyle KB, Murphy AA, Yaeger KA, LeFlore J, Halamek LP. Simulating 283 Extracorporeal Membrane Oxygenation Emergencies to Improve Human 284 Performance. Part I: Methodologic and Technologic Innovations: Simul Healthc J 285 Soc Simul Healthc. 2006;1(4):220–7. 286 9. Zakhary BM, Kam LM, Kaufman BS, Felner KJ. The Utility of High-Fidelity 287 Simulation for Training Critical Care Fellows in the Management of Extracorporeal 288 Membrane Oxygenation Emergencies: A Randomized Controlled Trial. Crit Care 289 Med. 2017 Aug;45(8):1367. 290 10. ECMOjo Simulator and Trainer - Home [Internet]. [cited 2017 Jun 5]. Available 291 from: http://ecmojo.sourceforge.net/ 292 11. Brogan MDLL MD Roberto Lorusso, MD,Ph D Graeme MacLaren, MD Giles Peek, 293 MD Thomas V. Extracorporeal Life Support: The ELSO Red Book. ELSO; 2017. 294 12. Extracorporeal Life Support Organization (ELSO). ELSO Guidelines for Training 295 and Continuing Education of ECMO Specialists [Internet]. 2010 [cited 2016 Dec 296 11]. Available from: 297 https://www.elso.org/Portals/0/IGD/Archive/FileManager/97000963d6cusersshyerd298 ocumentselsoguidelinesfortrainingandcontinuingeducationofecmospecialists.pdf 299 13. Oriot D, Alinier G. Chapter 1: Introduction to debriefing. In: Pocket Book for 300 Simulation Debriefing in Healthcare | SpringerLink [Internet]. Springer International 301 16 Publishing; 2018 [cited 2018 Jan 29]. Available from: 302 https://link.springer.com/book/10.1007%2F978-3-319-59882-6 303 14. Weems MF, Friedlich PS, Nelson LP, Rake AJ, Klee L, Stein JE, et al. The Role of 304 Extracorporeal Membrane Oxygenation Simulation Training at Extracorporeal Life 305 Support Organization Centers in the United States. Simul Healthc. 2017 306 Aug;12(4):233. 307 15. Weinstock PH, Kappus LJ, Garden A, Burns JP. Simulation at the point of care: 308 Reduced-cost, in situ training via a mobile cart. Pediatr Crit Care Med. 2009 309 Mar;10(2):176. 310 16. Briz-Ponce L, Juanes-Méndez JA, García-Peñalvo FJ, Pereira A. Effects of Mobile 311 Learning in Medical Education: A Counterfactual Evaluation. J Med Syst. 2016 Jun 312 1;40(6):136. 313 17. Lopreiato JO, Sawyer T. Simulation-Based Medical Education in Pediatrics. Acad 314 Pediatr. 2015 Mar 1;15(2):134–42. 315 18. Sanchez-Glanville C, Brindle ME, Spence T, Blackwood J, Drews T, Menzies S, et 316 al. Evaluating the introduction of extracorporeal life support technology to a tertiary-317 care pediatric institution: Smoothing the learning curve through interprofessional 318 simulation training. J Pediatr Surg. 2015 May;50(5):798–804. 319 320 17 Figures 321 322 Figure 1. Overview of ECMOjo Simulator. CDI monitor Vital signs monitor Ventilator monitor Virtual patient Pressure monitor Heater controller Gas blender Pump controller Virtual circuit 18 323 Figure 2. Study Flow Design. Icons made by Freepik from Flaticon is licensed by 324 CC 3.0 BY. 325 326 Demographic survey and randomization (n=44) Participant enrollment (n=51) Training on ECMOjo (n=20) Didactic classroom training (n=24) Pre-training questionnaires (n=44) Post-training wet lab assessment (n=44) Post-training questionnaires (n=44) Excluded participants (n=7): Non-medical personnel Missing data Data recording issues Pre-training wet lab assessment (n=44) 19 327 Figure 3. Post-Training Wet Lab Assessment Score. 328 329 Figure 4. Reaction Questionnaire (RQ) Score. 330 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1 2 3 4 5 6 7 8 9 Av e ra ge Sc o re (m ax 1. 0) Case Number Post-Training Wet Lab Assessment Scores Didactic Classroom ECMOjo 0 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 Av e ra ge Sc o re (ou t o f 7 ) Question Number Reaction Questionnaire (RQ) Scores Pre-Training Didactic Classroom Post-Training Didactic Classroom Pre-Training ECMOjo Post-Training ECMOjo 20 331 Figure 5. Learning Environment Questionnaire (LEQ) Score. 332 333 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 1 2 3 4 5 6 7 8 Av e ra ge Sc o re (ou t o f 5 ) Question Number Learning Environment Questionnaire (LEQ) Scores Pre-Training Didactic Classroom Post-Training Didactic Classroom Pre-Training ECMOjo Post-Training ECMOjo 21 Tables 334 Table 1. Study Demographic Overview. 335 Participants Enrolled Experienced ECMO Practitioners Novice ECMO Practitioners Nurses 15 13 2 Respiratory Therapists 7 3 4 Perfusionists 5 5 0 Physicians 10 (6 fellows and 4 faculty) 0 10 Other 7 4 3 336 Table 2. Data Analysis Summary. 337 Fisher’s Exact Test  −  Relationship Between Learning Method and Wet Lab Assessment Results 0.282 Relationship Between Learning Method and RQ Responses 0.720 Relationship Between Learning Method and LEQ Responses 0.634 338