The Future of Simulation Education: Harnessing Generative AI

Dr. James K. Stoller:

Hello and welcome to MedEd Thread, a Cleveland Clinic education podcast that explores the latest innovations in medical education and amplifies the tremendous work of our educators across the enterprise.

Dr. Tony Tizzano:

Hello, welcome to today's episode of MedEd Thread, an education podcast exploring how cutting-edge simulation scenario development is being enhanced through an AI-uminative collaborative approach. I'm your host, Dr. Tony Tisano, Director of Student and Learner Health here at Cleveland Clinic in Cleveland, Ohio. Today, I'm very pleased to have Dr. Stormy Sweitzer, Senior Program Manager in Cleveland Clinic's Office of Interprofessional Learning here to join us. Stormy, welcome to today's podcast.

Dr. Stormy Sweitzer:

Thank you. Glad to be here.

Dr. Tony Tizzano:

Also joining is Dr. Nicole Brown, senior education specialist at Cleveland Clinic, the Cleveland Clinic Simulation Advanced Skills Center. Nicole, glad to have you.

Dr. Nichole Brown:

Thank you. It's a pleasure.

Dr. Tony Tizzano:

And finally, we have Maureen Washock, Simulation Education Specialist, also with the Cleveland Clinic Simulation and Advanced Skills Center. Maureen, welcome to today's episode of MedEd Thread.

Maureen Washock:

Thank you. It's good to be here.

Dr. Tony Tizzano:

So please get us started by each of you telling us a little bit about yourselves, your educational backgrounds, what brought you to Cleveland and your roles here at Cleveland Clinic. Stormy, can we start with you?

Dr. Stormy Sweitzer:

Sure. So I'm a senior program manager in the Office of Interprofessional Learning. And our focus is really to support interprofessional clinical teams in improving team performance and effectiveness, as well as improving the relational quality of the members of the teams. And I have a PhD in organizational behavior and a master's in public health. I worked for a long time in quality improvement, patient safety, and digital transformation in healthcare. My PhD and other professional experience involved a lot of organizational team and leader development. And so I bring a lot of that to the work I do here.

Dr. Tony Tizzano:

Fabulous. Well, we've had you before and it's always a pleasure. Maureen.

Maureen Washock:

I am a simulation education specialist within the Simulation Center. My focus area is on in situ simulation, which is simulation that we conduct in the clinical spaces. That includes human factors for systems design and some work with our mock code program and quality improvement projects. I am a nurse by trade and background. Pediatrics is my clinical background space. I have a bachelor's in nursing. And here at the clinic, I was a nursing professional development specialist prior to moving into simulation and pediatric nursing prior to that.

Dr. Tony Tizzano:

Nicole.

Dr. Nichole Brown:

Yeah, I'm a senior simulation education specialist within the Simulation and Advanced Skill Center. I have been working in this realm of simulation now for the better part of 15 years, and really do many different aspects related to that, whether it's planning curriculum, helping train facilitators near and far, all in how to improve and advance simulation education here for Cleveland Clinic.

Dr. Tony Tizzano:

So today, we're going to be looking at the mastery of critical complex skills as an imperative in many aspects of medical education. How can we deliver the absolute best and realistic training possible in a time-efficient and cost-effective manner while at the same time not putting anyone real at risk. In healthcare, simulation-based learning has become the cornerstone of how professionals learn and perfect essential skills. But creating these crucial high-fidelity scenarios is a time-consuming and labor-intensive process. In today's episode, we will explore how human experts and generative AI have come together and collaborate to customize and elevate the succeedingly complex process. Stormy, please help frame today's topic by providing our listeners with some context around the importance of this novel work in transforming scenario development.

Dr. Stormy Sweitzer:

Sure. So I believe this began as a very informal conversation about how do we improve the process of developing scenarios for the simulation training that SASC does. And my background, I happen to have a very technical background, worked, as I said, in digital transformation, had a software company where I developed a predictive algorithm that was used in e-commerce sites. I come to this technology sort of as a curious, let me figure it out kind of person. And we decided, let's collaborate, let's figure out how to develop these scenarios in a way that maintains clinical accuracy. I think that was #1, to be reflective of accepted best practices around simulation education and scenario development. that included interprofessional competencies because a lot of these simulations happen in a team-based context. And so we wanted that to be the case. And so that they were reflective of the clinical pressures, the clinical topics, and some of the potential chaos that one might encounter in a scenario or in a real-life scenario, let's say. So we wanted simulations that were accurate and high fidelity in that regard.

Dr. Tony Tizzano:

And seeing some of the end products. I am so impressed with this work. So Nicole, what was the impetus for bringing generative AI into the equation and how does it impact what already must be a resource consuming endeavor?

Dr. Nichole Brown:

We started this project actually some time ago and it really began with understanding the existing strengths that we have as an organization and one of which is having a Stormy in our group and being able to leverage her experience because AI is only as good as what you put into it and really finding people who have the knowledge and the skill and how to use it in an effective way was one of the very first things. But the big driver from a simulation standpoint is we are not just a simulation center in Northeast Ohio. We have sites across Ohio, down into Union Hospital. We're down in Florida. We cross the pond, as the London folk would say. We are in London, we're in Abu Dhabi, and we need to make sure we are having quality education no matter where we are. That really is a pillar of our simulation center as well as Cleveland Clinic as a whole. and how can we provide the same level of competence and quality within our scenario development. And really this is where the idea of using this type of system really derived.

Dr. Tony Tizzano:

Does it help cut cost and time to put these things together, do you think, by using simulation and AI?

Dr. Nichole Brown:

I think it absolutely can reduce time once you have it set up. Maureen probably has a lot more to share on that topic.

Maureen Washock:

Yeah, I think the timeline that we quote for a new request start to finish is 12 weeks when you are generating everything from scratch. But to Nicole's point, we have locations that span quite the distance. But a lot of them encounter the same problems and have the same things that they need to train on. And so it doesn't make sense to have everyone doing the same work over and over again. We want to be able to standardize it. But what you standardize, you want to make sure is getting the highest quality and then the most accessible. So I think in some of those where we've identified people have similar objectives, Using AI helps us create a case that might be the foundational piece that can then be customized, modified, adapted to someone's individual workflow. In my area for insight to in systems design, every layout can be different. the resources they have available to them can be different. And having a originally structured core case that meets all of the criteria of standards of practice, both clinical and simulation, allows us a foundation to work from to make those modifiable differences that can really dramatically cut down that 12 weeks.

Dr. Tony Tizzano:

So really for an organization our size, of 80,000 plus individuals, the idea of systemness is important. How can we at each of our locations be delivering the same kind of quality care in some uniform tested manner? So I think that's great. So Maureen, what are some of the basic requirements needed to move the concept forward?

Maureen Washock:

I think some of that is truly that what your product is going to be customizable. So it's what are the standards that have to occur in the care of that patient? overall, and that could be adapted for various types of professionals. Their learning objectives may differ a little bit here and there. Most of my work on this with Stormy was to look at our resuscitation quality programs. So within the mock code program, we are asked to go and present a patient in cardiac arrest and see how they respond. educate if needed after the fact, but we're also doing some system testing to ensure that we're able to deliver CPR in a high quality fashion, that we're able to deliver a defibrillation in metrics in alignment with the AHA. And those have very clear parameters of what the timed metrics must be, what it is that they must achieve. So being able to go through and create with the prompt structure, every single time we run this type of program, we want it to run for this amount of time. It always starts with an initial baseline frame, a decompensation, a shockable rhythm, a non-shockable rhythm, and a recovery frame, as well as what are the measurements that are going to occur and what are those standard debriefing points is a really nice foundation to give to someone that might be doing it for the first time with not as many resources. They might be a clinical person that doesn't have the benefit of having the simulation center there to support them, but they can take that validated, structured framework with some guidance from us and then customize it to the type of patient population that area sees.

Dr. Tony Tizzano:

Yeah, I can only recall in medical school many years ago, we won't go into that, doing these sorts of things with mock codes, always with a different sort of preceptor and depending on who you had made a big difference. So this standardization is wonderful. Stormy, one must imagine that putting all of this together is very complex undertaking. What does the process look like?

Dr. Stormy Sweitzer:

So for us, it involved a number of steps. And as Nicole said a moment ago, your output is only as good as what you put into it. And as Maureen described, there are a lot of parameters that are needed in an effective scenario. And so we met regularly at the beginning, bringing together both the clinical expertise, the simulation expertise, their knowledge of what is needed to make these scenarios effective alongside my understanding of the AI platforms and what they need to generate the kind of output we were looking for and interprofessional parameters. And so I would say a couple of things happened in parallel. And one was to evaluate the different platforms. So as we were looking at what kind of scenarios are we trying to develop, we had a bit of a prototype of the prompt that we wanted to try. And all along this process, it was human AI, human AI. The output had to be evaluated by somebody with the clinical expertise and the educational expertise to say whether or not it was clinically accurate and provided the level of information needed to actually perform the simulation effectively. And so that expert review and feedback was critical in this process. So that prompt refinement, the incorporation of graduate medical competencies, interprofessional competencies, we needed to be able to also bring that information in there as resources. And so we ended up with a multi-stage prompt. that incorporated some of the competency information and the parameters that we specifically requested as well as the clinical information that Maureen and Nicole said, yes, this is what we want to test with. Here's our initial clinical situation. And we did this until we came up with something that worked effectively and could be iterated after that.

Dr. Tony Tizzano:

So make no mistake, behind every step, there's a human looking at this saying, did we weigh and measure this properly? Is this correct? And I think that's important for us to all know. We all think, oh, you hand it over to AI and it takes the ball and runs. No, there's a lot more to it than that. So Nicole, despite the growing familiarity with the confidence in generative AI, some might say we are at the threshold of a brave new world. What are the challenges to putting this process in motion and coming to fruition?

Dr. Nichole Brown:

I think what I will say, and then I would love to hear from everyone else, putting in information, there are sub thoughts that are happening inside of a human brain that just cannot come out of AI. And as an example, and what I mean by that is simulation is not just standard education. It's not throwing a mannequin in a bed and saying, oh, this patient, you no longer have a pulse. It's time to save its life. It is far more nuanced than that. And having the knowledge and the skill and the expertise as a simulation educator, there are things that I sometimes don't even remember to think that I need to ask, say, an AI to add into it. The facilitation plan, who is my learner level? What are my time constraints? What are some of the objectives? What are some maybe potential hidden curriculum that we're also trying to meet because we're balancing limited time from the people that are trying to learn. And so depending on what platform or software you are working on developing your scenarios, some of those extra steps may not be included. And so it does take that iterative process to not only look at what is the scenario I'm wanting to achieve, but how am I achieving it? What's the right modality? What is my facilitation plan? What is my debriefing plan? What structures am I following? All the things that make simulation such a superior way to learn through experience.

Dr. Stormy Sweitzer:

I just wanted to add as well, sometimes we have to ask if AI is the right tool to use to begin with. I have gone through this process with other teams and I think we have to temper expectations around what's possible and how much time it actually takes. It took us a long time to iterate through this process. process, I think some people think, oh, it's a simple matter of let's just collect a little information and it will spit it out perfectly. And I'm like, no, somebody needs to evaluate it. Somebody's going to need to maybe put this into a template. If you want to present it, has to show up in a different way. And so there are a lot of human steps that are still needed in this entire process for both validation and for presentation. And so I think that's the thing. People want to think that this could be a magic bullet, but it's a tool. It's a tool.

Dr. Tony Tizzano:

It's a tool and it's in the hands of a human at some point and many more points than we might think. So for any of you, please help give our listeners a sense of the impact of this exciting work by sharing some of your results and outcomes.

Maureen Washock:

I think that the comment of human sub thoughts that go on Oftentimes, I think that the human nature of things, once you know how to do something, is that you skip sub thoughts. And the beauty of AI is that it never does. Once you tell it to do those things, it does it every single time. And I think that what this has the possibility to do with these detailed scenarios is I think we've all experienced as educators when somebody leaves, they retire, they move to a new position, whatever amazing course or thing that they did leaves with them because no one else can do it quite right. And if you have utilized this tool effectively, you have built all of those sub things so that it is a perfectly packaged, I can hand this off to another qualified educator and it's just there. And I think that is a really powerful thing for us in that globalization when we have a few things that are just really well done. We've gone through the iterative process to make sure it's there. And we have a lot of amazing experts at our fingertips to validate and create things here at Cleveland Clinic. And they are not at every location all the time, but they can be the one who validates the clinical content material alongside with us because I'm a pediatric nurse and I know when I read a scenario, I can match it to the literature and say that it is correct, but I may miss the innovative processes that are occurring in a specialty that I've never worked in. And they can bring that in and validate it out for our entire system to use.

Dr. Tony Tizzano:

So there's a bit of succession planning in this. So you actually, once developed and done well, you can hand it off to somebody else who can pick this up despite you having left and get a good end product.

Maureen Washock:

I think so.

Dr. Tony Tizzano:

Excellent. Stormy.

Dr. Stormy Sweitzer:

Yeah, I think One of the things we have to think about when we're looking for output, and I'll look at this from a more technical standpoint, is what are the parameters? What are the requirements of the thing that we are trying to create? And can we articulate those? And that requires a level of domain expertise and experience with the process to be able to really specify what is it we need in order to do the thing that we want to do. And I think when you become expert at something, sometimes you do it in a rote, automated process, right? Part of our understanding of human factors is like the way we learn is we go from a very, let's say, needing to learn a high skill and it takes a lot of mental time and energy to learn to sort of automating a process and to try and then go back to say, what was it that needed to happen as a learner How do I articulate that and build something that's going to provide the same experience for that novice? And so sometimes that's a challenge, but you need that. You need somebody who can break it down and be able to include some of that in the templates or the prompts that you use to get the output to provide that information. And so that's part of both understanding your learner's needs and understanding your expertise about what needs to happen. And then being very structured, when we talk about how do we hand this over, it's not just about creating a prompt. There is also like, how do we use the prompt if there's multiple phases in it? So, what's the scenario? What are the debriefing questions that are associated with these particular issues? You know, how do we need to set up the technology or the mannequins and so on and so forth? These become different steps to look at. And so it may be that there's also instructional guidance for the people using the prompts. So I would just add that.

Dr. Tony Tizzano:

So Nicole said far more nuanced and I can see listening to this. Nicole, would you like to add?

Dr. Stormy Sweitzer:

Absolutely.

Dr. Nichole Brown:

So we've talked about developing beautiful curriculum in simulation and how AI and the human collaboration can yield fabulous outcomes. But you still have to implement it. And so it still continues despite all of the iterations and all of the time to build the curriculum. You then have to again do it. And you need strong facilitators. You need organizational support for learners to attend. There's a lot more than just having a great output. And so that's also something to consider where it's only a piece of the full educational implementation.

Dr. Tony Tizzano:

That's an excellent point. Having dedicated time in this day and age to be able to learn. We have so much work to do with the skills we already have. How do we take the next step? How do we continue to raise the bar?

Dr. Nichole Brown

You just said something that really sparked something in me and it's sometimes it's dedicated time of like on a clock. you have between this hour and this hour. But between those times, are you actually dedicating learning time in your own brain and in your own life? Are you able to stop and pause? We'll frequently do education while people are working on a clinical unit. Sure. Great. That can check a box that someone can physically be there. But is their brain present to learn? Are they able to give themselves that time?

Dr. Tony Tizzano:

Sure, Lots to think about. So again, for all of you, what are some of the key insights that have come from this very innovative work?

Dr. Stormy Sweitzer:

I think for me, it's really artificial intelligence or generative AI. It excels at sort of the pattern matching, the replication, developing processes and flows, frameworks. It can help with a lot of that and it can include the things that we need. But I think ultimately it struggles sometimes with the language that comes out of it. We're seeing sort of a homogenation of... of language, the nuance and the clinical insights that come from years of practice experience, all of that still needs to be added in and the verification of the output. And so we can't replace the human in this. It can help us with the initial frameworks and putting together like a straw man that we can then enhance and adapt to the way humans navigate the world.

Dr. Tony Tizzano:

It sounds like you're saying, how do we impart wisdom to AI? I mean, that's a way to look at that. You can give it the factoids, but you still need the wisdom. Maureen, what would you add?

Maureen Washock:

So for me, I think it was that all of the tools work differently. So as we were exploring in the early part, I learned a lot about the different tools and how some of them might create a reference. As we have proceeded in our journey since the initiation of this project, our center has brought in a curriculum system that allows us to share more seamlessly with our global partners, which is amazing, but it does have some of its own built-in AI tools. And I learned that I now have to learn how that one works too, because how I verify things looks very different depending on what tool it is. And so continuing ed that we are creating for our caregivers, I have to keep up with continuing ed on AI because it was flipped entirely. It pulled the resources in and I had to go chase them down, trace them and make sure that they are the right ones. So they were, which was amazing. And I was very excited to learn that. But how I verified looked very different than how I verified it in the method that we had set up with Stormy.

Dr. Tony Tizzano:

And you live in this world and you still have to. It just shows how complicated. Nicole, you have mentioned before the iterative process. Let's make sure we all understand exactly what that is because it seems like it's said over and over again. What does that look like?

Dr. Nichole Brown:

Unfortunately, it's somewhat difficult to answer because it is going to be very specific to kind of what platform you're doing and that you're using it. But ultimately, it's not a one and done. You cannot take any generative AI platform and Stormy, correct me if I'm wrong, enter one prompt and say, this is perfect. It is constantly taking review. and tweaking and adapting no matter what system you are using. So iteration, it again may look a little different, but it is constantly enter, evaluate, enter, evaluate until you're able to get to what you're looking for.

Dr. Tony Tizzano:

So considering all of this and kind of wrapping our arms around what we've heard, it sounds like we're already in the future, but what's on the horizon? What's on the wish list? Given Dr. Stoler's magic wand, if I could, I would have.

Maureen Washock:

I have one, and it's that sometimes the barrier in what AI can pull from, it doesn't have access to a piece of knowledge that I know, but I can't give it. If I'm targeting a specific quality improvement project with a specific patient that has protected health information, not every program can I put that information into because it's not covered in a data security fashion. So sometimes there's some steps that exist because of the privacy limitations. And it would be great to have a magic wand to fix that.

Dr. Tony Tizzano:

Yes, Boy, that's a complicated one. Anyone else?

Dr. Stormy Sweitzer:

I don't know that this is a wish, but I think we need to continue to provide people with education. AI literacy is very important. Algorithmic bias is real. A lot of these data sets are trained on particular data. If we look at when research began on women's medical concerns, and a lot of the research that's out there. And so there are biases baked into the system because of the biases in our own human processes, right? And the way we approach things and the way people access the system, right? And so it might be that a population doesn't gain care or access to care because of their socioeconomic status and they use healthcare differently. They engage with the system differently and that might come through in the way an electronic health record uses predictive AI, for example. So I think being aware of where are the biases, why do we need to evaluate and verify the outputs of things? in part because there are things in there that we may not continue to believe or know given the updates in science, the access to the research. What's possible with AI? What are the limitations of it? There's also more research coming out about de-skilling. There was a study that came out recently. They implemented an AI diagnostic system. And within three months, the providers had lost some of their diagnostic ability. They were no longer able to identify certain things in patients because they had outsourced the process of diagnosing patients to the machine essentially. And so we have to be very cognizant that outsourcing our thinking patterns and processes and knowledge to AI is having real impact on our learning. And if you think about somebody new into the system, who does not yet have the level of domain knowledge, let alone years of expertise and experience, they're no skilling. They're becoming more dependent on AI to provide the things that they don't know, but they also don't have the expertise to ask the questions to get the output that's needed. So we're in this interesting time right now.

Dr. Tony Tizzano:

I thought it was very interesting you mentioned if you're applying something, a scenario perhaps, to a different population, with the different socioeconomic foundation, you may need to tweak it for that population. It's so interesting. And the whole idea of deskilling, I mean, the stethoscope is the perfect example. When I was in medical school, we had an absolute savant with the stethoscope teaching us. He could do so many things with a stethoscope that now echocardiography and all these other things have supplanted that and you just go to it to get the answer. Nicole, would you add the final word to this horizon? Where are we headed?

Dr. Nichole Brown:

We are headed as far as the humans and our team can take us.

Dr. Tony Tizzano:

That is well said. That is a pearl. So finally, is there anything that I didn't ask that you feel is important for our listeners to know?

Dr. Stormy Sweitzer:

I think just to reiterate, it is a tool. It is not the answer.

Dr. Tony Tizzano:

Appreciate your time so much. Well, thank you so much, Stormy, Maureen, Nicole. This has been a very thought-provoking and wonderfully insightful podcast. To our listeners, if you would like to suggest an education topic to us or comment on an episode, please e-mail us at education@ccf.org. Thank you very much for joining us and we look forward to seeing you on our next podcast. Have a wonderful day.

Dr. James K. Stoller:

This concludes this episode of MedEd Thread, a Cleveland Clinic education podcast. Be sure to subscribe to hear new episodes via iTunes, Google Play, Stitcher, Spotify, or wherever you get your podcasts. Until next time, thanks for listening to MedEd Thread, and please join us again soon.