Building Smarter Testicular Cancer Data Sets with Large Language Models

Podcast Transcript

Dale Shepard, MD, PhD:

Cancer Advances, a Cleveland Clinic podcast for medical professionals, exploring the latest innovative research and clinical advances in the field of oncology.

Thank you for joining us for another episode of Cancer Advances. I'm your host, Dr. Dale Shepard, a Medical Oncologist and Co-director of the Sarcoma Program at Cleveland Clinic. Today, I'm happy to be joined by doctors Chris Weight and Tim Gilligan. Dr. Weight has been a guest on this podcast previously and most recently he discussed how AI is predicting prostate cancer recurrence and that episode is still available for you to listen to. They're here today to discuss building testicular cancer data sets with large language models. So welcome both of you.

Christopher Weight, MD:

Thanks for having us.

Timothy Gilligan, MD:

Thank you.

Dale Shepard, MD, PhD:

So Tim, let's start with you. What do you do here at Cleveland Clinic?

Timothy Gilligan, MD:

So I'm a Medical Oncologist. I treat urological cancers and have a particular focus on testicular cancer.

Dale Shepard, MD, PhD:

Excellent. Chris?

Christopher Weight, MD:

I am a Urologic Oncologist and I am a counterpart with Tim. I do surgical treatment of those with testicular cancer.

Dale Shepard, MD, PhD:

Excellent. Well, today we're going to talk about testicular cancer. We're going to talk about building data sets and using large language models to kind of help move the field forward. So I guess Tim, why don't we start with you. What are some of the issues related to managing testicular cancer that we're trying to address?

Timothy Gilligan, MD:

There are a lot of issues specific to testicular cancer that are a little different than certainly most solid tumors, which tend to happen in older patients. So the great thing about testicular cancer is that we can cure most patients, 96% of patients end up cured. And these are young patients who have many years of life ahead of them. And so it's very important not to miss an opportunity to cure patients, but also anything we do that results in long-term toxicity, these people live with that toxicity for a long time and it can be quality of life issues and it can also be significant medical issues like second cancers and cardiovascular disease. And testicular cancer comes in many different stages, there are many different treatments. Surgery and chemotherapy both play a very important role and both can have an impact on patients' long-term function. And it's a disease where the stakes are particularly high to get it right because the patients have decades of life ahead of them and hopefully good health if they're treated appropriately.

Christopher Weight, MD:

I think it's one of the really exciting examples of how multidisciplinary care has really moved from what used to be almost a uniformly lethal disease to almost a uniformly cured disease, but now we're experiencing sort of the side effects of that success as some of this toxicity. And so now our field is really kind of turning to, we can cure the vast majority. Now, how do we cure them and minimize the toxicity? And it's a really specific sequence of surgery and radiation at times, but mostly surgery and chemotherapy and in the right order and for the right patients. And we're trying to figure that out and minimize the toxicity. So it is very a nuanced management process.

Dale Shepard, MD, PhD:

It's a good way to think about perspective in terms of being different than a lot of other things we talk about because kind of as we started this, there were probably people thought, "Well, don't we already do a really good job with testicular cancer? Why aren't we looking at other things?" So that's good. So let's just sort of go basic here because a lot of people, different backgrounds might be listening. Large language models, what exactly is that? How does that help us?

Christopher Weight, MD:

These are computer algorithms that at their very core, they're trying to do something very simple. They're trying to say, "If you have a string of text before, can you predict the next word that would come? And what's the next most probable word to come after the word before and then predict the next next word?" So many people have probably, I'm sure at this point that are listening, have interacted with ChatGPT or Claude or now if you do any search on any browser, there's usually an AI response. And these are large language models. They've been trained on millions and millions and millions of pieces of language and they get very good at this. The reason we're talking about them in this context is that medical notes are written in unstructured language.

Whatever doctor or nurse, however they want to put it down, it can be down in any sort of form, different people have different speaking and typing styles. Sometimes doctors dictate and then it's transcribed by a transcriptionist. Sometimes AI is doing that now, but at the core, it is a bunch of unstructured data and it's unstructured stories about how patients go through their medical course and in this particular case, how young men are going through their course with testicular cancer. Testicular cancer is quite rare compared to a lot of the other cancers. Probably fewer than 10,000 people are diagnosed each year in the United States, although it is on the rise. But what that leads to is any one particular center may not have a lot of experience and some physicians rarely see anybody with testicular cancer. And so it leads to sort of this positive expertise and it also leads to a difficulty in getting together a lot of data on patients who have testicular cancer.

They're collected by a bunch of different physicians over time and it makes it a little bit harder to do research. And so what we've been working on and we started off in a different cancer in kidney cancer, but now we've refined this model and kidney cancer got a little bit better now working on it in prostate cancer, but we're really excited about it in testicular cancer because this is where we really need data from all the patients. And over many, many years we've seen a lot of patients here at Cleveland Clinic, but we've never had a systematic effort to collect that data and understand how these patients are doing.

And so this effort is to use these large language models and through a series of techniques we call prompt engineering and conditional questioning, we will ask the language model to go into the unstructured notes, the clinical notes and try to make sense of that story and put it into it a discreet way so we can describe that and then use it to understand better how these patients are going through their journey with testis cancer, identifying the treatments that work when the treatments go wrong and try to understand more clearly the variables that affect how these patients do so we can really continue to maximize the cure but simultaneously minimize the side effects from the treatments.

Dale Shepard, MD, PhD:

And I guess, Tim, if we think about care at the Cleveland Clinic, I mean, you are renowned with treating testicular cancer. Others, not so much. How are you guys approaching collecting data from a variety of scenarios to say, "This is kind of traditionally how we've done it." But you probably have very tight guardrails in terms of how you think usually, whereas other people may be doing some more random things and what do we learn from that?

Timothy Gilligan, MD:

Well, I think one of the things that's exciting to me about this kind of data is it gives us real world data. A lot of the data that we have is from clinical trials, which are super important, but clinical trials are done under a somewhat artificial environment. There's a lot more attention being paid. It's sort of like a restaurant when they know the New York Times food critic is eating in the restaurant, the kitchen snaps to attention, but what happens when the food critic isn't there? And I think this kind of data tells us what's actually happening to real patients who aren't getting the extra attention of someone on a clinical trial and how do variations in practice impact outcomes for patients. It's a powerful new way of looking at what we ought to be doing because certainly in my career, even just in testis cancer, there've been important examples of stuff that we've had a trial where something looks really exciting and then we look at what happens in the real world and it doesn't work as well in the real world as it did in the trial that got published.

And an example of that is the use of PET scans. There was a time that we though PET scans were going to be super important in helping us manage seminoma patients and they've turned out to be less accurate than the trials initially suggested. And so I think that for me, that's helpful because yes, if someone's seeing me, I have an active role in the guidelines. If they're seeing Chris, he does a ton of testis cancer. We have an awareness of a lot of the nuances that not all of oncologists are going to have, but we want to know what's happening and how do variations in practice have an impact on outcomes for patients.

And one piece of that for me also, which is separate from what I was just talking about, is how can we individualize care better? For testis cancer right now, like in my world when I'm giving chemo, for most patients, they have advanced disease when they're getting chemo. We divide them into three categories, small, medium, and large, basically good, intermediate, and poor risk. But that would be like you go to the shoe store and are your feet small, medium, or large, right? You wouldn't get shoes that fit you very well. So we'd like to get our treatment to fit the patient better than just small, medium, or large.

Dale Shepard, MD, PhD:

Yeah, makes sense. Chris, when we're doing this sort of work, clinicians or clinicians, they write notes in widely different styles. How do you account for that in terms of inevitably there's data that's missing, there's things that may or may not be accurate even.

Christopher Weight, MD:

Yeah, it's a great question and it's something we struggle with in this sort of type of research. We have two solutions. One is we first have to go through and pull the data with our first effort. Almost never is the first effort accurate enough or complete enough. And so we will pull the data the best we can in an automated way, then we'll do the old-fashioned way, which is send a medical student or resident into the chart and painstakingly, they pull out the data points by hand.

That's very valuable information. We find two things. One is that often that will allow us to identify where the model got it wrong and then we can fine tune the model so it doesn't make that mistake again. The second is we find that sometimes the data's not there and that data is not there because it was never recorded. So it's leading us towards moving towards semi-structured notes where we have not the whole note. I don't think we'll ever get all physicians to structure their note completely. Although we have examples of that in places like pathology where they have really pretty good structured notes. And the pathology notes are very easy to parse back out because the pathologists have been pretty rigid about saying, "You got to think about each one of these data points and say something about it." And when you do that, they get in really good data and then it's very easy to pull that back out.

Radiology is, on a spectrum, moving towards that still a lot of freedom in radiology notes and then unfortunately clinicians like Tim and I, it's all over the place. So in order to sort of combat that, we're sort of finding what are the most critical elements that we want and then having a subset, a smart text as we would call it, where you would just go in and say, "Just please answer these four or five questions in a structured way so then we can make sure that data is captured." The good news is there's some data we're looking for is already structured and discreet in there, things like labs. Those are already in there. So we can pull those out with ease, time in the hospital, types of chemotherapy they've gotten, types of surgery, that is all in there in a structured way. So we can pull those kinds of data out, but we do have to make sure we encourage clinicians. And part of our way of thinking about this going forward is we get a sort of self-perpetuating automated way we could build dashboards.

And then if we have, for example, a clinician who's never including critical data in their notes, we'll know that and we'll say it's not pulling it out and we'll be able to go to that physician and say, "Hey, you're not getting this data that's really important into the note. Please use these templates." The other comment I'd like to make about these things is the gold standard has always been the physician or a physician trained personnel, like a medical student or a resident or fellow looking through the medical record and writing it down. The real gold standard is actually doing it in real time, like we see on a clinical trial, but the vast majority is not done that way. It's going back through the chart and writing it down into a spreadsheet.

What we've found as we've done this is sometimes we'll say, "Okay, we've got some discrepancies here. We'll send a third person into the chart." So we have now a human generated set of data, computer generated, large language model set of data will compare them and usually they're pretty comparable, usually in the 90% plus. On the discrepancies, we've sent a third human in or a third reviewer in, one human, one computer, and now a second human to be the tie breaker and most of the time on most variables, about 60% to 70% of the time the human, the arbitrator will agree with the model, meaning the model is more accurate when you slow down and take a look.

And I don't know, Tim, you probably did this, you probably did this too, Dale. I reviewed a lot of charts when I was in training. After doing 100, you get tired, you get something off and we found that repeatedly that the model is usually a little bit more reliable than two humans pulling data out. So it also underscores a little bit that a lot of our clinical data science has come from a flawed method and there's some level of uncertainty of the data that is collected by humans on a level of about 3% to 5% in most data sets.

Timothy Gilligan, MD:

Yeah I actually, I think that ultimately the AI will do this better. I remember when I was early in my career and I was working with people who were building data sets and these were not clinicians, mostly who I was working with and they felt like clinician data entry was incredibly unreliable and they were much more confident if it was a research person entering the data.

Christopher Weight, MD:

Well, there is a little to that if the outcome bears on the reputation of the physician. If it's certain things, you always want to sort of err on the side of making it look good and even if it's not even intentional, right? I think it's just human nature. I think we also found that in one of these sort of more judgment sort of decisions from a physician collected data set and then we did the AI collected data set and there was quite a big discrepancy which we were surprised when we usually see a pretty tight correlation and we went back and looked and there was a lot more judgment that was a little bit more favorable towards the clinical team when humans are putting it in.

Timothy Gilligan, MD:

I think it's like self-driving cars, like ultimately they're going to be safer than human-driven cars, but we're wary of setting it loose.

Dale Shepard, MD, PhD:

Tim, you said before about can we learn things about treatment? Can we learn things about survivorship issues? I'm going to guess, I'm asking, survivorship questions sound like they might be a little bit harder to capture a lot of data on because young patients not so much formal survivorship over the years, people lost to follow up, they moved other cities, whatever. Has it been more difficult to sort of capture that survivorship piece?

Timothy Gilligan, MD:

It will be. I mean, if people would stay in the same healthcare system, then that would be a very answerable question, but people move around a lot. I think ultimately that will be helpful, but more and more different hospital systems are using linked records. So in the future, it really should be possible even when people relocate, to keep track of a large proportion of them, not all of them obviously, but like right now someone goes to UPMC in Pittsburgh, I can pull up a lot of the data from UPMC as long as permissions have been given and those systems are more and more linked, which is in patient's interest because if you're from Pittsburgh and you end up in the emergency room in Cleveland, it's really helpful if the doctors in Cleveland can see basic medical facts and treat you appropriately as a result. So I think we'll get there where we're not there yet.

Christopher Weight, MD:

It's a really good point though, Dale, because these patients are young. Some of them don't have insurance. Many of them are moving and changing jobs and changing locations and it's part of the reason probably took us a long time to figure out these side effects are catastrophic because some of these catastrophic side effects don't show up for 15 or 20 years and we were patting ourselves on the back and we still should. It's a really great example of long-term survivorship, but we didn't see some of these outcomes for many, many years because they are hard to follow and they live so long that they didn't show up. And so it's something that will be a challenge, but I think these tools will give us the best shot at the data that we have available to see those that have stayed in the area or had touches to our healthcare system that we'll be able to get some of that longer term data.

Dale Shepard, MD, PhD:

As you've been doing this, any data available yet that is surprising?

Christopher Weight, MD:

We did. Our very first foray into the testicular cancer space was to see how reliably we could parse data from unstructured operative notes and pathology reports. And we're going to present this at a meeting. Well, actually we've submitted it to the meeting. We don't know if they'll accept it. I suspect they will because there's so few studies on testis cancer, but the bottom line is that it did a pretty good job. Similar to what we've seen in other diseases, it was able to pull out in a fairly reliable way the histologic subtype, the percentage distribution, how big the tumor was, some of the basic pathologic details that are quite predictive of how that patient should be treated, the staging, those kinds of details. And so we're encouraged by this first foray and we're working on right now doing this on a much larger scale into the more complicated patients in the retroperitoneal lymph node dissections and what kind of chemotherapy they've gotten and that's the project that we're working on right now. But the early signal on the more straightforward pathologic report was pretty encouraging.

Dale Shepard, MD, PhD:

Tim, did you see what he did? He was trying to invite himself back for talking about the results.

Timothy Gilligan, MD:

That's right. It's exciting. I think we all want to see that.

Dale Shepard, MD, PhD:

So it started with kidney, now you're working within testis. What's next?

Christopher Weight, MD:

We're working on prostate. This method has been, because it has shown a robustness across disease, across location, we're doing it enterprise wide. So we have notes from physicians in Florida. We have notes from physicians in Akron and Hillcrest and Fairview. In the kidney dataset, we had 167 different surgeons, so that's 167 different types of notes that are written as well as a wide array of pathologists and radiologists. And it seems to be fairly robust method across that spectrum of clinicians. So we're excited about that. So we're in urologic cancers, genital urinary cancer. So we're focusing on those primarily, but we've been working closely with colleagues in Taussig, others in the Quantitative Health Sciences Center as well because I think this method and this approach helps us to get what is really appealing, the really appealing idea of taking all of the data from every patient, all the interactions from all of the treatments and how they do and getting it to bring to bear at the next decision that we need to make.

And I think it's estimated that if we were to convert the bit of data into a dot, it would be somewhere on the order of 120,000 dots on each patient each year. And right now on average, we use only like 3 or 4. And I think there's so much data being collected. We're really trying to work at a way to put it together where we can bring the imaging data and the clinical data and genomic data and laboratory data and bring all of that to bear in really helping guide us through. So I think there's a lot of excitement and a lot of different spaces for using these kind of tools to approach all kinds of diseases.

Dale Shepard, MD, PhD:

Fascinating work. Sounds like a great way to learn a lot and help sort of individualize care in the future, which brings up, I guess maybe as we close, really important disease. There's a reason this shows up on boards all the time because it's just something that's curable. You don't want to miss things, you don't want to miss a opportunity to treat and you don't want to over treat. So maybe both of your kind of perceptions, how do you work with people in the community to make sure that the right things are happening? How do we avoid that problem with overtreatment or under treatment? What's your guidance in terms of if you have the opportunity to say, "Please do this when you see a patient." What would that be?

Timothy Gilligan, MD:

I think every patient should have their case reviewed by experts in the disease. Kaiser Permanente in California has implemented that as a policy. Every patient of theirs who assessed this cancer has to go to Testis Cancer Tumor Board. They can get treated locally. They get treated by the person who diagnosed them and is managing them, but the case gets reviewed to make sure that it's the right treatment plan. And I would like to see that happen. That is starting to happen. I think Chris and I both get a lot of patients sent by us not to treat, but just to make sure, are we doing the right thing? Is this the right thing to do? And it's so much easier to do the right thing in the beginning rather than to try to clean up after not doing the right thing because people are going to end up with toxicity or needing more treatment than they otherwise would've need to.

I think a lot of the treatment can be given locally. The drugs we use are widely available. You don't have to come to Cleveland Clinic to get the drugs, but you do want to have the right treatment plan because I do see patients getting chemo when they don't need it or getting too much chemo or not having surgery when they should have had surgery and you wish you'd seen them six months earlier. I know Chris has a lot of stories like that as well. And so it's a rare disease. And I think one of the things that we see in any area of expertise is that there really isn't a substitute for experience. There's nuances in this.

I remember when I was in residency training and we asked one of the great... Our chair of neurology was a fantastic speaker and he said, we asked him, "How do you give a good talk?" He said, "The first thing to giving a good talk is knowing what you're talking about." He said, "I don't want to hear a talk by someone who's been reading about the subject for two weeks in the library. I want someone who's really lived it." And I think that applies here, that it's a complicated disease and it's nuanced and having someone who's seen a lot of it just review the treatment plan would be my request.

Dale Shepard, MD, PhD:

Yeah. Chris?

Christopher Weight, MD:

Well, it's one of my pleasures actually getting to work with Tim because he's been on the Guidelines Committee for quite a long time. And so I would completely concur with that and we are happy to work with any group. We're happy to have as much be managed locally and happy if they shuttle back and forth or if you just want to ask us for advice. We're happy to do that because we care about this disease and the patients who get this disease and we want them to get through it. I would say as opposed to chemotherapy, the surgical expertise is not easily concentrated. It's not easily distributable because it's such a rare disease and the real estate where this disease spreads is a difficult spot. It's right in between the two biggest blood vessels in the body. And so I think it does require a team that has experience because the results could be catastrophic and even lethal, which is obviously really poignant when the patient is in their 20s and has their whole life before them and many times, they're young fathers or just getting going in life.

So from a surgical standpoint, I would say if there is a need for removing the testicle, that's a very common straightforward surgery right at the beginning. But if they need the retroperitoneal lymph node dissection or sometimes it goes to the chest, I think that's probably best collapsed into a center of excellence and that's something we're also excited. We've never put together our experience on this, but I think it's going to be one of the larger ones in the world once we get the data altogether.

Timothy Gilligan, MD:

I think that's a really important point that the chemotherapy, as long as it's the right drug and the right dose, it doesn't matter so much where you get it, but the surgery is really very important. And as Chris said, it can be complicated. I remember case I trained in Boston to start my career there and I had a patient who had a tumor left in their chest that was very difficult to remove and the surgeons I was working with didn't feel like they could do it. And so we could have said it's inoperable, but actually I sent them to New York City and they had a surgeon there who he took it out and the patient was cured. So there's a big difference between chemotherapy and surgery and these operations are complicated and you need the right person doing it.

Dale Shepard, MD, PhD:

Well, you guys are doing great work and appreciate you joining us for some insights today.

Christopher Weight, MD:

Thanks, Dale.

Timothy Gilligan, MD:

Thanks, Dale.

Dale Shepard, MD, PhD:

To make a direct online referral to our Cancer Institute, complete our online cancer patient referral form by visiting clevelandclinic.org/cancerpatientreferrals. You will receive confirmation once the appointment is scheduled.

This concludes this episode of Cancer Advances. For more podcast episodes, visit our website, clevelandclinic.org/canceradvancespodcast. Subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Thank you for listening. Please join us again soon.