>>KOPELOW: I'm Murray Kopelow, the president and CEO of the Accreditation Council for Continuing Medical Education.
>>CERVERO: Hello, I'm Ron Cervero. I'm a professor of Adult Education and Associate Vice President for Instruction at the University of Georgia.
>>KOPELOW: So Ron, we're here to talk about a couple of the reports that you've written for the ACCME on the effectiveness of continuing medical education, on bias in continuing medical education, and to try to integrate them for our accredited providers on how they can apply it, how they can use it, how to link it back to some of the rules and regulations that we have.
>>CERVERO: Right
>>KOPELOW: So let's talk about the effectiveness of continuing medical education report. First, tell us how did you get involved with this report, you and Julie Gaines, and tell us a little bit about yourself and Julie Gaines.
>>CERVERO: So I first got involved in CME in 1979, working for what was then the Illinois Council on Continuing Medical Education, which was a arm of the Illinois State Medical Society. So that kind of dates my work in CME and we've had a longstanding interest in CME, and then I was on the faculty at the University of Georgia since 1986. Sometime in the early 90s, I started seeing all of these systematic reviews of the impact of CME and thought it would be really good to bring them altogether, to compile, and to synthesize them. I had a PhD student, Karl Umble, and asked him if he would be interested in working with me on this. So we did the first report in 1996, came out in 1996 in the evaluation of the health professions that reviewed all of the systematic reviews to that point.
>>KOPELOW: And that went back, the review is back to 1977?
>>CERVERO: Right. I believe the first one was '77. Brooks and Brooks-Bertram was the first systematic review. It was relatively primitive by today's standards, but still sought to bring together all the individual studies. And then at some point, probably in the early 2000s, you and I had a conversation, and you thought, "Geez, this should be updated. There's a lot more out there." And so we started a conversation with you and I had another PhD student, Katie Robertson, who I asked to join the team and to be a key partner as well as Karl Umble who was the author on the first study in '96.
>>KOPELOW: Part of that was, at that time, some articles that appeared in the medical education literature and the continuing education literature saying CME wasn’t effective, and there were some quotes that were talked about, like traditional CME wasn't effective. And that sort of activated us, and me because the literature didn't say that.
>>CERVERO: No.
>>KOPELOW: And it was an opportunity to look back to see what happened since 1996.
>>CERVERO: Right.
>>KOPELOW: So for us to be able to make the statement, right?
>>CERVERO: In fact, in 1996 article we said there's enough systematic reviews out there that demonstrate CME is effective. We should even stop asking the question because it's settled, and we should be really asking the more important questions, what are the mechanisms of action? What kind of CME? What types of CME are effective? And apparently, not everyone heard that. And so, we thought it was time to refresh and review the literature so we put together this team, we looked at the evidence, we added the additional systematic reviews, and in 2003, the Journal of Continuing Ed in the Health Professions published an update of that, which came to the same general conclusions that CME is effective. We have many systematic reviews that demonstrate that we really should not be spending our time on that question. The more important question is what types of CME are effective? And at that point we started to get a lot of good evidence around things like active learning and so on were essential.
>>KOPELOW: So I came to you again in 2014, and this time it wasn't a graduate student that you asked to help, it was Julie Gaines.
>>CERVERO: Right. Julie Gaines is a medical librarian on our campus at the medical school, and I have been working with her for five years because I've been involved with the medical school and their educational efforts there. I went to Julie and said, "You're a medical librarian, I need your assistance because I need to make sure that we have gotten all of the material that's out there," and so she was absolutely central to both of these studies actually, and she knew how to search the literature, she did a systematic review of all of it, and came up with the data that I was able to use and together we were able to write this report.
>>KOPELOW: And your role is different now? The first time we talked, you were the Head of the Department of Adult Education, and now you have a different role?
>>CERVERO: Right. So now I am at a university level position. I'm the Associate Vice President for Instruction. Our office is responsible for all the instruction that we deliver at the University of Georgia.
>>KOPELOW: Right, and I sort of liken your role in doing this like the physician who's become the head of the department but still sees patients twice a week, right? [chuckle]
>>CERVERO: Absolutely.
>>KOPELOW: I mean, this is your academic interest. This is what you
>>CERVERO: Yeah. No, I mean I'm still a professor in the department, I'm still actively involved in research. This has given me an opportunity to go back into the literature and see what's happened over the last decade which is quite a bit. So, and, of course, I've been involved at the Medical School now for five years. So this, it's all connected; this continuum of medical education is real. And, so, for me, and it comes to me, I'm in the middle of that right now. This opportunity has been great.
>>KOPELOW: So this review that covers, is a refresh of an update of what's been reported since 2003, since the second report, encompasses, am I right? It encompasses meta-synthesis or analysis of 450 articles.
>>CERVERO: Yeah.
>>KOPELOW: All together. There's a lot of additions since.
>>CERVERO: Oh yeah, no, it's been, the period of time, so the first study went from '77 to '96. The second study was a shorter time frame, from '96 through 2003. This is a longer time frame than the previous one, because I think the first article in this review is from 2002. So it covers 2002 and, in fact, there's a 2014 article, the one on problem-based learning, the systematic review on problem-based learning. So it covers, I think, a 12-year period.
>>KOPELOW: Right, right.
>>CERVERO: Of publications. And we only looked the publications. We didn't go outside of the peer review literature.
>>KOPELOW: Right, right, right. And one of the observations that you made early on in the method section of this, is that you've made the observation that the sophistication of the studies has changed in this period of time, reflects something that you had thought, not so much just looking at the effectiveness of CME, is that right?
>>CERVERO: Yeah. So, there's a variety of ways of doing reviews of literature. And some of the early ones were vote counts, like how many studies show change. By the time we get to this review, Murray, I believe all of the systematic reviews only included randomized controlled trials or quasi-experimental designs. So the rigor, the methodological rigor, is much greater than it was in either of the previous two reviews.
>>KOPELOW: And an example of that rigor is Mansouri and Lakier's article of 2007 that talked about effect size. I mean they, as I understand it, that means that they looked at the data from the original articles and sort of pooled the data and re-calculated what the effects are of each of the interventions.
>>CERVERO: Right. So, I mean, many people would believe that statistical meta analysis may be the most sophisticated of the approaches to syntheses, and so that's what that article did. It pooled it together. Of course, it came to roughly the same conclusion. It still showed CME as effective. It still showed there are certain types of CME that are more effective than others.
>>KOPELOW: Let's explore that 'cause that's the essence of the issue here in the article, in your report on effectiveness.
>>CERVERO: Right.
>>KOPELOW: That it's effective, but it's effective at changing what? There's knowledge, there's performance, let's talk about that and how much it changes some of those things.
>>CERVERO: Right. So there's a continuum from changing knowledge through changing performance through the most difficult which is changing patient health outcomes. I think, probably, the AHRQ study, the Marinopoulos 2007 study, was probably the most rigorous in that sense that it looked across all of those outcomes, and it found that knowledge is easiest to change and that it changes most often in these studies. Patient health outcomes is the most difficult to change and changed least often.
>>KOPELOW: Okay. And it isn't just, these are rates of change. It's not just that there's more education on change of knowledge. It's the fact that, if you're doing an educational activity on change of knowledge, you are more likely to make a change, than if you are doing an educational intervention on performance. Right?
>>CERVERO: Sure. Yes.
>>KOPELOW: And it isn't just that most education is knowledge based and that's why? It's not what that is?
>>CERVERO: No, no, no, not at all.
>>KOPELOW: All right, and the numbers are, they seem to sort of be consistent where 80%, 75-80 %, if you're changing knowledge, your success rate is about 75 or 80 %.
>>CERVERO: That's right.
>>KOPELOW: 50 or 60% if you're trying to change performance and, maybe, 40%. So, let's talk about that. I mean, that's, what would cause the variation in success rate?
>>CERVERO: Right. So, many of these articles will point to this but, certainly, knowledge is easiest to change because, generally, you're giving the physician a pre and post test. You're just changing what they know. They've done this their whole life. They've been successful at education. They can listen to material and report that they know something that they didn't know before. So, it doesn't require anything other than the individual physician to know something different. As you move progressively through this continuum, you see that changing physician performance is a social act, that you have many other, many other things have to align with whatever is being taught in the CME program for that practice change to occur. And then, of course, by the time you get to patient health outcomes, you have many other moderators to why patients may change their behavior. Or there might be two different patient health outcomes beyond just what the individual physician can do. So there's many other moderators in the system. So, I mean, it's pretty clear that's what's going on here. I don't think anyone would doubt that. It's just more difficult to pull all the levers as you go up the double line.
>>KOPELOW: Now our colleagues in hospital accreditation at the joint commission, some of the people there have published articles to talk about the fact, the mechanism for failure to change health outcomes, they use the term intervening variable. That there are a lot of things between changing what the doctor or the nurse is going to do and changing health. They're talking the same principle really.
>>CERVERO: Absolutely, there is a lot of intervening variables, it's not just the, I mean patient health outcomes, obviously the physician, you have the whole care team, you have the financial incentives, you have everything we know that impacts what happens to patients, and of course, education is a critical asset to you if you're trying to make those change, but it can't possibly be the only thing. That's just not how the world is, operates.
>>KOPELOW: Right, now to add to this complexity I mean that the fact that if you are trying to change performance or outcomes that you need to deal with the other things that are in the person's life and where they work, but also in your report you talk and you alluded to it already, about the form of intervention as you're trying to get further along away from what people know, that you talked about interactive methods and audit feedback detailing that those turn out to be the most effective. That's not just your personal observation; the literature says that?
>>CERVERO: No no. That's absolutely what's in the literature. I mean the closer you get to practice, the more embedded the education can be in practice more likely it is you are going to make those, you are going to change physician performance. Trying to change physician performance just with knowledge is like trying to cross an intersection with your eyes closed. You might get to the other side, but there's so many other moving cars coming across that street that's going to make a difference in whether you make it to the other side and that's the same thing with CME and impact on practice.
>>KOPELOW: There's an analogy what you said about getting close to practice a simulation. I mean the pilots have been doing this for decades with little wooden simulators, with little wooden sticks making the people to all these sophisticated ones now when can't tell if they are in reality or not. It's the same issue.
>>CERVERO: It is, and I mean I don't think this is any brilliant insight. I mean anyone who knows how medical care is delivered would understand that. I mean we all understand the world is complex to ask education or knowledge to have an impact independent of all of those other things simply is not realistic.
>>KOPELOW: Since, well, the early 2000s, 2005, 2006, 2007, 2008 we at the ACCME having been talking about continuing medical education as a strategic asset to change, and this is really evidence in support of that concept that CME is one of the tools. If you turn it around and make this CEO-centric or healthcare academic medical center centric, who is trying to improve health. What this evidence is, is that continuing medical education is one of the strategic assets that they can use in making that change, is that right?
>>CERVERO: I totally agree and would say when you are not doing that, it's a huge missed opportunity, huge missed opportunity. If you are a CEO and you are not using, because you already putting, there is effort going on in your hospital, right now. So the question is only is it aligned with the kinds of change that needs to occur to improve patient care and physician performance. I mean really that's, you need to take advantage of that opportunity.
>>KOPELOW: So if we are trying to build an answer to this statement of ‘CME is not effective.’ I mean we now know that we can say the literature doesn't say that. We can say there are 39 systematic reviews or at least that demonstrate the effectiveness, there are systematic reviews that describe the kinds of CME that are associated with changes of performance, and if you are at a place of changing performance, this is the kind of education that you should be using. And so maybe now just like the sophistication of the evaluation of the evidence has rised, maybe the answer to this statement can go beyond, though the literature says, you go beyond that and say, maybe in your hands it's not effective because you are not using the right stuff in the right place at the right time. And that if you work with continuing medical education, you might see and realize the results that the literature says are available to you.
>>CERVERO: Yes, that's exactly right that leadership, you need to take leadership on this, you need to use this asset, don't waste this asset. It can be hugely important to your business, to the bottom line, to patient care. Lots of reasons to make this part of your repertoire.
>>KOPELOW: Because we've talked about what the literature shows that CME is effective. We've talked about the context matters, how people utilize and take advantage of CME as a strategic asset. But we've got sort of a quandary that we've got to reconcile and it's that, well we at the ACCME, it's important, is that there's a series of national reports that talk about ‘change is needed’ that seem to fly in the face of what you've discovered. The literature's discovered that continuing medical education is effective, but we've got these series of reports that say, they talk about the literature and then they talk about change. Did you identify sort of a paradoxical thing in these reports?
>>CERVERO: Right, so the Macy Report, the 2008 Macy Report had a couple of excellent chapters in there by Don Moore, one by Don Moore one by Dave Davis and a co-author, that reviewed this literature, and I essentially came to the same conclusion that our three reports have come to, which is CME is effective. And I think that Don said in his, right in the first paragraph he said, ‘We know how to design effective CME. We have evidence for that. It's clear that if you do these, if you integrate these kind of principals into your CME program, that it will make a difference in performance and patient care.’ So, and likewise, in the 2010 IOM Report, excellent chapter on the scientific basis of continuing education. Reviewed the same material, came to roughly the same conclusion. So the evidence
>>KOPELOW: Same conclusion about the evidence?
>>CERVERO: About the evidence, that it is effective. And I think, that we did in our reports. So, I think paradoxically, even though the material in those two reports was consistent with our findings, I would say the overall tenor and some of the quotes from the overall report of the conclusions were more, I would say more critical of CME, than the evidence would suggest because the need to reform CME, to move it into certain direction that's more practice-based. And, while I think we would all agree with that, that's not consistent with the evidence that shows CME is effective.
>>KOPELOW: So, let me read you a couple of things from your report that were quotes, some articles. ‘CE as currently practiced, does not focus adequately on improving clinician performance and patient health. There is too much emphasis on lectures and too little emphasis on helping health professionals enhance their competence and performance in their daily practice.’
>>CERVERO: Yeah.
>>KOPELOW: Does the literature say that?
>>CERVERO: I mean that's not what, that's a completely separate question. Maybe there is too much. Who knows? But what we know is that CME, when it's delivered, does make a difference in physician practice and patient outcome.
>>KOPELOW: But one of the things that, well, it's another one.
>>CERVERO: It's almost like orthogonal to what the evidence is about. You could have that opinion, but that's trying to describe the enterprise as opposed to trying to make a claim that's based on evidence.
>>KOPELOW: And then another one. ‘However the CE system, as it's structured today, is so deeply flawed that it cannot properly support the development of health professionals.’ I mean, your literature review shows that people change and learn in the continuing medical education enterprise as from 1977.
>>CERVERO: Yeah.
>>KOPELOW: To 2014.
>>CERVERO: Right.
>>KOPELOW: So this system that is described in these opinions is not supported by the evidence?
>>CERVERO: No.
>>KOPELOW: There's a disconnect.
>>CERVERO: There is a disconnect. So, I wouldn't want to hypothesize how that came to be, but that's a different, that's not, that statement is not based on the evidence that is in our reports. And certainly not in the reports, the chapters, the evidence chapters that were in those two national reports.
>>KOPELOW: I mean one of the issues is when you think about how sometimes these national reports are created, they're based and founded on what you described, like Don Moore's contributions. They find the important information that the reviewers need. And they establish a database for them. So they come forward with what the literature says, what the evidence is, what the descriptions of the enterprise are. And then other people write the reports and recommendations. So it might be sort of a systematic flaw in the structure of the creation of these reports that aren't peer reviewed. And in the articles, like in JAMA and the New England Journal, the ones that you've made your career trying to publish. Your conclusions are based on the evidence that the data that you generate. And the editorial staff look to say, ‘Are the conclusions and recommendations supported by the data?’ In these processes that isn't necessarily the case.
>>CERVERO: No.
>>KOPELOW: Because there's no editorial practice.
>>CERVERO: Right. I mean I'm sympathetic in this way, Murray. As you know, because you run an organization, you're always trying to get better, so we all want CME to be a better enterprise. We all want it to have a greater impact on physician performance and certainly patient health outcomes. But I think if you were to base your recommendations on the evidence, you would say, ‘We know how to improve. If we're going to make it better, we actually know how to improve it, because we have evidence-based principals that suggest if more providers use these principals on a regular basis, and certainly, on this direction that we've just talked about with embedding CME as a strategic asset, we know that to be true from the evidence.’ So if I were writing the reports, that's probably the direction I would've gone in, which is say, this is not a knowledge problem. We just have to help providers understand how to do it better, and do it more often. And you have a lever here in accreditation to build those into the accreditation process. So, that's where I would've gone to try to make the CME enterprise stronger. Which we all want to do.
>>KOPELOW: Correct. And in 2006, when we re-wrote our criteria, we did that. It was informed by your evidence and the evidence that existed. We weaved into them something you've been saying since I met you 30 years ago, ‘Put your money in needs assessment and you'll get good education.’ And we linked needs to practice when we said in our criteria that the providers need to base their education on the needs that underlie professional practice gaps. So that's a tight link to practice. We said in the literature, in our requirements, that you need to choose the format for the education that's appropriate to what you're trying to accomplish. Again, supported by the literature, right? Everything shouldn't be a simulation. If you're trying to change what people know about pathophysiology, you don't put them in a simulator, unless the simulator's talking to you about what's the pathophysiology. So that makes sense. Those requirements are evidence-based. And that like you did in evaluating the success of education, we ask our accredited providers to evaluate their success in order to learn. I mean, those are three evidence-based requirements. Would you agree?
>>CERVERO: I think you and I have talked about this over the years. I mean, I think all of these principles, all of these requirements that are promulgated through ACCME are important. If I had to pick one, I would go with needs assessment.
>>KOPELOW: Yeah, you always have.
>>CERVERO: Because, and the reason is, is it's practice-based. If you don't start off in the right place, you're not going to end up in the right place. So if you start off in practice, you have to certainly align the methods to achieve those aims, but if you don't start there, then you're never going to get to a place that improves practice.
>>KOPELOW: But my wife Cathy says, ‘If you don't know where you're going, that's exactly where you'll get.’ Right?
>>CERVERO: Right. I think active learning is really, really important. All things being equal, I would definitely go in that direction, but lectures can change. If you have the right need, and you have the right strategic alignment within the organization, people will sit down and listen to a lecture and understand what they need to do, and do it, and have all the systems supporting that, so
>>KOPELOW: Context matters. The environment and the place of where you do it.
>>CERVERO: Yes, absolutely.
>>KOPELOW: Right. Now the three conclusions are; CME does improve physician performance in patient health outcomes. The right thing in the right place in the right context, it'll do it. CME has more reliably positive impact on physician performance than on patient health outcomes, and that's this sort of cascade. It's just more likely and there are more intervening variables. It doesn't mean it can't. It just means that if it's going to, you have to control some of these other things. We in our accreditation requirements have tried to get the providers to address that, about inserting containing medical education in systems that are designed to change performance. Getting involved with quality improvement, collaborating with others, understanding the barriers to physician change that exists, what are the factors associated with a current state? Those are the kind of things you're talking about, right?
>>CERVERO: Exactly, and the only thing I would change in your description is I wouldn't say 'control', I would say 'integrate.' You want to embed education in those systems of change, and that's when it's going to be the most effective.
>>KOPELOW: And that's why CME leads to greater improvement of physician performance and patient health, if it is more interactive, uses more methods, involves multiple exposures, is longer, and is focused on outcomes that are considered important by the physician, right?
>>CERVERO: Right, that's it.
>>KOPELOW: That's the constellation.
>>CERVERO: That's the constellation of the research that's been done. I think what we, the next frontier is understanding more about the organizational and social context in which CME operates. And we have some literature on that but I think that's where we really need to go next. How does the CME get integrated into these systems.
>>KOPELOW: Right. It's sort of the ‘Where's Waldo’ phenomenon that I described when we have the learner, and we help her, and we educate her, and we change her performance and her abilities, and put all this knowledge into action, and then she enters into the workplace. And we really need to be able to track her contribution as she weaves her way through her day and her week and her month to improving health, and changing what other people do, and what the patients do. And I think that's the fascinating place, is the changed and enabled professional, how does that changed and enabled professional impact on healthcare in the future? And I think that's an interesting question for our providers and the researchers. That's the mechanism of change that you're talking about.
>>CERVERO: Right, Murray. Underlying all of this is the concept of physician practice as a social act. It is not an individual act. And so, the big change, I've been working with our medical school, the big change going from the second year to the third year on the clerkships is now they're no longer in a classroom learning material, they're in the social context of the hospital starting to learn to be physicians. And so there's a whole set of skills, inter-personal skills, communication skills, system-based skills, I mean it's so interesting to watch the physicians on these clerkships and sit down with the students and say, ‘You need to understand how insurance works if you're going to be able to provide good care for this patient.’ And that is, that was never covered, of course, in the medical school. That's not a clinical content. And so you need to understand, in order to deliver good care, you need to understand the insurance system, you need to understand the family systems that these patients are embedded in. And CME lives in that world. It's part of that world.
>>KOPELOW: As a last word, let me leave you and our providers with an analogy to that, at the larger level that we've been talking to the continuing medical education providers about, is that the clinical care only contributes about 20% of the variance in health to the people. And the other factors that are associated with health are analogous and parallel to what you're talking about. How the family works, the behavior of the patient in their community, the physical environment in which they live. Those are the things that continuing medical education can also intervene in to change the health of people; not just through changing how the physicians practice.
>>KOPELOW: Thank you, Ron. Thank you.
>>CERVERO: Thanks.
[music]
This is a transcript of Discussion of ACCME 2014 Study on the Effectiveness of CME - http://www.accme.org/education-and-support/video/interview/accme-2014-study-effectiveness-cme
© 2014 Accreditation Council for Continuing Medical Education; all rights reserved. For noncommercial educational use only. For permission to reproduce and/or distribute for other purposes, please contact info@accme.org.
