Hello, and welcome to the first of this four-part series on using examinations in education. I am Dr. Michael Reichert. I'm very happy to have you with us for this series. In the first part of this four-part series, we're going to talk about why we test. We're going to go through the process in the four parts about how to test, how to write exams, how to identify the material that goes on exams, and then some quality assurance about our exams. But we're going to start off with this very basic objective of saying, let's understand why we use assessment, how we use assessment in education, and how we can understand the goals of assessment, first of all, in order to then lay the foundation for understanding how we're going to put the exams together and how we're going to create them. I've had the great pleasure and the great honor of being a professional educator as well as a professional CRNA for many, many years, but spent the past 17 years as a program director and have led not only development of examinations and assessments within my own program, but also served on the NBCRNA exam committee and a variety of other committees with the NBCRNA, where I got a lot of experience with writing exams and evaluating exams and assessing how the exams are performing. And so I'm glad to bring some of that experience to all of you today. So the very basic question that we're going to answer here is just why do we test? What is the whole purpose? What are we looking for? And we are in a high-stakes profession here, right? When you think about if you are in a faculty role and you've got students that are under your charge, there's a real important reason that you're teaching them a lot of stuff, because there's some very high stakes involved with what they're going to be doing in clinical and as professional anesthetists. So obviously, you know, whereas students often see an exam as, you know, a hurdle that they need to get over, and we'll talk about that point here in a little bit, it really is a very important imperative for them to have the knowledge that is required for them to provide safe care, right? So obviously, that's the bottom line. But let's delve into it a little bit more. Of course, there's, first of all, the accountability for the knowledge, right? We are going to teach them something. We want to ensure that they take that knowledge away. And one thing that we know, if we look at psychological literature, we know that we as humans, and this is not any commentary on people's intention, no malintent assumed behavior, we don't do a great job of assessing our own knowledge. We as humans do not do a great job of saying, here's what I know very well, and here's what I don't know very well. Because if we did, if we were really good at that, we would just provide the classroom lectures or whatever activities we're going to provide, and we would just assume that the person has, you know, got everything that they need from that. But instead, we know that people don't do a great job of recognizing their own learning needs and their own shortcomings. And therefore, we need to do some assessment, and we're going to do some testing to see what they did learn and what they did not learn and what they're able to apply. In this article, which is a really great exploration of a lot of the different processes that create the processes in our brains that create this difficulty in self-assessment, there's a great quote in there that I love that said, on average, people believe that they are above average in a given skill, which obviously, if everyone's above average, well, then there really is no average because someone's got to be below average in order for there to be an average. And so we know that, you know, a really important piece of testing is that we don't do a great job of recognizing, here's what I know well and what I don't know well. So that part's easy. The second one is the opportunity to correct misunderstanding, and we're going to make a distinction here in a little bit. I'm going to talk about the difference between formative and summative assessments, where a formative assessment is really geared, really has the objective toward giving feedback and saying, let's check in. You might think of it as a check-in. Let's do a check-in on what you've learned so far, where you're at, how much of this you really understood and are able to apply before we move forward. And so, you know, it gives you the opportunity, and this can be done even with end-of-semester evaluations, which are more summative, as we'll talk about. Even with more summative assessments, it gives the student the opportunity to say, wow, I guess I really didn't know that as much as I thought, and therefore, hopefully, the assumption is that they're going to go on and try to kind of catch up on those topics, those concepts a little bit more. The last couple of points here, we don't tend to think about as much. And the first one of those two is assessing instructor performance. And so, it's easy sometimes as instructors to assume that we know everything, of course, and that we give the exam, and the students miss, you know, a question or two on the exam, and we say, well, that student didn't really get it. That was their fault. But as we'll talk about later on in this series, that we can use some statistics and some psychometrics as we look at the performance of our exams to say, maybe we did not do a good job, either A, of teaching, or B, of writing the exam question in an effective, efficient way. And that's really very important for us, because in order for the test to do what we think that they're going to do, which is to assess accountability for the knowledge, you know, ensure accountability, and that the students got the knowledge we wanted, we have to make sure that the test itself is valid and is telling us the information. And so, there's a difference between testing and assessment. And so, testing really is a test is a more straightforward sort of thing. I'm going to answer this question, or I'm going to weigh these two things to see which one weighs more. The term assessment, which people often use interchangeably with testing, really means much more broadly. You know, we're assessing, you know, and you can assess something through a variety of means. So, whereas we're giving tests to students, we also have a bit of assessment there for ourselves. We can also assess our own teaching and our own classroom methods, our own laboratory methods, our own whatever methods that we use to convey the knowledge and hopefully help the students to learn through the use of the test. So, the tests not only tell us what the student knows or doesn't know, but it also gives us a little bit of feedback. If we interpret the results the right way, it gives us the feedback on our own styles and our own performance. So, that's going to be an important part of it. And then the other one, the last one that people probably think about even less or the least is that testing actually enhances front-end learning. So, we tend to think of exams as only summative, like we're going to teach you the stuff, and then I'm going to give you the exam, and then I'm going to say whether you did well or not or learned the stuff or not. But the process of testing itself, really importantly, whether we're talking about a test in school or an NCE certification exam or a CPC exam, also helps us with our learning along the way. The process of studying for an exam, the process of preparing, helps us for that. And I'm going to digress a little bit from talking about testing because this topic is really important for all of us since we've got CPC exams to take, and it can help us individually, it can help our students, when we think about the teaching and learning process. And so, when we think about learning, I think there's an important distinction to make to recognize that the challenge, when we say that you're going to learn something, the challenge is not typically in the learning per se, the bringing it into your head, but the remembering part of it. And so, there was a really landmark publication written in the late 1800s by this gentleman, Hermann Ebbinghaus in Germany, and he wrote about this idea about what learning really means. And what he did is he gave people a series of random words to learn, to memorize, essentially. And then he assessed them. And so, at 30 minutes, he said, tell me how many of those words you remember. And again, at 60 minutes and so on and so on down various intervals of time. And what he found is that over time, the memory, the ability to remember these words dropped off pretty drastically, in fact. And so, then he kind of sliced and diced that a variety of ways, and he found, for example, as you see in the red lines there on the screen, that when people got some repetition, so they repetitively restudied something, you know, studied the same thing over and over again, that the line, the curve, the forgetting curve, as he called it, kind of flattened out. And so, but then the important point about this is that when we talk about learning, it's not really so much that the challenge is not getting the information in, it's finding the information once it's in there and getting it back out again. Because, you know, if I tell a student, this is a list of drugs that are amino amide local anesthetics, they get it. That's not difficult. There's really nothing inherently difficult to teach nurses that point. The really important point is that when they're in the clinical arena and now they have to remember which of those drugs falls in that class versus a different class, that's where the learning really takes place. And so, we misconstrue sometimes what learning really means. And so, you'll see things like this, you know, this YouTube video about this kid that, you know, recites the first 2,500 numbers of pi, you know, decimal numbers of pi. Is that a really important feat of learning or is it just kind of a mind trick, you know? Does that apply to anything at all? And so, you know, we want to think about that. When we think about what we're asking the student to do or the person that we're assessing, what we want them to do, we want it to be useful. We want it to serve a purpose, right? So if you think about, for example, there are a lot of things that are very easy to write test questions about, right? So I could write a test question and say to the student, please list the 16 families of the cytochrome P450 enzymes in human or please list the 300,000 cytochrome P450 enzymes that are recognized. And someone might or might not be able to do it. Probably most would not be able to do it. Or if you, you know, told people, hey, memorize these 16 words and then I'm going to ask you to write back down on an exam, they might be able to do it, but that doesn't really demonstrate that they're able to do anything good for patients, right, in patient care. And ultimately, when we think about why we're giving the test, going back to step number one there, is we want them to have accountability for knowledge that they're going to then apply very importantly, right? So maybe more important, if we think about that, if we construct an exam to lead back to that main objective of we want to ensure that the person not only has memorized words or memorized lists or things like that, but actually understands the stuff in a useful way. Now we're getting to maybe writing a question that sounds something like this. If a patient suddenly increases their consumption of grapefruit juice, which of these following medications effects will be exaggerated, right? That's a very clinically relevant thing. That's something that can really make a difference in patient care. So again, thinking back, starting off always with what's the purpose of the testing? What is the reason that I'm giving them the exam? Is going to lay a very, very low foundation, a very baseline foundation for how we're going to then construct the exam. We're going to talk about that later. I'll give you a great example of the proof that this sort of repetitive learning does nothing. And this kind of relates to repetitive learning as well as repetition and testing. If I had you look at these eight different renditions of the penny coin and asked you which of these is correct, you'll see that they all have a slightly different configuration. If I asked you which of these is correct, I'll give you a second to look at that here on the video. You might have a hard time, and most audiences will have a hard time typically when I've done this in live audiences. The majority of people will get the correct one, but there will be people that will have chosen just about every other incorrect penny. And if you think about how many times you have handled, used, conducted commerce with this coin, it'll really solidify the point to you that looking at something over and over again or being able to recite something over and over again really does not confer that you've learned it or understand it or can apply it very well. If I said, hey, have you ever seen a penny? Yeah, I've seen one millions of times. Great. I'm going to assign you to be a Secret Service agent and you determine which of these is counterfeits. Now, all of a sudden, that million times that you've handled a penny becomes really very useless, doesn't it? So we want to think about this when we think about testing, we want to make sure that we're testing on things that are useful, that are relevant, and things that tell us something that we want to know. When we're providing the test and testing is a part of a bigger assessment, which is has the student learned what they need to learn and will they be able to provide good care for patients? Then this gets to that question, you know, are we teaching them and are we testing them on trivial facts that are easy, you know, either to memorize or not memorize? Or are we really assessing that they have the right thought processes, the right skills in mind, right? Typically, historically, we learn things backwards. We teach things backwards and we assess things backwards. And what I mean by that is when you think about how you have probably yourself learned something in the past, you might have the book chapter and you might say, I'm going to read the chapter and I'm going to read the chapter again. And I have students say this to me all the time, I've read that chapter 12 times over, I don't know why I did not do better on the exam. Because when you're reading and repetitively taking something in, what you're practicing doing is putting it in, you're putting it in, putting it in. That's great. But you can read, here are all the signs and symptoms of malignant hyperthermia 50 times. But when the patient shows up in the OR, the patient does not say to you, I might have malignant hyperthermia, please recite all the signs and symptoms, right? The patient instead says, or demonstrates, I've got this muscle stiffness, my temperature is going up, my heart rate is going up. And now it's up to the student not to recite what they memorize as the list of things, but to say, now what was the difference between malignant hyperthermia and neuroleptic malignant syndrome and thyroid storm? Now the trick is not putting the information in, which generally is not really very difficult to do, it is pulling it back out. And pulling it out in the right context and the right framework, and importantly, making the right distinctions and differentiation between this is the subset of symptoms that looks like MH, this is the other subset that looks like neuroleptic malignant, this is the other subset that looks like thyrotoxicosis, and so on. So when we think about testing, we want to think of it as opportunities to learn as well, right? So in this case of this, we want to think about kind of changing our lens on how we think about the tests and assessments we're giving. Again, think about the assessment, the purpose that it serves. So this also helps to change the lens for the student, help the student to recognize, hey, this is not just a hurdle. And this concept about windows instead of doors, I took from a blog post by Shannon Dee at University of Waterloo there, that was a great quote, that she said, think about assessments not as a doorway that you have to get through, you know, typically we think of this as here's a threshold, here's a hurdle that you've got to get through, and once you get through this doorway, now you're on to the next thing. And instead of that, it's really helpful if we, both for ourselves as we put the exam together, but also for the students, to help them recognize, you know, this is just a window into what you've learned, a window in how well you've learned it, and a window that's going to help you to move on to the next thing. This isn't just a matter of like, I made it through this hurdle, and I'm going to forget about that and move on to the next thing. But rather, let the exam tell you what you know and what you don't know, how well you understood the concepts, and then we're going to use that to build, right, because so much of what we do builds on to the previous stuff. There are two basic types of assessments that we can do, formative versus summative assessments, with the primary difference being one of them is really made, intended to give that feedback along the way, and one of them is much more kind of intended to be that final little hurdle. In every case, especially in anesthesia education, you know, I feel like everything kind of builds on everything else. If you go to college, you know, you take that one art class elective in your nursing program, you're probably not going to need to build on that. You're probably not going to need, it's not going to be a big deal if you don't remember some of that stuff. But with anesthesia education, the students are always kind of building on the things. But we might change the format of the assessments a little bit when we're thinking about it as a formative versus summative. So a formative assessment might be, for example, something where we give a quiz before the class session to help the student get that window, get that little glimpse into what they already know as their baseline knowledge to help cue them into, hey, here's some things you're going to need to know. So, you know, as you see, the questions I asked you on the quiz is going to help you to think about what to pay more attention to in the class, because I'm kind of giving you a little cue here that things are going to be important out of this topic. And so with a formative assessment, the primary goal is to give feedback, to provide some feedback to the learner. Here's where you are at this point in your training, in the course, in the session, whatever it is. Usually they're low stakes. It might be a small percentage of the grade that is attributed to the formative assessments, or it might not be a grade at all. It might just be, it's either, you know, a complete or incomplete, you have to do it because it's going to help you to kind of think about what's important and cue in on some things that are important. With a formative assessment, the most important piece is that there's a feedback piece to it. Here are the things that you didn't get right, and here's why. Let's look at this and let's give you some feedback on that. With a summative assessment, it's much more of that sort of end of course, you know, typically end of course exam, final exam is a very traditional summative exam. There's not really an intention that we're going to go back and repeat, look over what the person got right or wrong and give them some chance to, hey, you need to really work on this piece. It's much more, okay, you've had the entire course, you've had the entire module, and now we're going to take the exam before we move on to the next group of things. Obviously there's less opportunity to remediate, but there might be. That's one thing that, as I said, with testing, you know, one of the really important pieces of it is to help the student recognize that it's not just an all or nothing thing. I passed it or I failed it, but rather let the exam, even if it is a summative end of module or end of course exam, help the student to recognize we want you to look at what you got right, what you got wrong, where you've gone awry, and then move forward from that and have the opportunity to remediate the stuff where you fell short. I had a student not very long ago who said, oh, I feel really badly, I missed eight questions on that exam. And so I met with a student and we looked over the exam and I said, wow, you know what, I'm noticing that five of those eight were all math-related problems. So I don't think that you really missed the whole point of this entire module. I think that you really just need some practice with these math questions and doing some of these calculations. So it's really great if you can sort of change your lens for the student as well and to help the student change their lens on the purpose of the exam and for them to recognize, well, I can learn from the exam as well as the fact that the exam is just trying to test, you know, as a kind of summative nature, whether I've gotten past this hurdle or not. So let's answer the question finally. Why do we test? So number one, ensure accountability of useful applicable knowledge. We want to ensure for our interstitial students that they have the knowledge that they need and particularly that it's not just the trivial knowledge, but it's the actual stuff that they're going to need to make decisions in patient care. We want to ensure understanding of foundational concepts, things that they're going to build upon as they move on, not only in their educational course, but also as professional practitioners as well. We want to provide the opportunity for them to demonstrate the application, not just the memorization, not just the fact that they can memorize a bunch of words or put things in a certain list, but actually demonstrate the application, the retrieval and the use, the differentiation, the differential decision-making that needs to be done in order to make decisions about patient care. And then the final thing is we'll think about testing as a window, not only into the student's learning, but into our own teaching effectiveness. And so, you know, it's a really important piece of testing that we'll talk about in a later module about quality assurance of testing is that we want to be able to look at the exam itself and to recognize when it is doing and not doing those other three things. In other words, one is it that our exam maybe is telling us the wrong information, is telling us that the student did not know this thing, but the fact of the matter is that we apply some statistics of looking at the exam performance to say, you know what, it probably wasn't that the student didn't get it, it's probably that our teaching methodology, our teaching effectiveness was confusing, or sometimes that the test question itself was confusing or misleading or led the person down the wrong path. And so a really important piece about testing also is it gives us the opportunity not only to assess the test itself, but maybe more broadly assess our teaching methods and our approaches and it can give us good feedback to make sure that we're putting the right kind of timeline, the right kind of attention, the right kind of framework around the activities that we do with the students to help them on this path of knowledge acquisition and skills and practice acquisition that they've come to us as experts to be able to give them. So that leaves a good foundation for us. We'll move on to talking about more practical aspects in the following modules about how to actually put the exam together and then how to ensure that the test is doing what we want it to do. I'm very glad that you're with us. So, to wrap up this first part of the conversation, why do we test? We want to ensure accountability, but accountability for useful, applicable knowledge. And so, when we talk about selecting testable material, we're going to keep that in mind. That's going to be a really important piece. We don't just want to say, I'm just going to write down a bunch of, you know, can you recite the, you know, 300,000 enzymes that do this or that. That's not really applicable and that's not really very useful for our students when we think about nurse anesthesia education. Secondly, we want to ensure understanding of foundational concepts and knowledge, right? So, we want to make sure that the students understand this drug fits in this class, this drug fits in that class, this drug has these side effects or whatever it might be. Here's how to recognize, you know, these various complications. We want to ensure that the students have the knowledge that they need to move forward. So, that's going to be an important piece when we select what concepts and what kind of material go on to the assessment. And then finally, provide the opportunity to demonstrate application, retrieval of the concepts, right? So, it's not just the important, the important piece is not just that they're able to hear the lecture, to read the book, to watch the videos or whatever it is, to get the information in. The getting it in is the easy part. It's the getting it out at the right time, in the right place, in the right application and making the right differentiation between numerous things that might be very similar. That's really the trick, right, so to speak. That's the trick that we're trying to get the person to do, mentally speaking. And when we give them an examination, we have the opportunity to assess their ability to do just that, to get the stuff out in the right format at the right time. So, that's a great first introduction to why we test them. If you'll stay with us for the next piece, we're going to talk about how we then determine what is the material that we need to assess and how do we select the stuff that's going to be important to put onto that examination. Thanks for being with us. Bye.

education

testing

formative assessments

summative assessments

feedback

learning effectiveness