Undergraduate nursing programs across the United States have multiple goals that they set forth to achieve prior to the students' graduation. One priority goal consistent across programs is to provide high-quality education and experiences to produce safe, competent, professional nurses. The variability becomes apparent in how nursing faculty evaluate students' competency. To decrease this variability, the use of simulation has become more prevalent in nursing education.
Simulation is supported by National League for Nursing and the National Council of State Boards of Nursing (NCSBN).1,2 The NCSBN has taken a leadership position to promote patient safety and protection of the general public by setting high standards for nursing education with the NCLEX-RN. While passing the NCLEX-RN is the final determination after graduation for entry into practice, NCLEX-RN evaluates only the cognitive domain of learning, whereas simulation has the ability to be used for evaluation of cognitive, psychomotor, and affective domains of learning. Assessments strategically placed throughout the nursing curriculum can assist the faculty in determining if students are developing competencies as they progress through the program and if they are safe and competent prior to graduation.
The Institute of Medicine declared 5 core competencies essential for healthcare providers in the 21st century. The core competencies are to provide patient-centered care, work in interdisciplinary teams, use evidence-based practice, apply quality improvement, and use informatics.3 The Quality and Safety Education for Nurses (QSEN) project funded by the Robert Wood Johnson Foundation set forth the goal to prepare nursing students with these 5 core competencies and a sixth competency of safety. By threading the 6 core competencies into an undergraduate nursing curriculum, the educational foundation for nurses would be strengthened, preparing graduates for contributing to higher-quality healthcare for the nation.4
One must be careful, however, not to assume that because a core competency is threaded through the curriculum it was met by the student; therefore, students should be periodically evaluated on their achievement of these competencies. The use of simulation for high-stakes evaluation of nursing students may be a way to accomplish this. High-stakes evaluation is defined as an evaluation process associated with a simulation activity and that has a major consequence or is the basis for a major grading decision, including pass-fail implications.5 High stakes refer to the outcome or consequences of the process.5
High-fidelity simulation was used for summative evaluation of approximately 100 nursing students in their last semester of our large Midwestern university's baccalaureate nursing program. While the majority of the class demonstrated successful competence, the high-stakes evaluation assisted faculty not only in identifying students who needed remediation but also in developing individualized learning plans for students who were unable to demonstrate proficiency. An unexpected finding of the high-stakes evaluation highlighted areas in the curriculum that needed revision.
High-Stakes Evaluation Using Simulation
A transformation in nursing education has been occurring over the past several years because of limited clinical experiences, a decrease in the opportunity for students to have hands-on experiences, record high numbers of student enrollment into nursing programs, and too few nursing faculty.6 One essential component in the transformation is the addition of simulation into the nursing curricula. Simulation offers students a safe learning environment during the teaching/learning process and can also provide a standardized environment for more formal testing. Simulation-based methods, such as high-fidelity simulation, have shown great promise as an evaluation tool of healthcare professionals.7 Faculty can evaluate students' competencies and their level of performance on the continuum of becoming a safe, competent professional nurse.
Limited information can be found regarding high-stakes evaluation using simulation in the discipline of nursing, although the National League for Nursing (NLN) provides high-stakes testing considerations and recommendations. The NLN suggests that requiring a passing score on a standardized examination for students to achieve prior to the end of a program could have negative effects on students.8 It is recommended that students are provided with the opportunity to take the examination until they are successful, communication of the high-stakes evaluation is appropriately planned and communicated with students, and should include multiple sources of evidence for evaluation.8
Medicine has been incorporating assessment of competency using various types of simulation of medical students and residents for the past 35 years.9 Over the course of the last decade, medical educators have attributed greater weight to clinical simulation evaluations. Simulations for high-stakes evaluations are used primarily in medical education for licensure and specialty certification for physicians. Computer simulations were added as a component of the US Medical Licensing Examination in 1999. The Educational Commission for Foreign Medical Graduates and the National Board of Osteopathic Medical Examiners quickly followed suit by incorporating multistation objective structured clinical examinations in addition to an academic examination.7
In theory, simulation evaluations allow the faculty to assess an individual's level of knowledge and critical thinking, which makes simulation appropriate for high-stakes evaluation with caution. A high-stakes evaluation may determine an individual's level of competence to provide nursing care at the bedside. Because of the consequences of making such a determination for an individual, it is essential to establish reliability and validity of the assessment prior to administering it. Medicine has demonstrated that high-stakes evaluation can be successful, but it is a daunting process to implement and one that should not be underestimated.
High-Stakes Evaluation: Rationale, Objectives, and Procedure
After a few years of implementation of simulation experiences, perceived prior success, and excitement with this teaching strategy, faculty launched into uncharted territory. Despite the lack of evidence in the nursing literature regarding high-stakes evaluation using simulation and in true pioneering spirit, our nursing administrators and faculty sought to document students' competencies. The rationale for this high-stakes evaluation was to assess the student's competencies at the end of the program, determine student's ability to apply the QSEN competencies, and identify areas for student remediation.
The objectives for the evaluation were based on the program outcomes and 4 QSEN competencies (patient-centered care, teamwork/collaboration, evidence-based practice, and safety). An expert panel of nursing faculty developed the assessment tool from these objectives. The tool consisted of 6 behaviors that evaluators observed during the simulation experience; anecdotal comments were recorded to support the evaluator's decision of the student's performance (Table 1). A priori decision was made that any student who did not demonstrate competency on any of the first 5 items on the tool would need to participate in remediation.
The procedure for the high-stakes evaluation began with the faculty selecting an evidence-based scenario on an elderly male patient experiencing sepsis, septic shock, and multiorgan dysfunction syndrome developed by Medical Education Technologies Inc (METI).10 This scenario provided the senior nursing students numerous opportunities to demonstrate the program and selected QSEN competencies. Students were sent a learner guide that accompanied the scenario 2 weeks prior to the scheduled simulation experience. The guide included preparatory questions, references, the patient history, and the initial healthcare provider's orders. The students were instructed to work and collaborate as a team. No specific roles were assigned for each student; therefore, students self-selected their own role as they began organizing the patient care.
The senior nursing students were divided into 20 groups with 5 students per group. Observation of smaller groups provided the faculty with a reasonable number of students to assess. Each group was allotted 90 minutes for the simulation experience followed by an individual consultation with nursing faculty to discuss student performance. All groups were assessed over a 2-week period.
Three faculty observed the simulation through a 1-way mirror; the faculty were not present in the simulation bay. The same 3 faculty were present for all of the simulations, which provided consistency in the evaluation process. Additionally, each simulation was video recorded, which provided the opportunity for faculty to review as evidence for interrater reliability of the faculty. The faculty did not communicate with each other during the evaluation process. Faculty were directed to observe the student's ability to meet the 6 behaviors on the tool and document their findings and evidence to support their final student recommendation. After the simulation experience, discussion among the 3 nursing faculty evaluators occurred to determine if the student met the objectives or needed remediation.
Student Evaluation Process
The nursing faculty dedicated much time and effort to the student evaluation process to ensure fairness to all students. Video recordings were reviewed as needed to validate the assessments and anecdotal notes as well as confirm faculty agreement, which provided support for reliability. After faculty consensus was reached, the students received individual feedback with one of the nursing faculty. At that time, the students received an immediate response to whether they were successful in the simulation or required remediation. The evaluation contained detailed documentation of the student's performance during the simulation experience. When a student did not meet the objectives, the student was asked to repeat the simulation. If the student was not successful in meeting the objectives at the second attempt, the student was required to participate in remediation.
Each student was allowed to demonstrate proficiency of the competencies in their own way. Some students verbalized what they were doing or anticipated doing, which gave the faculty explicit information on their thought processes. Others collaborated with a colleague regarding appropriate interventions. Many students demonstrated their ability to organize data in the form of SBAR (situation-background-assessment-recommendation) reporting to the healthcare provider.
The assessment was also exciting in some cases, where the "light bulb moment" was achieved by the student leading to a successful performance. Observing students assisting each other and collaboratively developing a plan of care was positive, and the nursing faculty as well as students celebrated these successes during the individual consultations.
High-Stakes Evaluation Outcomes
After analyzing the evaluation data, 4 repeating concepts emerged from the students who required remediation. The students lacked the ability to perform a complete physical assessment relevant to the scenario in the simulation experience. These students struggled with evidence-based practice competency by demonstrating improper psychomotor skills such as insertion of an intravenous catheter, indwelling catheter insertion, oral suctioning, and specimen collection. Safety competencies were violated by the students as evidenced by medication errors, incorrect drug dosage calculations, improper identification of the patient prior to drug administration, and lack of universal precautions. Students also did not take the initiative to engage in teamwork or collaboration.
The most incredible finding concerned the students' grade point average (GPA). Some students with high GPAs did not perform well in the simulation and therefore required remediation. These findings led the undergraduate nursing faculty to conduct an extensive evaluation of the undergraduate curriculum. Investigation into some of the current curricular issues revealed that because of healthcare institutional policy changes and restrictions, students were not given adequate opportunities for medication administration and psychomotor skill experiences that had been occurring in the past. Additional simulation experiences could remedy this situation. Complete integration of simulation throughout the curriculum had not occurred at this point.
Remediation Process
The remediation process was mandatory for approximately 25% of the senior class. All remediation was conducted in 4-hour blocks of time in a 2-week period. The sessions were deliberately kept small to facilitate student participation, allow hands-on experience, and decrease stress for the students. The faculty content experts used current evidence-based practice to develop 4 remediation case studies using METI simulation scenarios. These were adopted for the student remediation sessions including: asthma, bowel obstruction, deep vein thrombosis, and anaphylactic reaction.
The strategy for the remediation sessions was to utilize static mannequins to decrease the students' anxiety, as well as avoid the use of the high-fidelity simulator for remediation. Four static mannequin case studies were planned for the purpose of application of knowledge, pathophysiology, anatomy and physiology, nursing process, skill interventions, medication administration, and evaluation.
Concept maps were also used to provide a visual representation of the concepts that were being taught. The most problematic nursing skills determined from the student evaluation data were incorporated into each of the remediation case studies to enhance the students' critical thinking and enable them to successfully achieve the evaluation outcomes. These areas included intravenous medication administration, intravenous insertion skills, medication dosage calculations, head-to-toe physical assessment, and documentation. Through discussion and hands-on approach to the case study, critical thinking skills were fostered, and students strengthened the areas in which they had not met the performance standards during the high-stakes evaluation.
Initially, the added course work was met with resistance, anger, and anxiety from the students. However, through skillful teaching, true desire on the faculty's part to help students succeed, and the use of humor and honesty, the students learned to apply their theoretical knowledge to evidence-based nursing interventions for their patients during the remediation case studies. Encouragement and support were key in assisting the students to meet the objectives. After the remediation process, all students successfully met the objectives for the high-stakes evaluation as well as earned a baccalaureate nursing degree.
Recommendations
The most important lesson learned was that students needed more experience and familiarity with simulation prior to high-stakes evaluation. The 2 previous simulations the students experienced in the curriculum were not sufficient for the students to feel comfortable with the capabilities of the mannequin. This is imperative to build their confidence in their own skills and allay anxiety through the evaluation process.
Results of this high-stakes evaluation indicated a need for curricular reform, which led to the integration of simulation into each clinical course. Each successive simulation throughout the curriculum was developed to build on the previous one with increased complexity and higher-level critical-thinking skills. To that end, 3 simulations were developed for the sophomore fundamentals course, which included physical assessment, psychomotor skills, dosage calculation, and the nursing process. For the junior-level students, a mental health simulation experience was added to the curriculum. Pediatric and community simulation experiences were added to the senior-level curriculum. In addition, the clinical facilities used by the school of nursing were reevaluated, and contracts renegotiated to allow a more hands-on approach versus observation to enhance student learning.
The next steps in simulation will include development of a second simulation into each clinical course throughout the curriculum to include interdisciplinary and multiple patient simulations. Although the methods used to assess the students were found to be reliable among the faculty, more attention to validity and reliability of the tool is critical. Based on our experience, high-stakes evaluation using simulation has the potential to revolutionize current practices in the use of simulation in nursing education.
Acknowledgment
The authors thank Marilyn Oermann, PhD, RN, FAAN, ANEF, for critiquing this article prior to its initial submission.
References