This article describes implementation of the Creighton Simulation Evaluation Instrument (C-SEI) developed by Todd, Manz, Hawkins, Parsons, and Hercinger (2008). In this study, student performance during a simulation scenario was rated by both a faculty member and a group of student peer evaluators. Students participated in simulation in lieu of one full clinical day during their community health clinical course; these students were in their final semester of a baccalaureate nursing program at a public university in the southeastern United States.
The study was guided by a conceptual framework proposed by Adamson and Kardong-Edgren to address the complexities of evaluating student performance during simulated clinical learning experiences. Adamson and Kardong-Edgren identified the need for valid and reliable instruments to support evidence-based teaching practices.
LITERATURE REVIEW
A recent meta-analysis of 20 studies exploring the impact of simulation on student skill improvement found that effect sizes were greater in studies that used performance-based evaluation rather than self-assessment, particularly related to psychomotor and affective domains (Shin, Park, & Kim, 2015). Several studies have described peer evaluation as highly valued by students (Cushing, Abbott, Lothian, Hall, & Westwood, 2011; Kim-Godwin et al., 2013). With the use of peer assessment during formative simulation sessions, first-year nursing students benefited from vicarious learning and developed reflective skills while also gaining skills in providing constructive feedback (Rush, Firth, Burke, & Marks-Maran, 2012).
Previous research on student and faculty scoring of performance during simulation found little differences on scores between faculty and student raters (Kakar et al., 2013; Moineau, Power, Pion, Wood, & Humphrey-Murto, 2011). However, a review of the literature found limited studies examining interrater reliability between peer and faculty raters during simulation in nursing education.
METHOD
This study was reviewed and approved by the university institutional review board before any data collection activities took place. The selection of the C-SEI for use in this study was based on its alignment with specific intents and purposes of the simulation, prior reports of established validity and reliability, and the ability to tailor the assessment of student performance of specific behaviors relevant to the simulation experience.
The C-SEI was modified for use in the National Council of State Boards of Nursing study on the use of high-fidelity simulation in lieu of clinical rotations and student learning (Hayden, Keegan, Kardong-Edgren, & Smiley, 2014). Each component area within the C-SEI (assessment, critical thinking, technical skills, and communication) includes between two and six specific behaviors. In order to obtain a point for the area being assessed, students need to exhibit a minimum of two of the identified behaviors, with a maximum score of 22 points.
Students engaged in the simulation in groups of two to three students. One faculty member observed and scored student performance through a control room with a one-way window. A peer group of four to five students concurrently watched the simulation through a live-feed recording in a separate location and scored students using the C-SEI.
A separate faculty facilitator provided a 30-minute briefing to the peer reviewers on use of the tool prior to the simulation. The faculty facilitator also provided clarification for scoring following the simulation and prior to the peer reviewers providing feedback to their peers during the debriefing session.
RESULTS
A total of 48 groups of students participated in the simulation. An independent t-test was use to examine differences between faculty and peer raters in mean total scores on the C-SEI and subscales. Faculty members were found to score student groups higher than the peer evaluators; however, the differences were not found to be statistically significant.
Cronbach's alpha found reliability of the C-SEI to be acceptable (.776). Interrater reliability was examined using Pearson's r and a two-way mixed model average measures intraclass correlation coefficient (ICC) for each subscale and the total score. Average measures ICCs (3, 1) were calculated for each subscale and the total score.
Low consistency was found on the total C-SEI score. The lowest ICC was on the Critical Thinking subscale; however, ICC coefficients for the other subscales were fairly consistent. See Table 1 for Pearson's r coefficients and interrater reliability analysis findings. According to Halgren (2012), ICC values between .40 and .59 indicate fair consistency between raters; those below .40 are indicative of poor consistency of scores between raters.
DISCUSSION
Faculty raters in this study scored student performance higher than peers; however, no significant differences on mean scores were found between types of raters on any of the subscales or the total score on the C-SEI. These findings differ from prior studies, which found faculty ratings to be lower than peer ratings (Jensen, 2013).
Interrater reliability was found to be low to fair depending on the subscale. ICC scores were particularly low on the Critical Thinking subscale. Low agreement between raters may be a result of a number of factors, including familiarity among faculty scorers with the instrument and the ability to identify evidence of critical thinking being displayed by students during the simulation. Peer reviewers, who were also students, may have lacked sufficient skills to identify evidence of critical thinking.
Jensen (2013) compared nursing student self-assessment and faculty ratings of performance during simulation using the Lasater Clinical Judgement Rubric. The Jensen study found significant differences among raters related to "making sense of data" as part of the Interpreting subscale and the Responding subscale, both of which involve use of critical thinking.
CONCLUSION
This study has a number of limitations, including a small and homogenous sample and a potential scoring bias by peer and/or faculty raters. As reliability findings are sample specific, findings from this study have limited generalizability. Additional studies should include a larger and more diverse sample and should consider including enhanced training for peer reviewers on use of the C-SEI.
Despite these limitations, this study provides new knowledge about interrater reliability between student and faculty raters of student performance during simulation. Student evaluators may lack sufficient skills to determine whether peers are using critical thinking while being engaged in a complex scenario, such as the home visit scenario used in this study. Evidence from the literature suggests that students may realize some benefit from serving as a peer evaluator through vicarious learning (Rush et al., 2012; Secomb, 2008).
Nurse educators should continue to examine the use of peer evaluation during simulation for its utility in enhancing learning. However, findings in this study indicate that there are limitations of using peer evaluators to evaluate student performance, especially related to demonstration of critical thinking skills.
REFERENCES