In Part 1 of this article (Frith, 2020), I describe the classic assessment framework of online education by Billings (2000) and present assessment methods for technology, support for faculty and students, and educational practices. This article uses the Billings framework to discuss the assessment of outcomes in online education. Standards from the Accreditation Commission for Education in Nursing (2017) and the Commission on Collegiate Nursing Education (2018) require assessment of outcomes. Moreover, engaging faculty in improvement cycles is vital to having robust outcomes in online nursing programs.
ASSESSMENT OF LEARNING
Faculty can improve online student learning outcomes using rigorous assessment plans that specify aims, milestones, and outcomes. For prelicensure programs, results from the National Council of State Boards of Nursing licensure exams (NCLEX(R)-RN and NCLEX(R)-PN) are programmatic measures of student learning outcomes. In a study of NCLEX outcomes from nearly 2,300 nursing programs, of which 23 percent were online, Spector et al. (2020) found hybrid programs were more likely to have pass rates higher than face-to-face or online programs. This finding raises interesting questions about teaching and learning strategies in prelicensure programs. Reports showing the program's performance and percentile rank in all categories of the NCLEX blueprint provide another programmatic assessment strategy. These data, along with results from end-of-program standardized examinations, can provide programs a wealth of information for building an improvement plan.
In online graduate programs, faculty track certification rates for programs preparing advanced practice nurses (APNs). Results from APN certification preparation courses could also be a source of data for understanding strengths and weaknesses in student learning outcomes. Presentations/publications are typical student learning outcomes in other master's or doctoral programs. Faculty can add fields in a student-tracking database to note these scholarly contributions; these data can be aggregated to understand learning in these writing intense degrees.
At the course level, results from teacher-made examinations and simulations blueprinted to NCLEX categories or APN core competencies, along with standardized examination results, provide data for assessment of learning outcomes. Results can be aggregated longitudinally by course or by cohort to understand the effects of curricular changes, new teaching strategies, or educational technologies. Formative assessments gathered in online courses help faculty understand where students grasp concepts or struggle to understand them. These assessments can automatically offer additional instruction for struggling students while letting students who grasp the concepts move forward. Over time, these formative assessments yield rich data to use in planning future course offerings with strategies to address commonly misunderstood concepts.
ASSESSMENT OF RECRUITMENT, RETENTION, COMPLETION
The student life cycle for enrollment management includes marketing, recruitment, retention, and graduation/completion. Faculty can collaborate with college or university administrators to gather data about online programs and use the data for strategic purposes. For example, a nursing program whose strategic goal is to increase the diversity of its online student body could use data to plan marketing and recruitment activities tailored to underrepresented groups. The program can also use college or university data to track withdrawals and failing grades in prerequisite courses. Once data are aggregated, faculty can work with colleagues in other academic departments to brainstorm ways to support students in these required courses.
ASSESSMENT OF ACCESS AND CONVENIENCE
The Billings (2000) framework identifies access as an outcome of online programs. Access to programs because of geographical distance has generally improved because of online education. However, the abrupt changes brought about by the coronavirus pandemic have highlighted a major barrier: lack of access to high-speed Internet in rural and low-income areas. This problem was exacerbated by the sudden use of synchronous videoconferencing for classroom and clinical instruction. There is a need to assess student access to high-speed Internet for all online learners so appropriate technologies can be planned. In addition, data from learning management systems can uncover the distribution of access time across courses to better understand the needs of students in online programs.
Before the pandemic, conventional wisdom said students select online courses and degree programs for convenience. Students can work and care for their families because of the flexibility of online education. However, students are often surprised about the number of study hours needed to succeed. Billings (2000) identified the need for faculty to assess convenience and to determine what it means to students so that educators can appropriately prepare. Faculty can survey students at the start and end of an online course using Likert-type statements, such as: 1) I can learn/learned at my own pace. 2) I can work/worked on assignments at my own convenience. 3) I can balance/balanced my work, home, and school life. 4) I will take another online course because of convenience.
ASSESSMENT OF CONNECTEDNESS
Student engagement is a cornerstone of online education; otherwise, students should just read the textbook! To assess connectedness, faculty can use the Online Student Engagement Scale (Dixson, 2015). This validated scale has 19 items with four factors: skills, emotion, participation, and performance. Dixson suggested the scale be used for research on course design; instructor feedback about student engagement with teaching strategies; and the demonstration of teaching effectiveness for merit raises, awards, or tenure decisions.
College and universities that subscribe to the National Survey of Student Engagement (NSSE) can purchase the NSSE Module Experiences With Online Learning found at https://nsse.indiana.edu/nsse/survey-instruments/topical-modules/index.html. The module was developed in collaboration with Quality Matters and will be available in 2021.
ASSESSMENT OF PREPARATION FOR REAL-WORLD WORK/SOCIALIZATION
A nursing program aims to prepare students to be practice ready. This outcome is typically measured by evaluation of clinical performance by a supervising faculty member. The Billings (2000) framework clearly identifies the need to assess students for clinical practice in online programs, but different approaches may be used. Because students are typically geographically separated from faculty, nursing programs often use preceptors or adjunct faculty in distant clinical sites, with faculty retaining the responsibility for assessing competence. Online clinical conferences provide a method to assess clinical knowledge and debrief students. Another method to assess readiness is the online Objective Structured Clinical Examination, which provides direct observation of student performance by a faculty member using a structured scoring tool.
Educating nurses at any level requires nurturing the values of professional nursing. Mentoring of the next generation of RNs, advance practice nurses, or nurse scientists can be challenging in online courses. A carefully constructed course addresses knowledge, skills, and values, but without assessment, it is difficult to know if students learn the values of caring, accountability, and advocacy. Faculty can use the Nurses Professional Values Scale-3, a three-factor 28-item scale with high overall reliability (Weis & Schank, 2017).
ASSESSMENT OF INFORMATION LITERACY
Lack of computer skills was a concern of many faculty when online courses were first offered and is reflected in the Billings (2000) framework. Although some students still struggle with computer skills, the vast majority enter college with basic skills. Information literacy, the ability to find, retrieve, and use valid information, is a more contemporary concern. Press releases on scientific findings and misinformation on the Internet can lead students to use weak or wrong evidence to guide practice. Faculty can use the Self-Efficacy Survey and Information Literacy Knowledge Test to assess their students' confidence and competence about information literacy (Tepe & Tepe, 2015). The tool also includes a 25-item test of knowledge about how to find, retrieve, and use information for practice. The survey and test have acceptable reliability and validity.
ASSESSMENT OF SATISFACTION/CONCLUSION
Satisfaction with online courses or programs is the final outcome in the Billings framework. Typically, this outcome is measured with institutionally developed course evaluations. Such evaluations can provide longitudinal data for program evaluation if questions specific to online education are included. A tool that can be purchased is the Educational Testing Services Student Instructional Report (SIR II(TM) Online and e-SIR) for distance learning found at https://www.surveyresearchonline.com/se/7321E7ED793D460E.
Faculty who teach online can develop plans to assess their course to make data-driven changes and then evaluate the effect of changes in cycles of improvement. In this way, online education in nursing can continue to offer high-quality options for students who wish to become nurses or increase their academic preparation for more advanced nursing roles.
REFERENCES