The call for higher levels of educational preparation for nurses in the Future of Nursing report has contributed in part to the increase of online education (Institute of Medicine, 2011). The outcomes of online education have been compared to those of on-campus education. Findings are consistent: Students learn as well or slightly better in online and hybrid learning environments as compared to on-campus environments (Jowsey et al., 2020; Kummerow et al., 2012).

Accreditation standards require that programs offered at a distance be equivalent to the programs offered on campus (Accreditation Commission for Education in Nursing, 2017; American Association of Colleges of Nursing, 2020). Indeed, to make progress increasing the educational level of nurses from associate to bachelor's and graduate degrees, online education has become a critical method. The ongoing COVID-19 pandemic has also caused many nursing programs to shift courses and programs to online delivery, without the necessary preparation time. These factors show the need to assess online courses to ensure a high-quality online education.

ASSESSMENT FRAMEWORK

Billings (2000) presented a classic systems-based framework to assess technologies, support for faculty and students, educational practices, and outcomes in online nursing courses. In this column, I describe tools and resources to organize a thorough assessment.

Technology

My inbox is full of emails from companies that tout the benefits of teaching technologies. Although it is tempting to try the newest and flashiest technology, Billings' (2000) framework indicates two major characteristics of technology that should be examined: 1) the accessibility and reliability of the technology infrastructure and 2) the effect of the technology on educational productivity. In my first column for the Emerging Technologies Center, I described how to design apps for the best user experience (Frith, 2019). These same methods can be used to assess online technology and include learnability (with first clicks), memorability (with eye tracking), productivity (time to complete technology task), error rate (number of errors in a single task), and student/faculty satisfaction. The System Usability Scale and other usability methods are found at https://www.usability.gov/how-to-and-tools/methods/usability-evaluation/index.ht.

Support for Students and Faculty

As the Billings (2000) framework suggests, students and faculty need orientation to online education, training modules for specific technologies, and 24-hour technical support, including weekends. Students need the full range of support services offered to on-campus students. Support for students can be assessed by a "mystery shopper" who is hired by the university. The mystery shopper examines the student experience, from admission to graduation, by interacting with vendors used by the university or with university offices such as bursar, registrar, financial aid, counseling services, writing centers, and tutoring services (McKenzie, 2018). Results from the mystery shopper can be used to improve the experiences of online students.

Another effective assessment method is to survey students about "negative critical incidents" in the administration of support services, functionality of software and learning management systems, teaching strategies, and interactions with faculty and students (Lin et al., 2011). The researchers tested their model, Satisfaction Assessment From Frequency of Negative Critical Incidents for E-Learning, and found higher numbers of negative critical incidents were associated with lower satisfaction and intent to continue in the course. The quantification and analysis of negative critical incidents could provide insights useful in improving online courses to retain students through graduation.

Educational Practices

The assessment of educational practices in Billing's framework is based on the Seven Principles for Good Practice in Undergraduate Education (Chickering & Gamson, 1987). The seven principles are active teaching, time on task, high expectations, prompt feedback, interaction between students and faculty, cooperation among students, and respect for diversity.

ACTIVE TEACHING

A faculty peer or an instructional technology specialist can assess the presence of active teaching strategies in online courses using the Quality Matters Rubric, found at https://www.qualitymatters.org/qa-resources/rubric-standards, or the Online Learning Consortium's OSCQR Course Design Review Scorecard, found at https://onlinelearningconsortium.org/consult/oscqr-course-design-review/.

TIME ON TASK

Time on task is a critical factor in any student's ability to learn. The faculty member must assess the workload and communicate this information to students. Because of increasing attrition of students from the RN-BSN program, Northrup-Snyder and colleagues (2020) conducted a study to estimate the course workload of five online courses. They used the US Department of Education's guidelines for credit hour to clock hour to set the benchmark for online instruction. For example, a three-credit hour course would have three clock hours of instruction (e.g., recorded lectures, video conferencing, discussion boards) led by the faculty member, and the student would be expected to study independently for two clock hours per credit hour. Thus, a three-credit hour course should have nine hours of weekly workload. The authors used the Course Workload Estimator (Barre, 2016) to calculate the time required to complete modules in five nursing courses and found a wide variety of hours per week required, from 3.8 to 83.9 hours for three-credit hour courses.

This study illustrates the critical need to thoroughly assess time on task for readings, videos, discussions, assignments, and quizzes. After estimates are made, the faculty member should list the estimated time to complete work and ask students to confirm or disconfirm the estimates. Adjustments in course requirements should be made if time on task falls short or exceeds the credit hours awarded for the course.

HIGH EXPECTATIONS

Having high expectations does not mean loading students down with unmanageable readings or assignments, but rather selecting assignments that help students meet the course learning outcomes or objectives. Online educators typically use rubrics to reveal their expectations about assignments and to grade written or oral student work (Fisher et al., 2019). A less frequently practiced strategy is the aggregated assessment of student performance by rubric category. For example, a rubric for a paper on delegation of tasks to unlicensed personnel might include an introduction, description of the circumstance, task analysis, assessment of scope of practice or job responsibilities, plan of supervision and communication, conclusion, and technical writing. If scores in each of the categories are aggregated, the faculty member could use the aggregated rubric to objectively assess the group's strengths and need for improvements. Just-in-time instruction might be necessary to reteach a concept, or different approaches to teaching the concept might be planned for future course offerings.

PROMPT FEEDBACK

Prompt feedback has been shown to have a positive effect on student satisfaction and academic achievement (Bonnel, 2008). The assessment of faculty feedback can be completed by peers who look for the following characteristics of feedback: 1) provide feedback within the time frame stated in the syllabus, 2) be positive, 3) list reasons for the scores in rubric categories, 4) offer specific suggestions for improvements, and 5) give reminders about relevant support services. Results from peer review of feedback need to be used by faculty members to improve their educational practices.

INTERACTIONS AND COLLABORATION

Interactions between students and faculty and among students are critical educational practice. The Community of Inquiry framework defines the need for three types of presence in online educational experiences: cognitive (interaction with content), social (interaction among learners), and teaching (interaction between teacher and learners; Garrison et al., 2020). The research group that developed the Community of Inquiry framework also developed and validated an instrument to measure the three types of presence (Arbaugh et al., 2008). It is available free of charge at https://coi.athabascau.ca/coi-model/coi-survey/. Results from this survey can be used regularly to assess adequacy of educational experience from the student perspective.

Another tool for assessing the influence of teaching presence on deep learning in nursing was developed by Takase et al. (2020). This validated tool measures student perceptions in three aspects of teaching: 1) providing relevant knowledge, 2) facilitating learning activities, and 3) being enthusiastic and supportive. The researchers used path analysis and found significant relationships. Providing relevant knowledge and facilitating learning activities were directly related to deep learning; being enthusiastic and supportive was indirectly related to deep learning.

RESPECT FOR DIVERSITY

Respect for diversity in online courses is a critical educational practice. The authors of a systematic review of inclusiveness and diversity of learners in online environments provide a comprehensive review of research reports and mind maps showing effective educational practices (Fermin-Gonzalez, 2019). Typically, faculty members include text, audio, video, and simulation learning resources to reach different types of learners, but students with disabilities have needs for accessible content. The US government has guidelines for creating accessible content found at https://www.usability.gov/what-and-why/accessibility.html; tools to check accessibility can be found at https://www.section508.gov/test.

CONCLUSIONS

In order for nurses to increase their educational preparation, courses and programs need to be accessible and high quality. The Billing's Framework for Assessing Outcomes and Practices in Web-based Courses provides a logical way to organize assessments. The use of data from assessments is critical to the continuous improvement of online courses in nursing. In Part 2 of this column, I will describe methods to assess outcomes in online education.

REFERENCES