With most hospitals now using an electronic medical record (EMR), a new trend has emerged. As organizations hire new nurses, they arrive with previous EMR experience. This has served as a catalyst to rethink how EMR training should be conducted within our organization. A traditional approach is instructor-led training (ILT), which is time-consuming for both information technology (IT) departments and nursing units. A more efficient training methodology that capitalizes on existing EMR knowledge could have many positive effects, including faster onboarding, increased end-user satisfaction, and potential organizational savings. One such training methodology is a self-paced, electronic learning (eLearning) format, which caters to existing EMR use. According to MacDonald et al,1 eLearning involves the use of technologies to deliver learning solutions that enhance knowledge and skills electronically with high levels of interactivity, flexibility, information access, and communication. Evidence suggests that there is no difference in knowledge, skills, and satisfaction between ILT and eLearning.2-6 If organizations can successfully transition to this format of EMR content delivery, there could also be potential for reduced training time, increased user satisfaction, and organizational revenue.

BACKGROUND

The Health Information Technology for Economic and Clinical Health (HITECH) Act was created in 2009 as part of the American Recovery and Reinvestment Act, which began the push for healthcare organizations to convert from paper to EMRs.7 The Centers for Medicare & Medicaid Services enforced the HITECH Act with incentive payments to eligible hospitals that were meeting the criteria for meaningful use of EMRs.8 Meaningful use relates to using certified health IT to maintain privacy and security; improve care coordination; engage patients and families; improve quality, safety, and efficiency; and reduce patient disparities in hopes of achieving optimal clinical outcomes.9 For those who did not meet meaningful use, financial penalties were enforced. Because of this, organizations began implementing EMRs at a rapid rate.10

At the same time that the HITECH Act prompted organizations to convert to EMRs, The Joint Commission (TJC) began to recognize the inherent risks of conversion. Sentinel Event Alert (SEA) #42 was issued by TJC to address the safe use of IT in healthcare.11 In this alert, TJC11 expressed concern that any form of technology might adversely affect the quality and safety of care if it is implemented improperly. They suggested training programs and refresher courses for all types of clinicians as one of many actions within SEA #42 that may help to prevent patient harm related to health IT.11 As more organizations began utilizing an EMR, an additional alert was issued; SEA #54 addresses the communication workflow and human-computer interface, which relates to usability, ergonomics, and data-related errors.12 The Joint Commission recommends that organizations provide training and have users demonstrate competency before accessing the system.12

PURPOSE OF THE PROJECT

The purpose of this project was to evaluate the eLearning usability and return on investment (ROI) of a nursing EMR training conversion from ILT. The International Organization for Standardization has established three goals for usability: effectiveness, or how accurately and completely the user achieved goals in specific environments; efficiency, which entails the resources expended in relation to the accuracy and completeness of goals; and satisfaction, involving the perception of comfort and acceptability associated with the product.13 To improve usability, organizations need to focus on users, along with the processes and procedures for user-centered design, while possessing the funding and resources to make positive change.14

REVIEW OF THE LITERATURE

Effectiveness

Several studies have indicated there is no difference in effectiveness and knowledge gained between ILT and eLearning.2-5,15-20 However, Sung et al21 found that nurses who received eLearning training had a higher level of knowledge related to medication administration compared to those receiving ILT. In addition, Kontio et al22 found that general self-efficacy strengthened in the eLearning group at the 3-month time point. Wutoh et al6 revealed that Web courses were more effective than ILT formats.

Efficiency

The literature indicates that eLearning may facilitate faster training than traditional, instructor-led methods. When incorporating eLearning methods, Sung et al21 found reduced lecture time, which eased the burden and increased the satisfaction of instructors. Wutoh et al6 also found that learning efficiency was higher with online education. Berke and Wiseman23 noted a transition to eLearning from traditional classroom training could save time by as much as 25% to 60%. In their study comparing classroom training to eLearning, Sheen et al24 discovered that the convenience of eLearning also saved time in the learners' professional lives. They further stated that if nurses could self-initiate and self-direct to improve their knowledge and skills at any time and location with eLearning, the costs to hospitals and nurses for continuing education could be reduced.24

Satisfaction

The evidence suggests that nurses are equally satisfied with an eLearning training methodology when compared to ILT.3,4,15,21,24,25 Determinants of satisfaction with eLearning have included learning climate, performance expectations, learner interface, system functionality, intention to use, and computer self-efficacy.26 It was also noted that system quality, information quality, and service quality of a nursing e-learning system have a direct effect on nurse satisfaction with system use.27 The concept of eLearning has been studied across disciplines, resulting in user satisfaction with eLearning approaches.28-30 No evidence was found to suggest that online training methodology had a negative overall impact on nursing satisfaction.

Theoretical Basis

The Information Systems Success Model (ISSM) was developed in 1992 by DeLone and McLean31 to evaluate system quality, system use, information quality, and user satisfaction related to the subsequent success of information systems. They updated the model in 2003 to include the impact of the organization and individual from use of information systems.32 This theory has been used extensively to evaluate eLearning success27,30,33-36 and provides the ideal framework for the project. The conceptual definitions of this theory will be applied to the project interventions and outcomes, while serving as a framework for evaluation (Table 1).

Table 1 Theoretical Substruction of the ISSM and EMR Training Conversion to eLearning

METHODS

Design

This study was conducted at a large, Midwestern academic medical center that comprised seven hospitals on two campuses, with a combined total of 1382 beds.37 The Hospital Feasibility Committee reviewed the project and determined it to be a quality improvement project; thus, institutional review board submission was not necessary. A convenience sample of newly hired, onboarding nurses was used with a mixed-methods design. The eLearning conversion began in August 2016 for inpatient nurses, followed by the evaluation in November and December 2017.

Procedures

Newly hired nurses working on inpatient units made up the sample. These participants were in their first week of hire, when they were required to begin their EMR training via eLearning in 4-hour time slots on 2 separate days. Participants arrived at a proctored computer laboratory and were given instructions that served as a guide for completion. They were provided initial guidance on how to access the organization's learning management system (LMS) for the completion of the required EMR training sessions in an eLearning format. A proctor was present to assist with answering questions and troubleshooting issues and provide verbal cues to encourage completion of the evaluations.

Measures

The usability categories of effectiveness, efficiency, and satisfaction were determined using a post-eLearning evaluation that all end participants took at the completion of training. The evaluation included measures pertaining to demographics, which included years of nursing and Epic EMR experience, satisfaction and effectiveness, and playground feedback. The eLearning evaluation was adapted from the initial ILT evaluations. Both eLearning and ILT evaluations contain content validity, which was determined by experts on the IT Training and Optimization team. The eLearning development process involved several members of this team who ensured quality for the product delivered.

The initial eLearning conversion began in August 2016, and the evaluations were launched into the system in October 2017. Anonymous evaluation data from participants were collected every 2 weeks beginning November 1, 2017, through December 2017. The evaluation contained 13 questions to be answered with a Likert scale or yes/no response. The Likert scale ranged from 5 = strongly agree to 1 = strongly disagree. An additional question collected qualitative responses. The evaluation was distributed in the LMS, and results were exported into an Excel file (Microsoft, Redmond, WA) as aggregate, deidentified data. Evaluations from ILT from November and December of 2015 were used to assess satisfaction and effectiveness of that learning methodology for comparison with eLearning. Deidentified, aggregate data were collected for both ILT and eLearning evaluations.

The LMS captured eLearning efficiency in a course timings report for each eLearning module. Participants were encouraged to limit breaks while a lesson was open, in an attempt to get the most accurate completion time. The eLearning completion times were compared to the time allocation for the same ILT course to determine efficiency and ROI using the average salaries for a newly hired inpatient nurse.

Data Analysis

Data were analyzed using SAS 94 (SAS Institute, Cary, NC). Descriptive statistics were used in the form of mean, SD, and frequency percent to show how the eLearning met nursing perceptions of effectiveness and satisfaction. Inferential statistics in the form of independent-samples t tests were used to determine group differences with respect to confidence, satisfaction, and effectiveness among participants taking the eLearning. Comparisons were made based on nurses with and without Epic EMR experience and their years of nursing experience. Finally, one-sample t tests were used to assess whether eLearning and ILT differed with respect to meeting course objectives and nursing satisfaction.

The last item on the survey was an open-ended question that asked participants to share feedback on the eLearning regarding what they liked and what could be improved for the future. Qualitative data collected from this question were analyzed by coding the information and placing it into themes38 reflective of the concepts of usability.

FINDINGS

The eLearning evaluations were completed by 75 participants, all inpatient nurses in their first week of hire. Four separate EMR training sessions were used, three in November 2017 and one in December 2017. Results show that 76% of participants (n = 57) reported coming to the organization with prior experience with an Epic EMR. The range of prior Epic EMR experience was 1 year up to 10 years (mean = 2.49 [SD = 1.90]). Years of nursing experience were reported by 46 participants, with the range from new graduate (n = 10) to 24 years (mean = 6.46 [SD = 6.68]).

Effectiveness

Participants were asked whether the eLearning was an effective means to facilitate learning. Eighty-four percent (n = 63) agreed that the eLearning was an effective means for learning. Overall, the respondents were positive about most aspects that made the eLearning an effective tool (Table 2). The playground is an EMR environment where nurses can practice workflows to gain confidence in system use. While the playground exercises were made available to everyone, 93% (n = 70) of the participants indicated they were effective, and 77% reported they had time to access them. Participants were neutral (mean = 3.65 [SD = 0.95]), when asked if the playground exercises enhanced their experience.

Table 2 Effectiveness Data From eLearning Evaluation

Efficiency

With ILT, each of the four classes, Nursing Foundations, Admission, Shift, and Discharge, lasted 4 hours, a total of 16 hours to complete. These same classes were converted to eLearning modules and took a combined 485.77 minutes, or 8.09 hours, for an approximate 50% reduction in completion time.

Satisfaction

When participants were asked if they were satisfied with the online courses, 76% reported they agreed (n = 41) or strongly agreed (n = 16), while 13% (n = 10) were neutral and 11% (n = 8) disagreed. Although the majority of participants reported satisfaction, the satisfaction mean was 3.87. Participants who disagreed cited reasons in qualitative feedback such as not having enough time to complete the eLearnings, preference of having the content be delivered as instructor-led, proctor issues, and too slow for someone with Epic EMR experience.

Return on Investment

To calculate the ROI for this project, the midpoint of the current pay range for a staff nurse role was provided by human resources. The ROI for the project duration was $18 500 and for the calendar year of 2017 was $156 935. When converted to hours gained, the nursing department gained 593.25 hours during the project period and 5022.85 hours during 2017 (Table 3).

Table 3 Return on Investment for Project Duration and for 2017 Calendar Year

Group Comparisons

Independent-samples t tests were conducted to identify any differences among participants taking the eLearning based on demographics. The first group comparison analyzed the variables of confidence, satisfaction, and meeting the learning objectives of participants with (n = 57) and without (n = 18) Epic EMR experience. Results showed that with confidence (P = .44), satisfaction (P = .30), and whether the eLearning met the intended objectives (P = .13), no statistical differences were found (Table 4).

Table 4 Participants With and Without Prior Epic EMR Experience Compared With Confidence, Satisfaction, and Meeting Objectives

Next, participants were divided between those with 2 years of nursing experience or less (n = 27) and those with more than 2 years of experience (n = 19) to see if any differences existed among confidence, satisfaction, and meeting learning objectives. Results showed that confidence (P = .71) and satisfaction (P = .52) were not statistically different between the two groups at the 0.05 level, but less experienced participants had higher average scores regarding whether the lessons, topics, and content were directly related to the learning objectives (P = .0.04) (Table 5).

Table 5 Nursing Experience Compared With Confidence, Satisfaction, and Meeting Objectives

The ILT and eLearning results were further compared using one-sample t tests based on meeting course objectives and user satisfaction, which were present in both postcourse evaluations. One hundred thirty-seven ILT evaluations were used: 9 from Foundations, 33 from Admissions, 43 from Shift and 52 from Discharge. The ILT satisfaction mean was 4.41, and for meeting course objectives, the mean was 4.61. When the eLearning satisfaction mean of 3.87 was compared to ILT average of 4.41, statistical significance was found (P < .0001). Similarly, when the ILT mean for meeting the learning objectives (4.61) was compared to the eLearning (mean = 4.36), this demonstrated statistical significance (P = .0010) (Table 6).

Table 6 Comparison of eLearning and Instructor-Led Sessions in Terms of Objectives and Satisfaction

Qualitative Data

The last item on the survey was an open-ended question that asked the participants to share feedback regarding what they liked about the eLearning and what could be improved for the future. These responses were sorted into common themes based on keywords that reflected effectiveness, efficiency, and satisfaction.

In order to further assess nursing satisfaction, participants offered feedback on what they liked about the eLearning. Eight participants responded to this question and the common theme was the positive review of the self-paced format. One participant stated, "I liked that it was go at your own pace." A related common theme was that experienced users would like the opportunity to test out of content. However, even though self-paced was preferred by most, two participants stated that the content should be delivered as ILT.

Twenty-five participants offered qualitative feedback for what could be improved. The most common theme related to allotted time for completion of the eLearning. A participant stated, "Not enough time for the amount of education required." Another said, "[horizontal ellipsis]no time for playground due to it taking the entire time to complete just the online course." Another common theme related to the proctor overseeing the eLearning. One comment read, "I would have benefitted from communication from the staff person overseeing it at the beginning, giving us a welcome, overview, instructions, telling us what to expect, when to ask for help, basically any (communication)".

DISCUSSION

Project Outcomes

The eLearning usability goal was met and comparable to the inpatient nursing eLearnings for EMR training. The majority of participants were satisfied with the eLearning, even though the mean of 3.87 shows that there is room for improvement. The eLearning was found to be an effective tool for content delivery. The results confirmed that eLearning is more efficient than an ILT method of delivery. The 50% reduction in training time was achieved compared to ILT, which leads to a substantial ROI for the organization. Upon completion of the inpatient nurse curriculum via eLearning, this project showed no difference between nurses with and without prior Epic EMR experience related to confidence, satisfaction, and meeting eLearning objectives. Furthermore, no statistical difference was found between participants with less than 2 years of experience compared to participants who had more than 2 years of experience regarding satisfaction with the eLearning and future confidence to use the system. However, new participants reported that the eLearning objectives were met more often than did experienced participants.

Compared to eLearning, ILT was superior in terms of satisfaction and meeting course objectives. Qualitative feedback demonstrated that proctor performance affected learner satisfaction and that many participants liked the self-paced format of eLearning. Those who need more time can take it, while those who know the workflows can move faster. Improvements related to this feedback may raise satisfaction with eLearning to the ILT level.

Limitations

There were several identified limitations to this project. Some participants did not answer all of the questions, leading to missing data. Additionally, the course timings report pulled the amount of time that the learner had the eLearning lesson open. Because of this, it should be noted that the time for completion of the eLearning may actually be less than what was indicated in the course timings report. Finally, while 137 ILT evaluations were used, we do not know how many participants completed evaluations.

CONCLUSION

With the majority of inpatient nurses onboarding with Epic EMR experience, our inpatient nursing eLearning conversion allows our organization to capitalize on existing experience and increase satisfaction with a self-paced format. In the future, a mechanism to allow users with prior Epic EMR experience to test out of content would be beneficial and allow for an increased ROI. However, with the average time of completion at 8.09 hours, we need to provide some users with additional time. The eLearnings also provide a means for ongoing reference for nurses that was not possible with ILT. The important task of orienting newly hired nurses to an organization's information systems and workflows has grown more challenging in the increasing presence of electronic record technologies utilized with patient care.39 New nurse hires' EMR training has an important function of incorporating workflows and policies, in addition to introducing system functionality. This project has shown that an eLearning format does yield confident nurses who are equipped with the knowledge they need to be successful with patient care.

References