Authors
- Sormunen, Marjorita PhD, RN
- Heikkila, Asta PhD, RN
- Salminen, Leena PhD, RN, PHN
- Vauhkonen, Anneli MHSc, RN
- Saaranen, Terhi PhD, RN, PHN
Abstract
Implementing digital technologies has become a policy priority worldwide among all types of education. The COVID-19 pandemic has further accelerated educational institutions' efforts to reorganize their teaching and introduce new digital learning technologies. Although using digital technologies in higher education, including nursing, is considered a modern and innovative way of teaching and learning, uncertainty exists concerning these technologies' actual usefulness in achieving positive learning outcomes. The aim of this scoping review was to examine the current evidence related to the effects of using digital technologies on learning outcomes in higher education. The authors searched five electronic databases for relevant studies and used a scoping review method to analyze and synthesize the evidence. Eighty-six articles from six disciplines met the selection criteria. As a key finding, the outcomes of the interventions were mainly positive. Increased professional knowledge, skills, and attitudes reflect the advancement of professional competence. Academic, collaborative, and study skills, in turn, contribute to general competence development. Our findings suggest that digital technology has the potential to improve learning in various disciplines.
Article Content
The use of digital technologies in higher education has become ubiquitous in all disciplines and has grown exponentially in recent years.1-5 Moreover, the innovative use of digital technologies in higher education has become a policy priority worldwide, leading to an emphasis on educational reforms and modernization supporting digitalization in education.6-8 As digital learning methods are also increasingly used in nursing education, it is necessary to examine their usefulness and effectiveness.4
The first digital education policies at the end of the 20th century focused on infrastructure development of learning institutions. Further, the Lisbon Strategy9 and the eLearning Action Plan7 focused more on strategic policy objectives fostering digital competences with integrating digital technologies into education.7 Today, e-learning policies are focused on building up digital competences, along with other 21st century skills8 of citizens, teachers, and learners.6 Moreover, since early spring of 2020, the COVID-19 pandemic has required educational institutions worldwide to reorganize their teaching and introduce new digital learning technologies in an unprecedented way. Hence, the students had no choice between online learning and face-to-face learning, and many faculty members were forced to adopt the new teaching approach rapidly despite a lack of digital teaching experience and with little preparation and support from technological teams.10 From a positive perspective, the sudden shift to online teaching has also created new insights and strengthened earlier observations about teaching and learning. For example, increased live online learning during the outbreak has demonstrated the importance of finding ways to promote students' collaborative learning via online platforms, as well as using appropriate resources and encouragement to increase and maintain motivation and readiness for live online learning.11
Digital technology enhances the possibilities of learning. Digital learning strategies have been seen as a way of facilitating teaching excellence, promoting research-based teaching, supporting interdisciplinary education, and bringing practical innovations to classrooms.12,13 Through digital technology, the learning can be transformed into a more active and personalized process.13,14 For example, for learners, technology offers an opportunity to share and collaborate in learning across the Web, and potentially around the world, with peers, academics, and professionals.15 In addition, online learning could benefit nonnative English speakers in reading, vocabulary acquisition, comprehension, and writing.16 However, more research on digital collaborative learning is needed regarding, for example, nursing education environments.5
Digital education, which consists of teaching and learning via digital technologies, helps overcome resource, geographical, and time barriers.4 For instance, digital technology allows students to access and distribute learning materials and to study at their own pace without limitations on time or location.4,16,17 A further advantage of digital learning is that it commonly limits the number of tutors needed. Moreover, the number of participants in this type of education can be higher than that in face-to-face lectures or seminars.17 However, it must be noted that digital education may involve many challenges, which exist alongside its advantages. For example, there are specific costs associated with using technology, like site license, technical support, and technology infrastructure,18 and some educators may resist and struggle to integrate technology into their instruction optimally or in innovative ways.1 In online education, teachers also find it difficult to assess students' abilities and performance, and there is an increased tendency toward exam misconduct.19
In digital education, the role of teachers has become more significant to help students learn to sort, understand, analyze, and use information.14 Hence, teachers need sufficient technological competence to play these roles and to design suitable learning environments.5 However, Cushion and Townsend18 point out that educational approaches should be developed together with technologies, as pedagogy is the key aspect to promote learning. In addition, it has to be considered that students may need substantial support to use technology effectively.5 For example, Mannisto et al5 stated that teachers need training on how to plan and improve the use of collaborative digital learning in nursing education. Accordingly, to become a technologically competent teacher may require a transformation of the way we currently think about teacher training.20
In digital learning interventions, different technological solutions have been used individually, like virtual reality,13 or comparatively, alongside other methods, such as face-to-face feedback sessions vs blended digital video.21 Digital learning interventions have been applied in theoretical5 as well as in practical21 education. There is evidence suggesting that learning interventions using digital solutions, like virtual reality,22,23 and adaptive e-learning3 improve learners' knowledge and skills or are at least as effective as traditional education. In contrast, Kyaw et al22 focused on the effectiveness of virtual reality learning interventions, but results concerning attitudes and satisfaction among the health professionals were ambiguous. In addition, in a review by Tudor Car et al,24 the studies reported little or no difference in pre- or post-registration healthcare professionals' behavior regarding the adoption of clinical practice guidelines. Hence, research evidence related to digital learning intervention outcomes are often of inadequate quality, and there is high heterogeneity across populations, interventions, comparators, and outcomes, which limits the interpretation and application of findings.3
In higher education, earlier research and reviews have often been focused on a certain academic discipline, like nursing5 and health sciences,3,4,17,22 sports science,18 business administration,14 education science,20,21 or architecture.13 This review focuses on learning outcomes of digital learning interventions in higher education without imposing any restrictions regarding academic discipline. Following this idea of broad inclusion of learning interventions, this study addresses one research question: What are the reported learning outcomes of digital learning interventions in higher education?
METHODS
Scoping Review
A scoping literature review, which gives visibility to a large body of recently published intervention studies, was used in this review. The methodology for this scoping review was adopted by Arksey and O'Malley25 and includes the following stages: (1) identifying the research question; (2) identifying relevant studies; (3) selecting studies; (4) charting the data; and (5) collating, summarizing, and reporting the results. Four authors were involved with stage 1; two authors, with stages 2, 3, and 4 (with the exception of consensus discussions among the four authors); and all authors, with stage 5. The authors have previously published an article based on the same data, focusing purely on technological outcomes of digital learning interventions.26
Search Strategy and Eligibility Criteria
CINAHL (EBSCO), MEDLINE/PubMed, Eric (ProQuest), Scopus, and a Finnish database, Medic, were searched, with consultancy of a university librarian, to identify intervention studies published between 2015 and 2018 in English or Finnish language. Two searches were conducted in four international databases using the following search terms: digital, mobile, virtual, games, social media, Web-based, Internet, information technolog*, Information and communication technology (ICT), communication technolog*, technology uses in education, educational technology, E-learning, electronic learning, online courses, teach*, learn*, education, higher education, universities, postsecondary education, interve*, teaching method, and teaching technique. Additionally, the first search included the search term "intervention"; and the second, the search term "teaching method." In the Medic database, the search included Finnish search terms: digital, mobile, virtual games, social media, Web-based, Internet, information technology, ict, communication technology, technology uses in education, educational technology, e-learning, electronic learning, eoppiminen, digitaal*, virtuaal*, mobiil*, opetus, teach*, learn*, education, and koulutu*. As the number of retrieved articles was high, manual search was not implemented. The inclusion criteria were as follows: empirical, peer-reviewed intervention studies; a focus on digital learning intervention related to a curricular subject; students in higher education (universities, colleges, universities of applied sciences); degree programs; intervention with (1) two or more groups, experimental and control, (2) one group, two measures, or (3) one group, one measure; and quantitative, qualitative, and mixed methods studies. Exclusion criteria were as follows: review article; simulation as a learning method related to health behavior change of a certain group; patient/client/clinical care; noneducational setting; and continuing education.
Screening Process
The database searches identified 1252 articles, which were imported to the bibliography management program RefWorks. After duplicates were removed, 1159 titles were screened based on the inclusion and exclusion criteria. Two authors (MS and AH) individually reviewed each study for the titles and abstracts, removing 918 articles in the title screening phase and 109 articles in the abstract screening phase. A total of 132 articles further proceeded into full-text eligibility assessment. Of these, 91 were initially accepted, and after consensus discussions among the four authors (MS, AH, LS, and TS), 86 articles were included in the review.
Data Analysis
Data extraction was performed on the included studies (n = 86) according to author, year, country, study purpose, academic discipline(s), main digital means of intervention, methodology or design, and data collection method(s). An inductive content analysis of studies led to identification of 37 sub-categories. These were then reduced into six categories and further formulated to consist of two main categories. Weakened learning outcomes from six studies were reported only in text in the Results section.
RESULTS
Characteristics of the Interventions
A majority of the interventions described in articles were conducted in the academic discipline of health sciences, making up 51% (44/86) of the review. Teaching/learning methods varied between being individual or group-based, and most often, purely online courses (42%, 36/86). Intervention duration was not reported in all articles or was described poorly, but the variation was large, from several hours to an entire academic year. The largest number of studies was conducted on the continent of Asia (38%), followed by North America and Europe (28% and 21%, respectively) (Table 1).
Reported Learning Outcomes
Reported learning outcomes of digital learning interventions in higher education were divided into two main categories: professional competence and general competence (Table 2), which are presented in the following chapters. References from each category are mentioned as an example.
"Professional competence" was composed of three categories: professional knowledge, professional skills, and professional attitudes. In the category of professional knowledge, several studies reported that digital learning interventions simply increased knowledge of the learning content, but many studies also assessed the level of knowledge enhancement. Learning outcomes were found in the knowledge levels of understanding and applying the knowledge (Table 2). For example, in the understanding subcategory, virtual lab training increased engineering students' conceptual understanding,55 serious games increased medical students' understanding of patient safety issues,45 and Web applications increased chemistry students' understanding and retention of knowledge in spin-spin spitting.38 In terms of applying the knowledge, studies reported outcomes such as vocabulary knowledge among language students,56 application of psychology knowledge among social science students,61 and nursing students' capability to apply nursing informatics knowledge in clinical practicum by using mobile academic medical record application.60 Engineering students engaged in virtual laboratory training increased application of the knowledge in new scenarios, as well as the retention of knowledge after several weeks.55 One study reported reduced understanding, but the same finding was found in all study groups.110
Learning outcomes in the category of professional skills were reported in health sciences, humanities, social sciences, engineering, and technical sciences. In health sciences, digital learning interventions increased clinical skills, such as diagnostic skills,77,79,81 examination skills,72 nursing skills,34,78 and long-term clinical skills retention.79,81 For example, interactive learner-centered nursing education mobile application enhanced nursing students' nursing skills performance,34 and interactive e-learning course increased medical students' x-ray interpretation skills; skill retention was higher in the e-learning group than in the control group in a follow-up assessment after 1 year.81 In addition, several studies reported an increase of professional confidence and self-efficacy in clinical skills,33,73 and even two studies found decreasing outcomes.32,38
Positive learning outcomes in humanities were found in speaking,56,83,86,91 listening,89,90 reading,87-90 and writing84,85 foreign languages. ICT skills were increased in information management,93 in computer science,92 and in education,50 and math skills among mathematics students94 and nursing students.95 Interventions also promoted students' problem-solving and decision-making skills,55,97 teaching96 and counseling73,82 skills, and professional communication skills.73,98
A few more critical outcomes also existed; according to the study by Van Lancker et al,111 math-skill learning occurred less in the test group than in the control group. In addition, Thomas and Fellowes112 explored biology skills, and Nguyen103 probed operation management skills; students in both studies achieved less learning, relative to the control group.
Digital learning interventions improved professional attitudes, especially among health sciences students. Studies reported that learners increased medical students' positive attitudes and empathy toward patients73,99 and positive attitudes toward psychiatry,48 and promoted attitudes toward and willingness to engage in interprofessional collaboration. Further, the studies reported positive outcomes related to respect for cultural differences.58,83 Online peer discussions58 and online tutorials48 increased health sciences students' valuation and awareness of cultural differences. In addition, social media application increased nursing students' awareness of their own and others' cultural values and biases,32 and dialect awareness training promoted language students' positive attitudes toward dialects.83
"General competence" was divided into three categories: academic skills, collaborative skills, and study skills. Some studies reported positive outcomes related to academic skills, such as increased reflective thinking,55,57 promotion of deep learning,39,102 and critical appraisal of online sources75 and research.104 For example, in reflective thinking subcategory, completing virtual laboratories before physical laboratories increased engineering students' reflective learning,55 and in Web-based intercultural peer-learning interventions, nursing students were able to reflect on their personal knowledge and actions, as well as the actions of others, on intercultural aspects on nursing care.58
Digital interventions promoted collaborative skills in several studies. Digital learning interventions using, for example, asynchronous discussion boards, social media applications, and virtual coaching enhanced communication skills32,57,64 and promoted teamwork64,69 or interaction.28,56 For example, Facebook was used to develop cultural communication skills among nursing, pharmacy, and nutrition students.32 Further, the studies reported positive outcomes in learners' study skills. Several studies reported an increase of active participation and engagement in the course and promoted orientation to the learning topics. The use of smartphone application107 and learning analytics92 encouraged participation by and engagement of passive and less successful students. A great number of studies also reported increased learning motivation,30,40 self-directed learning,92,106 and enhanced confidence and self-efficacy in learning.31,106 For example, the immediate feedback on a serious game involving cardiopulmonary resuscitation instruction motivated nursing students to practice and solve problems.40 In addition, the use of mobile applications,34,56,87 virtual learning objects,29 and mobile video clips35 extended learning beyond the classroom, especially among nursing students.29,34,35 Some interventions also enhanced study performance.39,41
DISCUSSION
Of 86 intervention studies included in this review, the majority reported positive learning outcomes. There is also earlier research evidence suggesting that digital learning interventions improve learners' knowledge and skills.3,21,23 Increased knowledge and understanding of the learning content, as well as capability to apply the knowledge, are indisputably essential and desirable goals of education, regardless of discipline or profession. Technological solutions seem to play a clear role in this development: in addition, by furnishing a possibility to enable alternative ways of learning, skillfully prepared contents can enrich the learning content visually and attract the students to learn more broadly and/or more deeply.
Moreover, related to skills needed to perform in a profession, the results of this review clearly demonstrate that, in particular, clinical skills, which demand accurate performance, and skills that require repetition benefit from digital technology. Although learning by simulation was excluded from this review, other digital methods for rehearsing skills have certain advantages, due to the possibility of trial and error before an actual situation with clients, patients, or other groups of interest occurs. Moreover, and interestingly, acquiring skills related to interacting with and transmitting knowledge to others, namely, communication, counseling, and teaching skills, were found to increase using digital devices. It seems that, as McDonald and Glover15 have stressed, technology offers opportunities to share and collaborate with peers, academics, and professionals. Thus, these fundamental skills, which are often emphasized less in many public professions, such as medical practice, can be cultivated using, for example, videos or role-plays that provide real-life examples of good practices.
Although increases in knowledge and skills are somewhat easily predictable learning outcomes of technology use, increased positivity of attitudes is not necessarily as easy to anticipate and thereby requires more emphasis. Without a properly positive attitude, for example, toward collaboration, diversity, or learning content, the outcomes can remain short-term, and professional competence may lack an important dimension. Using digital technology to develop positive attitudes of the learner can, at best, provide a better foundation for the whole learning process.
In addition to the learning outcomes related to professional competence described above, several key outcomes were also found among general competence areas. These were related to academic skills that are needed to succeed and make progress in studies, collaborative skills that promote students' abilities to communicate and interact effectively with each other, and study skills that are essential to keeping up motivation, engagement, and performance. Similarly, other studies13,14 confirm that the use of technology broadens the learning process into a more participative, personalized, and active process. All such abovementioned competence areas are needed to achieve and maintain professional competence; they prepare learners for their future profession in an ever-changing working environment. Digital technologies can be seen as reinforcing the teaching of these essential general competences; using digital resources alone, without a strong teacher commitment and continuous supervision, probably does not offer as strong a basis as is offered when they are attended by such commitment and supervision. Therefore, it is important to pay attention to instructional strategies alongside technologies, as pedagogical scheme is the key factor impacting learning.18 The role of teachers becomes even more significant in digital education.14 Hence, it is vital to rethink teacher training,20 to allow teachers to become technologically competent and support students effectively and in innovative ways.1,5
Not all learning outcomes were positive. Some reflected equal effects, meaning that there was no statistical difference in intervention outcomes, even when the learning outcomes improved as a whole. This was observed in interventions related to professional knowledge, skills, and attitudes, as well as in general competences on a small scale. Reduced outcomes were reported in only six studies, but it was not possible to detect any unified pattern from these studies. Similarly, other studies have found equal17 or partial3 effects, or no difference in learning.24
For example, in a review by Kyaw et al,22 the effects of virtual reality learning interventions on attitudes and satisfaction among the health professionals remained inconclusive.
As only a few studies reported negative outcomes in this scoping review, this inevitably raises the question of whether positive results are more desirable to disseminate among academia than negative or "null" findings. Rapidly increasing technological applications offer multiple choices for educators, and it may therefore be important to identify and share the information of the interventions that are found to be feasible and effective, and produced increased learning outcomes, as well as those that are found to be the opposite.
The innovative use of digital technologies in higher education has become a policy priority worldwide.7,8 Digital education, which consists of teaching and learning by means of digital technologies, helps overcome resource, geographical, and time barriers.4 This study, along with earlier research,2-5 shows that the use of digital technologies in higher education has grown exponentially and this forms a solid foundation for the future. Moreover, as experienced, various global health, social, and economic occasions and influences, like the COVID-19 pandemic, can push developers and educators to quickly evolve innovative digital teaching and learning solutions to overcome the topical challenges. Hence, there is a need for more research about the aptitude of digital learning interventions in higher education, those that are discipline specific as well as those that are suitable for all.
LIMITATIONS
This review used the same data as a previously published review.26 When the database search and the screening process were finalized and the number of included articles was deemed high and the content rich, the authors decided to analyze the data in two equally important aspects of digital teaching and learning: technological and learning outcomes. Except for similar data, the two reviews are independent in their content. Moreover, together, they give a comprehensive overview of the aspects of digital learning interventions implemented in higher education from the perspectives of teachers and students, and they highlight the importance of teachers achieving high digital competence.
The articles included in the review were not assessed for their quality based on pre-set criteria because scoping reviews do not require assessment of the quality of evidence but provide an overview of the available literature. Even though the methodological variety and heterogeneity of included studies were large, this study provides a broad view of several disciplines, so it provides an opportunity to view a variety of technologically facilitated learning interventions that have been implemented in higher education. Despite these limitations, the present study contributes important theoretical and practical implications regarding digital learning interventions in higher education.
CONCLUSION
In conclusion, this study's findings indicate that digital technology can improve learning in the areas of knowledge, understanding, and application, so it can be applied to multiple learning situations in higher education. Strengthening all teachers' digital competence can support learners' adaptation and self-management in the face of unexpected challenges. Awareness of the possibilities and the courage to use digital technologies provides multiple alternatives for innovative learning and teaching in applied disciplines, like nursing education, in which rehearsing clinical skills and applying knowledge is vital to being able to perform in demanding real-life situations.
References
1. Casey A, Goodyear VA, Armour KM. Rethinking the relationship between pedagogy, technology and learning in health and physical education. Sport, Education and Society. 2017;22(2): 288-304. doi:. [Context Link]
2. Ellis RA, Bliuc A-M. Exploring new elements of the student approaches to learning framework: the role of online learning technologies in student learning. Active Learning in Higher Education. 2019;20(1): 11-24. doi:. [Context Link]
3. Fontaine G, Cossette S, Maheu-Cadotte M-A, et al. Efficacy of adaptive e-learning for health professionals and students: a systematic review and meta-analysis. BMJ Open. 2019;9: e025252. doi:. [Context Link]
4. George PP, Papachristou N, Belisario JM, et al. Online eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. Journal of Global Health. 2014;4(1): 010406. doi:. [Context Link]
5. Mannisto M, Mikkonen K, Kuivila HM, Virtanen M, Kyngas H, Kaariainen M. Digital collaborative learning in nursing education: a systematic review. Scandinavian Journal of Caring Sciences. 2019;39(4): 191-200. [Context Link]
6. Conrads J, Rasmussen M, Winters N, et al. Digital Education Policies in Europe and Beyond: Key Design Principles for More Effective Policies. Luxembourg: Publications Office of the European Union; 2017. Doi:. [Context Link]
7. European Commission. Communication from the commission to the European parliament, the council, the European economic and social committee and the committee of the regions on the Digital Education Action Plan. 2018. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52018DC0022. [Context Link]
8. van Laar E, van Deaursen AJAM, van Dijk JAGM, de Haan J. The relation between 21st-century skills and digital skills: a systematic literature review. Computers in Human Behavior. 2017;72: 577-588. doi:. [Context Link]
9. European Parliament. The Lisbon Strategy 2000-2010. An analysis and evaluation of the methods used and results achieved. Final report. Brussels. 2010. http://www.europarl.europa.eu/activities/committees/studies.do?language=EN. [Context Link]
10. Bao W. COVID-19 and online teaching in higher education: a case study of Peking University. Human Behavior and Emerging Technologies. 2020;2: 113-115. doi:. [Context Link]
11. Tang YM, Chen PC, Law KMY, et al. Comparative analysis of student's live online learning readiness during the coronavirus (COVID-19) pandemic in the higher education sector. Computers in Education. 2021;168: 104211. doi:. [Context Link]
12. OECD. Innovating Education and Educating for Innovation: The Power of Digital Technologies and Skills. Paris, France: OECD Publishing; 2016. ISBN 978-92-64-26509-7. http://dx.doi.org/10.1787/9789264265097-en. [Context Link]
13. Sanchez-Sepulveda MV, Torres-Kompen R, Fonseca D, Franquesa-Sanchez J. Methodologies of learning served by virtual reality: a case study in urban interventions. Applied Sciences. 2019;9: 5161. doi:. [Context Link]
14. Torres Kompen R, Edirisingha P, Canaleta X, Alsina M, Monguet JM. Personal learning environments based on Web 2.0 services in higher education. Telematics and Informatics. 2019;38: 194-206. doi:. [Context Link]
15. McDonald K, Glover I. Exploring the transformative potential of Bluetooth beacons in higher education. Research in Learning Technology. 2016;24: 32166. doi:. [Context Link]
16. Cottom CA. Distance Learning and the Non-native English Speaker: A Correlational Study [dissertation]. Lynchburg, VA: Liberty University; 2018. [Context Link]
17. Brusamento S, Kyaw BM, Whiting P, Li L, Tudor Car L. Digital health professions education in the field of pediatrics: systematic review and meta-analysis by the digital health education collaboration. Journal of Medical Internet Research. 2019;21(9): e14231. doi:. [Context Link]
18. Cushion CJ, Townsend RC. Technology enhanced learning in coaching: a review of literature. Education Review. 2019;71(5): 631-649. doi:. [Context Link]
19. Ebohon O, Obienu AC, Irabor F, et al. Evaluating the impact of COVID-19 pandemic lockdown on education in Nigeria: insights from teachers and students on virtual/online learning. Bulletin of the National Research Centre. 2021;45: 76. doi:. [Context Link]
20. Magen-Nagar N, Shachar H, Argaman O. Changing the learning environment: teachers and students' collaboration in creating digital games. Journal of Information Technology Education: Innovations in Practice. 2019;18: 61-85.5. doi:. [Context Link]
21. Prilop CN, Weber KE, Kleinknecht M. Effects of digital video-based feedback environments on pre-service teachers' feedback competence. Computers in Human Behavior. 2020;102: 120-131. doi:. [Context Link]
22. Kyaw BM, Saxena N, Posadzki P, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. Journal of Medical Internet Research. 2019;21(1): e12959. doi:. [Context Link]
23. Papanastasiou G, Drigas A, Skianis C, Lytras M, Papanastasiou E. Virtual and augmented reality effects on K-12, higher and tertiary education students' twenty-first century skills. Virtual Reality. 2018;23: 425-436. [Context Link]
24. Tudor Car L, Soong A, Kyaw BM, Chua KL, Low-Beer N, Majeed A. Health professions digital education on clinical practice guidelines: a systematic review by Digital Health Education collaboration. BMC Medicine. 2019;17: 139. doi:. [Context Link]
25. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1): 19-32. doi:. [Context Link]
26. Sormunen M, Saaranen T, Heikkila A, Sjogren T, Koskinen C, Mikkonen K, et al. Digital learning interventions in higher education: A scoping review. CIN: Computers, Informatics, Nursing. 2020;38(12): 613-624. doi:. [Context Link]
27. Agrawal N, Kumar S, Balasubramaniam SM, et al. Effectiveness of virtual classroom training in improving the knowledge and key maternal neonatal health skills of general nurse midwifery students in Bihar, India: a pre- and post-intervention study. Nurse Education Today. 2016;36: 293-297. doi:.
28. Alnabelsi T, Al-Hussaini A, Owens D. Comparison of traditional face-to-face teaching with synchronous e-learning in otolaryngology emergencies teaching to medical undergraduates: a randomised controlled trial. European Archives of Oto-Rhino-Laryngology. 2015;272(3): 759-763. doi:. [Context Link]
29. Alvarez AG, Dal Sasso GTM, Iyengar MS. Persuasive technology in teaching acute pain assessment in nursing: results in learning based on pre and post-testing. Nurse Education Today. 2017;50: 109-114. doi:. [Context Link]
30. Bai H, Aman A, Xu Y, Orlovskaya N, Zhou M. Effects of web-based interactive modules on engineering students' learning motivations. American Journal of Engineering Education. 2016;7(2): 83-96. [Context Link]
31. Bellhauser H, Losch T, Winter C, Schmitz B. Applying a web-based training to foster self-regulated learning-effects of an intervention for large numbers of participants. The Internet and Higher Education. 2016;31: 87-100. doi:. [Context Link]
32. Chang L-C, Guo JL, Lin H-L. Cultural competence education for health professionals from pre-graduation to licensure delivered using Facebook: twelve-month follow-up on a randomized control trial. Nurse Education Today. 2017;59: 94-100. doi:. [Context Link]
33. Dankbaar ME, Richters O, Kalkman CJ, et al. Comparative effectiveness of a serious game and an e-module to support patient safety knowledge and awareness. BMC Medical Education. 2017;17(1): 30. doi:. [Context Link]
34. Kim H, Suh EE. The effects of an interactive nursing skills mobile application on nursing students' knowledge, self-efficacy, and skills performance: a randomized controlled trial. Asian Nursing Research. 2018;12(1): 17-25. doi:. [Context Link]
35. Lee NJ, Chae SM, Kim H, Lee JH, Min HJ, Park DE. Mobile-based video learning outcomes in clinical nursing skill education: a randomized controlled trial. Computers, Informatics, Nursing. 2016;34(1): 8-16. doi:. [Context Link]
36. Remon J, Sebastian V, Romero E, Arauzo J. Effect of using smartphones as clickers and tablets as digital whiteboards on students' engagement and learning. Active Learning in Higher Education. 2017;18(2): 173-187. doi:.
37. Woodham LA, Ellaway RH, Round J, Vaughan S, Poulton T, Zary N. Medical student and tutor perceptions of video versus text in an interactive online virtual patient for problem-based learning: a pilot study. Journal of Medical Internet Research. 2015;17(6): e151. doi:.
38. Azman AM, Esteb JJ. A coin-flipping analogy and web app for teaching spin-spin splitting in 1H NMR spectroscopy. Journal of Chemical Education. 2016;93(8): 1478-1482. doi:. [Context Link]
39. Basitere M, Ndeto Ivala E. An evaluation of the effectiveness of the use of multimedia and Wiley Plus web-based homework system in enhancing learning in the chemical engineering extended curriculum program physics course. Electronic Journal of e-Learning. 2017;15(2): 156-173. [Context Link]
40. Boada I, Rodriguez-Benitez A, Garcia-Gonzalez JM, Olivet J, Carreras V, Sbert M. Using a serious game to complement CPR instruction in a nurse faculty. Computer Methods and Programs in Biomedicine. 2015;122(2): 282-291. doi:. [Context Link]
41. Davidson SJ, Candy L. Teaching EBP using game-based learning: improving the student experience. Worldviews on Evidence-Based Nursing. 2016;13(4): 285-293. doi:. [Context Link]
42. Fernandez-Aleman JL, Lopez-Gonzalez L, Gonzalez-Sequeros O, Jayne C, Lopez-Jimenez JJ, Toval A. The evaluation of i-SIDRA-a tool for intelligent feedback-in a course on the anatomy of the locomotor system. International Journal of Medical Informatics. 2016;94: 172-181. doi:.
43. Fleming R, Stoiber LC, Pfeiffer HM, et al. Using U-pace instruction to improve the academic performance of economically disadvantaged undergraduates. Journal of Computer Assisted Learning. 2016;32(4): 304-313. doi:.
44. Ijaz K, Bogdanovych A, Trescak T. Virtual worlds vs books and videos in history education. Interactive Learning Environments. 2017;25(7): 904-929. doi:.
45. Kow AWC, Ang BLS, Chong CS, Tan WB, Menon KR. Innovative patient safety curriculum using iPAD game (PASSED) improved patient safety concepts in undergraduate medical students. World Journal of Surgery. 2016;40(11): 2571-2580. doi:. [Context Link]
46. Liu M, McKelroy E, Corliss SB, Carrigan J. Investigating the effect of an adaptive learning intervention on students' learning. Educational Technology Research and Development. 2017;65(6): 1605-1625. doi:.
47. Martin F, Ertzberger J. Effects of reflection type in the here and now mobile learning environment. British Journal of Educational Technology. 2016;47(5): 932-944. doi:.
48. Murphy R, Clissold E, Keynejad RC. Problem-based, peer-to-peer global mental health e-learning between the UK and Somaliland: a pilot study. Evidence-Based Mental Health. 2017;20(4): 142-146. doi:. [Context Link]
49. Nickerson C, Rapanta C, Goby VP. Mobile or not? Assessing the instructional value of mobile learning. Business and Professional Communication Quarterly. 2017;80(2): 137-153. doi:.
50. O'Bannon BW, Skolits GJ, Lubke JK. The influence of digital interactive textbook instruction on student learning preferences, outcomes, and motivation. Journal of Research on Technology in Education. 2017;49(3): 103-116. doi:. [Context Link]
51. Tan AJQ, Lee CCS, Lin PY, et al. Designing and evaluating the effectiveness of a serious game for safe administration of blood transfusion: a randomized controlled trial. Nurse Education Today. 2017;55: 38-44. doi:.
52. Thalluri J, Penman J. Social media for learning and teaching undergraduate sciences: good practice guidelines from intervention. Electronic Journal of e-Learning. 2015;13(6): 455-465.
53. Velan GM, Goergen SK, Grimm J, Shulruf B. Impact of interactive e-learning modules on appropriateness of imaging referrals: a multicenter, randomized, crossover study. Journal of the American College of Radiology. 2015;12(11): 1207-1214. doi:.
54. Virtanen MA, Kaariainen M, Liikanen E, Haavisto E. The comparison of students' satisfaction between ubiquitous and web-based learning environments. Education and Information Technologies. 2017;22(5): 2565-2581. doi:.
55. Achuthan K, Francis SP, Diwakar S. Augmented reflective learning and knowledge retention perceived among students in classrooms involving virtual laboratories. Education and Information Technologies. 2017;22(6): 2825-2855. doi:. [Context Link]
56. Avci H, Adiguzel T. A case study on mobile-blended collaborative learning in an English as a foreign language (EFL) context. International Review of Research in Open and Distributed Learning. 2017;18(7): 45-58. [Context Link]
57. Blau I, Shamir-Inabal T. Digital technologies for promoting 'student voice' and co-creating learning experience in an academic course. Instructional Science. 2018;46(2): 315-336. doi:. [Context Link]
58. Carlson E, Stenberg M, Chan B, et al. Nursing as universal and recognisable: nursing students' perceptions of learning outcomes from intercultural peer learning webinars: a qualitative study. Nurse Education Today. 2017;57: 54-59. doi:. [Context Link]
59. Chaves RO, von Wangenheim CG, Furtado JCC, Oliveira SRB, Santos A, Favero EL. Experimental evaluation of a serious game for teaching software process modeling. IEEE Transactions on Education. 2015;58(4): 289-296. doi:.
60. Choi M, Lee H, Park JH. Effects of using mobile device-based academic electronic medical records for clinical practicum by undergraduate nursing students: a quasi-experimental study. Nurse Education Today. 2018;61: 112-119. doi:. [Context Link]
61. Diliberto-Macaluso K, Hughes A. The use of mobile apps to enhance student learning in introduction to psychology. Teaching of Psychology. 2016;43(1): 48-52. doi:. [Context Link]
62. Djukic M, Adams J, Fulmer T, et al. E-learning with virtual teammates: a novel approach to interprofessional education. Journal of Interprofessional Care. 2015;29(5): 476-482. doi:.
63. Evens M, Larmuseau C, Dewaele K, Van Craesbeek L, Elen J, Depaepe F. The effects of a systematically designed online learning environment on preservice teachers' professional knowledge. Journal of Digital Learning in Teacher Education. 2017;33(3): 103-113. doi:.
64. Gyamfi SA, Gyaase PO. Students' perception of blended learning environment: a case study of the University of Education, Winneba, Kumasi-Campus, Ghana. International Journal of Education and Development using Information and Communication Technology. 2015;11(1): 80-100. [Context Link]
65. Pusponegoro HD, Soebadi A, Surya R. Web-based versus conventional training for medical students on infant gross motor screening. Telemedicine Journal and E-Health. 2015;21(12): 992-997. doi:.
66. Swart R. Critical thinking instruction and technology enhanced learning from the student perspective: a mixed methods research study. Nurse Education in Practice. 2017;23: 30-39. doi:.
67. Schneider AT, Albers P, Muller-Mattheis V. E-learning in urology: implementation of the learning and teaching platform CASUS(R)-do virtual patients lead to improved learning outcomes? A randomized study among students. Urologia Internationalis. 2015;94(4): 412-418. doi:.
68. Sobocan M, Turk N, Dinevski D, Hojs R, Pecovnik Balon B. Problem-based learning in internal medicine: virtual patients or paper-based problems?Internal Medicine Journal. 2017;47(1): 99-103. doi:.
69. Yilmaz EO, Yurdugul H. Design and effects of a concept focused discussion environment in E-learning. Eurasian Journal of Educational Research. 2016;(63): 353-374. doi:. [Context Link]
70. Karabag Aydin A, Dinc L. Effects of web-based instruction on nursing students' arithmetical and drug dosage calculation skills. Computers, Informatics, Nursing. 2017;35(5): 262-269. doi:.
71. Davie E, Martin M, Cuppett M, Lebsack D. Effectiveness of mobile learning on athletic training psychomotor skill acquisition. Athletic Training Education Journal. 2015;10(4): 287-295. doi:.
72. Fernandez-Lao C, Cantarero-Villanueva I, Galiano-Castillo N, Caro-Moran E, Diaz-Rodriguez L, Arroyo-Morales M. The effectiveness of a mobile application for the development of palpation and ultrasound imaging skills to supplement the traditional learning of physiotherapy students. BMC Medical Education. 2016;16(1): 274. doi:. [Context Link]
73. Lanken PN, Novack DH, Daetwyler CR, et al. Efficacy of an internet-based learning module and small-group debriefing on trainees' attitudes and communication skills toward patients with substance use disorders: results of a cluster randomized controlled trial. Academic Medicine. 2015;90(3): 345-354. doi:. [Context Link]
74. Leung JYC, Critchley LAH, Yung ALK, Kumta SM. Evidence of virtual patients as a facilitative learning tool on an anesthesia course. Advances in Health Sciences Education: Theory and Practice. 2015;20(4): 885-901. doi:.
75. Long JD, Gannaway P, Ford C, et al. Effectiveness of a technology-based intervention to teach evidence-based practice: the EBR tool. Worldviews on Evidence-Based Nursing. 2016;13(1): 59-65. doi:. [Context Link]
76. Park JY, Woo CH, Yoo JY. Effects of blended cardiopulmonary resuscitation and defibrillation e-learning on nursing students' self-efficacy, problem solving, and psychomotor skills. Computers, Informatics, Nursing. 2016;34(6): 272-280. doi:.
77. Pourmand A, Tanski M, Davis S, Shokoohi H, Lucas R, Zaver F. Educational technology improves ECG interpretation of acute myocardial infarction among medical students and emergency medicine residents. The Western Journal of Emergency Medicine. 2015;16(1): 133-137. doi:. [Context Link]
78. Rojjanasrirat W, Rice J. Evidence-based practice knowledge, attitudes, and practice of online graduate nursing students. Nurse Education Today. 2017;53: 48-53. doi:. [Context Link]
79. Rolskov Bojsen S, Rader SBEW, Holst AG, et al. The acquisition and retention of ECG interpretation skills after a standardized web-based ECG tutorial-a randomised study approaches to teaching and learning. BMC Medical Education. 2015;15: 36. doi:. [Context Link]
80. Sabatino JA, Pruchnicki MC, Sevin AM, Barker E, Green CG, Porter K. Improving prescribing practices: a pharmacist-led educational intervention for nurse practitioner students. Journal of the American Association of Nurse Practitioners. 2017;29(5): 248-254. doi:.
81. Salajegheh A, Jahangiri A, Dolan-Evans E, Pakneshan S. A combination of traditional learning and e-learning can be more effective on radiological interpretation skills in medical students: a pre- and post-intervention study approaches to teaching and learning. BMC Medical Education. 2016;16: 46. doi:. [Context Link]
82. Steinke EE, Barnason S, Mosack V, Hill TJ. Baccalaureate nursing students' application of social-cognitive sexual counseling for cardiovascular patients: a web-based educational intervention. Nurse Education Today. 2016;44: 43-50. doi:. [Context Link]
83. Bozoglan H, Gok D. Effect of mobile-assisted dialect awareness training on the dialect attitudes of prospective English language teachers. Journal of Multilingual and Multicultural Development. 2017;38(9): 772-787. doi:. [Context Link]
84. Fattah SFESA. The effectiveness of using WhatsApp messenger as one of mobile learning techniques to develop students' writing skills. Journal of Education and Practice. 2015;6(32): 115-127. [Context Link]
85. Franciosi SJ. The effect of computer game-based learning on FL vocabulary transferability. Journal of Educational Technology & Society. 2017;20(1): 123-133. [Context Link]
86. Gokturk N. Examining the effectiveness of digital video recordings on oral performance of EFL learners. Teaching English with Technology. 2016;16(2): 71-96. [Context Link]
87. Hazaea AN, Alzubi AA. The effectiveness of using mobile on EFL learners' reading practices in Najran University. English Language Teaching. 2016;9(5): 8-21. doi:. [Context Link]
88. Shang H-F, Chen Y-Y. The impact of online autonomous learning on EFL students' reading skills. International Journal on E-Learning. 2018;17(2): 227-249. [Context Link]
89. Shih R-C. The effect of English for specific purposes (ESP) learning-language lab versus mobile-assisted learning. International Journal of Distance Education Technologies. 2017;15(3): 15-30. doi:. [Context Link]
90. Wang Y-H. Integrating self-paced mobile learning into language instruction: impact on reading comprehension and learner satisfaction. Interactive Learning Environments. 2017;25(3): 397-411. doi:. [Context Link]
91. Yang L. The effects of L2 proficiency on pragmatics instruction: a web-based approach to teaching Chinese expressions of gratitude. L2 Journal. 2017;9(1): 62-83. doi:. [Context Link]
92. Lu OH, Huang JC, Huang AY, Yang SJ. Applying learning analytics for improving students engagement and learning outcomes in an MOOCS enabled collaborative programming course. Interactive Learning Environments. 2017;25(2): 220-234. doi:. [Context Link]
93. Tsai C-W. Exploring the effects of online team-based learning and co-regulated learning on students' development of computing skills. Interactive Learning Environments. 2016;24(4): 665-680. doi:. [Context Link]
94. Chingos MM, Griffiths RJ, Mulhern C. Can low-cost online summer math programs improve student preparation for college-level math? Evidence from randomized experiments at three universities. Journal of Research on Educational Effectiveness. 2017;10(4): 794-816. doi:. [Context Link]
95. Fernandes Pereira FG, Afio Caetano J, Marques Frota N, Gomes da Silva M. Use of digital applications in the medicament calculation education for nursing. Investigacion y Educacion en Enfermeria. 2016;34(2): 297-304. doi:. [Context Link]
96. Wake D, Dailey D, Cotabish A, Benson T. The effects of virtual coaching on teacher candidates' perceptions and concerns regarding on-demand corrective feedback. Journal of Technology and Teacher Education. 2017;25(3): 327-357. [Context Link]
97. Chao S-Y, Chang Y-C, Yang SC, Clark MJ. Development, implementation, and effects of an integrated web-based teaching model in a nursing ethics course. Nurse Education Today. 2017;55: 31-37. doi:. [Context Link]
98. Gartmeier M, Bauer J, Fischer MR, et al. Fostering professional communication skills of future physicians and teachers: effects of E-learning with video cases and role-play. Instructional Science. 2015;43(4): 443-462. doi:. [Context Link]
99. Foster A, Chaudhary N, Kim T, et al. Using virtual patients to teach empathy: a randomized controlled study to enhance medical students' empathic communication. Simulation in Healthcare. 2016;11(3): 181-189. doi:. [Context Link]
100. Carpenter R, Estrada CA, Medrano M, Smith A, Massie FS. A web-based cultural competency training for medical students: a randomized trial. The American Journal of the Medical Sciences. 2015;349(5): 442-446. doi:.
101. Oner D, Adadan E. Are integrated portfolio systems the answer? An evaluation of a web-based portfolio system to improve preservice teachers' reflective thinking skills. Journal of Computing in Higher Education. 2016;28(2): 236-260. doi:.
102. Desplaces D, Blair CA, Salvaggio T. Do E-learning tools make a difference? Results from a case study. The Quarterly Review of Distance Education. 2015;16(4): 23-34. [Context Link]
103. Nguyen TN. Motivational effect of web-based simulation game in teaching operations management. Journal of Education and Training Studies. 2015;3(2): 9-15. doi:. [Context Link]
104. Millis K, Forsyth C, Wallace P, Graesser AC, Timmins G. The impact of game-like features on learning from an intelligent tutoring system. Technology, Knowledge and Learning. 2017;22(1): 1-22. doi:. [Context Link]
105. Yilmaz O. The effects of 'live virtual classroom' on students' achievement and students' opinions about 'live virtual classroom' at distance education. Turkish Online Journal of Educational Technology - TOJET. 2015;14(1): 108-115.
106. Chyr W-L, Shen P-D, Chiang Y-C, Lin J-B, Tsai C-W. Exploring the effects of online academic help-seeking and flipped learning on improving students' learning. Journal of Educational Technology & Society. 2017;20(3): 11-23. [Context Link]
107. Lameris AL, Hoenderop JGJ, Bindels RJM, Eijsvogels TMH. The impact of formative testing on study behaviour and study performance of (bio)medical students: a smartphone application intervention study. BMC Medical Education. 2015;15: 72. doi:. [Context Link]
108. Pellas N, Kazanidis I. On the value of second life for students' engagement in blended and online courses: a comparative study from the higher education in Greece. Education and Information Technologies. 2015;20(3): 445-466. doi:.
109. Tezer M, Cimsir BT. The impact of using mobile-supported learning management systems in teaching web design on the academic success of students and their opinions on the course. Interactive Learning Environments. 2018;26(3): 402-410. doi:.
110. Carter EB, Wiles JR. Comparing effects of comedic and authoritative video presentations on student knowledge and attitudes about climate change. Bioscene. 2016;42(1): 16-24. [Context Link]
111. Van Lancker A, Baldewijns K, Verhaeghe R, et al. The effectiveness of an e-learning course on medication calculation in nursing students: a clustered quasi-experimental study. Journal of Advanced Nursing. 2016;72(9): 2054-2064. doi:. [Context Link]
112. Thomas RL, Fellowes MDE. Effectiveness of mobile apps in teaching field-based identification skills. Journal of Biological Education. 2017;51(2): 136-143. doi:. [Context Link]
Supplemental References
References for the review articles are available at http://links.lww.com/CIN/A112