BACKGROUND AND SIGNIFICANCE

Incidence and Consequences of Untreated Neonatal Pain

Infants admitted to the neonatal intensive care unit (NICU) undergo multiple invasive and painful procedures during their routine medical care. A recent systematic review indicated that infants admitted to the NICU experienced between 7.5 and 17.3 painful procedures daily.1 The most recent audit conducted in 14 Canadian NICUs demonstrated that infants receive an average of 5.8 tissue-breaking procedures each week, ranging from 0 to 89 procedures.2 Only half of the 582 infants included in this Canadian study received a pharmacological or nonpharmacological intervention to relieve their pain.2 Furthermore, infants may experience repeated painful procedures or continuous pain with no or limited recovery time between the events.3 This is significant because early pain exposure in preterm infants is associated with immediate- and long-term negative consequences2,4-9 such as increased physiological responses to painful stimuli, blunted behavioral responses, and alterations in neural development.6,10-15

Importance of Accurate Pain Assessment to Prevent Untreated Neonatal Pain

In children and adults, pain is typically assessed through self-report measures. However, this is not possible in the preverbal infant population and health professionals must interpret pain levels by observing contextual, behavioral, and physiological indicators.3 One of the challenges in ensuring the optimal treatment of neonatal pain is the accurate use of assessment tools by healthcare professionals. There has been substantial research on developing pain assessment tools and their validity and reliability.3 However, the Premature Infant Pain Profile-Revised (PIPP-R) tool is the only pain assessment tool that has been evaluated in extremely low gestational age (ELGA) infants within the first days to weeks of age and hospitalization.3

Premature Infant Pain Profile-Revised

The PIPP-R is a composite measure of pain indicators in infants: behavioral (brow bulge, eye squeeze, and nasolabial furrow facial actions); physiological (heart rate and oxygen saturation); and contextual (gestational age and behavioral state).3,16 The 7 indicators are scored on the basis of a 4-point (0-3) Likert-type scale, with possible scores ranging from 0 to 21. Clinical cutoff scores indicate no pain (0) or mild (1-6), moderate (7-12), or severe (>12) acute biobehavioral pain responses. It is a valid and reliable measure for both preterm and term infants up to 44 weeks' postnatal age.3,16 To enhance clinical utility and to incorporate findings of its use with ELGA infants, the PIPP-R was revised in 2010 from the original Premature Infant Pain Profile (PIPP) tool developed by Stevens and colleagues in 1996.3,8,17 The PIPP-R includes the same 7 behavioral, physiological, and contextual pain indicators. Specific changes made to the PIPP-R included changes in the scoring of the oxygen saturation and facial grimacing indicators to improve the feasibility of the PIPP-R for clinical use, and revision of the order of scoring for the contextual indicators of behavioral state and gestational age. These indicators are modifying variables of pain responses rather than contextual variables that are inherent to the infant, thus minimizing the likelihood of overinflated scores based on baseline characteristics of the infant.16

LITERATURE REVIEW

Electronic Learning in the Health Professions

While there has been substantial research and validation of clinical tools used in neonatal pain, educating health professionals to accurately use these measures has been a barrier to pain prevention and treatment. e-Health technology has been identified as one potential avenue for teaching, where learners access the material online either individually or in synchronous courses with others.18-20 The benefits of electronic learning (e-learning) include flexibility,21-23 convenience,18,21,22,24 lower cost,22,23,25,26 and the ability to distribute the content widely.26 While many studies have reported high levels of satisfaction by users of e-learning modules,19,20,22,23,25,26 other authors have asserted the importance of examining knowledge uptake and skills acquisition following e-learning interventions.19,20 In a study that examined the effectiveness of e-learning on nursing assessment skills and knowledge, McDonald and colleagues27 suggested that e-learning alone does not exceed the benefits gained from face-to-face interactions. They recommended a scaffolding approach in which e-learning initiatives were integrated with traditional teaching methods.27

In a systematic review and meta-analysis of studies comparing e-learning interventions with no intervention among medical professionals, Cook and colleagues24 found that e-learning had a significant positive effect on learners' knowledge, skills, and behaviors compared with no educational intervention. In a subsequent review, they reported that feedback, repetition, and practice exercises improved the learning outcomes of participants who used an e-learning tool.21 Using Kirkpatrick's model as a guide,28 Lahti and colleagues19 conducted a systematic review examining the impact of nurses' and nursing students' knowledge, skills, and satisfaction following e-learning interventions. In their review of 11 studies, they found that the e-learning approaches were not statistically significant at improving the knowledge levels of participants when compared with traditional learning methods and that more research was needed.19 Although the body of evidence examining e-learning is growing, many authors have emphasized the need for more research examining the immediate knowledge uptake and long-term skills acquisition and competency following e-learning initiatives19,20,27,29-31 and whether e-learning modules result in a positive impact on patient outcomes.20,32

What This Study Adds

* The development of an effective e-health PIPP-R learning module for teaching pain assessment strategies is a promising avenue to improve competency related to the accuracy and reliability of pain assessments and improve the clinical outcomes of hospitalized infants.

* The use of an e-health learning module to educate nurses on how to use the PIPP-R assessment tool resulted in immediate improvement in competency to accurately assess infant pain, which suggests that e-health technology is effective for improving nurses' skills and competency in pain assessment in the NICU.

* Sustained infant pain assessment competency was lower than immediate competency following training. Further investigation regarding sustained competency and potential need and timing of training updates is warranted.

PURPOSE

Thus, the aim of this study was to evaluate the immediate and sustained pain assessment competency of NICU staff nurses following completion of an e-learning module about the PIPP-R pain assessment tool. The results of this study will guide further refinements to the e-learning module and decisions made on how to optimally use the module as a training tool in clinical practice.

METHODS

Design

This descriptive repeated-measures study is a single-center follow-up study to a previous research project that was developed to evaluate implementation (ie, acceptability, feasibility, usability) and clinical utility of the PIPP-R e-learning module as assessed by 98 nurses from 2 tertiary NICU settings in Canada, The Hospital for Sick Children (SickKids) in Toronto, Ontario, and IWK Health in Halifax, Nova Scotia.33 The study received approval from the Quality Management Department at SickKids and the Research Ethics Board at IWK Health.

Preliminary Work Informing This Study

The larger previous study employed a user-centered design approach3 with 3 phases. In phase 1, between May and September 2015, feedback was collected from clinical and knowledge translation experts at a large tertiary care facility on the module content in terms of clinician learning needs and design preferences. Four experts reviewed the module during individual audio-recorded "think aloud" sessions.34 In phase 2, 22 NICU nurses reviewed the module and completed an 11-item survey on its feasibility, effectiveness, and potential clinical utility.

The initial 2 phases of the study produced a prototype of a self-administered e-learning module with clear prompts, navigation bars, immediate/real-time feedback, and engaging and user-friendly features such as diverse virtual characters, colorful visuals and illustrations, quizzes, and clinical case scenarios. Content included information on the similarities and differences between the original (PIPP) and revised (PIPP-R) versions of the measures, definitions, and illustrations of each behavioral and physiological pain indicator, and directions on how to use the PIPP-R measure to score pain in term, preterm, and ELGA infants. Phase 3 of the larger project sought to evaluate the latest e-learning module prototype in the clinical setting using a mixed-methods convergent approach and following quality improvement methods. The study was conducted at 2 tertiary centers, one in Ontario and the other in Atlantic Canada, with 50 staff nurses from the NICU participating from each site. Additional details on previously published findings are available in Bueno et al.33

Sample

Registered nurses who participated in phase 3 of the larger study described earlier were able to opt in or opt out, through provision of consent, to be contacted to participate in the current follow-up study. Consenting participants were asked to provide data on PIPP-R scoring at 1 week and 3 months following completion of the e-learning PIPP-R module. Nurses were eligible to participate if they were providing neonatal care at the IWK Health NICU, had greater than 6 months of NICU experience, could read and write English, and provided informed consent. The goal was to recruit a minimum of 25 nurses.

Setting

This article reports on data from nurses from IWK Health. IWK Health is a tertiary-level antenatal and pediatric referral center with a 45-bed NICU providing level 3-4 inborn and outborn care.

At IWK Health, the PIPP is the standard pain assessment measure used in the NICU. IWK nurses are educated on how to use the PIPP tool during their orientation training for the NICU. There are plans to switch to using the PIPP-R tool in the unit. There is significant interest in the potential of a self-administered e-learning module, accessible to nurses at any time, to standardize training and clinical application of the PIPP-R measure to improve pain assessment practices in the NICU.

Data Analysis

All data were stored in an encrypted secure file at the study center made only available to the research team. Data were analyzed using the Statistical Package for the Social Sciences (SPSS) 20 (IBM Corporation, Armonk, New York). Demographic data were analyzed using basic descriptive statistics. Categorical data were analyzed by reporting frequency counts and percentages. Differences in nurses' scores from the criterion were calculated for each pain indicator, and frequency counts were done to show which individual indicators were being scored incorrectly.

Procedure

To determine immediate pain assessment competency acquisition, participants were required to view one of 6 possible vignette videos where a neonate was undergoing a heel lance within 1 week of completion of the e-learning module. Participants were then asked to assess and score the infant's pain level using the PIPP-R measure. This procedure was repeated 3 months following the initial 1-week assessment using an alternate video with a different neonate undergoing a heel lance procedure. All videos were previously scored by an independent, experienced, and reliable PIPP-R coder blinded to the study purpose or design. These pretest scores were considered to be criterion for the purpose of the study and used for comparison of participants' scores to determine reliability of score consensus. Consensus between the study participants and the criterion scores was obtained for both time points.

The vignette videos used for scoring pain intensity were obtained during a medically indicated heel lance procedure in the clinical setting. Parental consents were obtained for their use for training, education, and research. To minimize response bias in the scoring of the videos at study follow-up testing, the research assistant provided the case study, initiated the recorded film, and asked each participant to complete a PIPP-R assessment form, with no prompting or reinforcement from the research assistant. If a participant did not completely fill in the assessment form, the research assistant asked the participants to complete all sections. Only the research assistant and research coordinators had knowledge of participant responses, and the research assistant de-identified all responses prior to analysis.

RESULTS

Sample Demographics

Of the 25 nurses who consented to participate in the follow-up study, 25 completed the follow-up at 1 week and 22 completed the follow-up at 3 months. Participant demographics are reported in Table 1.

TABLE 1. Participant Demographics

Competency

At the 1-week follow-up, a majority of nurses scored the video perfectly compared with half of the nurses at the 3-month follow-up. At 1 week, 16% of participant scores differed from the expert criterion scores: 12% by 1 point on the PIPP-R scale, and 4% differed by 2 or more points. At 3 months, 50% of participant scores differed from the expert criterion scores: 27% by 1 point on the PIPP-R scale, and 23% differed by 2 or more points. Of the participants whose scores differed at the 1-week follow-up, all of the participants scored the infant's pain lower than the criterion PIPP-R coder. While a greater number of participants underscored the infant's pain response at 3 months (27% vs 16%), the percentage of participants over scoring pain response (calculating a PIPP-R score higher than the criterion PIPP-R coder) was similar at 3 months (23%). See Table 2 for a complete description of differences in total PIPP-R scores over time.

TABLE 2. Number of Raters With Interrater Exact Agreement in Total PIPP-R Score

When examining the 7 unique pain indicators of the PIPP-R, participants were able to accurately score the physiological indicators (heart rate, oxygen saturation, and gestational age) without error when compared with the criterion scores. In contrast, discrepancies in PIPP-R scores were mainly based on participants' low reliability in scoring the behavioral pain indicators. These included the infant's baseline behavioral state and 3 facial expressions: brow bulge, nasolabial furrow, and eye squeeze. See Table 3 for a description of behavioral pain component scores at 1-week and 3-month follow-ups.

TABLE 3. Raters' Differences in Scores for Subjective PIPP-R Indicators

DISCUSSION

The aim of this study was to examine the impact of an electronic health (e-health) intervention on NICU staff nurses immediate and sustained calculation competency of PIPP-R scores and accurate assessment of infant pain. Competency to perform accurate pain assessment of infants undergoing a medically indicated tissue-breaking procedure was high immediately-1 week-following completion of the PIPP-R e-learning module.

The authors found that immediate knowledge acquisition and accuracy of total scoring were considerably higher at 1 week following completion of the e-learning module than sustained accuracy of pain scoring at 3 months, with the latter decreasing by almost one-third. This suggests that the e-learning module may not promote sustained competency for assessing pain using the PIPP-R. Further research is warranted with a more adequately powered sample to confirm and generalize these results.

Our findings are consistent with others examining the effectiveness of using e-learning to improve NICU care provider knowledge35 as well as online pain resources to potentially improve pain care.32 In a systematic review of 32 studies examining online pain resources, reporting on knowledge, skill acquisition, confidence/competence, and attitudes/beliefs, findings demonstrated that participants receiving online training had higher knowledge than those receiving training through traditional or usual training methods. Of note, while a majority of studies in this review reported knowledge to be higher immediately following online pain training, few studies assessed knowledge at a subsequent follow-up and findings were variable regarding the sustained impact of e-learning on knowledge. Of the 4 studies including follow-up, 2 reported that the participant's level of knowledge was sustained36,37; one reported knowledge scores remained higher than preintervention scores, but it was not clear if the level of sustained knowledge was significant38; and one reported significantly lower scores during a retention test.39 Of the 2 studies reporting on the impact of online resources on attitudes/beliefs and/or confidence/competence compared with those receiving training as usual/alternative training (health professionals and students), no significant differences were found. It is important to note that large variation in rigor, design, and outcomes limited conclusions from this review related to overall impact on patient receipt of care.32

While differences in overall pain scores were small, a decrease in one-third of the participants accurately scoring in a 3-month period would be a clinically meaningful finding, as inaccurate assessment of neonatal pain may lead to inadequate pain treatment, which has both immediate- and long term adverse consequences.40,41 Previous studies using the PIPP or PIPP-R have a 1-point difference in scores to evaluate potential differences in efficacy of pain-relieving interventions in superiority trials of multiple known pain-relieving interventions.42,43 A 2-point difference is considered a clinically meaningful difference when determining effectiveness of pain-reducing interventions when compared with no-treatment controls.44 These authors' findings provide support for the importance of ongoing continuing education of healthcare providers to ensure sustained competency related to infant pain assessment. To our knowledge, while there are no studies examining the immediate and sustained competency of users of different pain tools or other assessment measures in different contexts regarding sustained competency, our findings are consistent with others reporting a decline in sustained knowledge over time using both traditional and e-learning methods.20

When examining the specific components of the PIPP-R score, some components were more likely to be incorrectly scored than others. This was most pronounced at the 3-month follow-up time point compared with at 1 week. Nurses maintained near-perfect scoring on the PIPP-R physiological indicators such as heart rate, oxygen saturations, as well as gestational age at both 1 week and 3 months. However, there was more variability and more incorrect scores on the behavioral indicators such as facial responses of pain and infant behavioral state. These findings likely reflect the objective nature of the physiological findings obtained from monitors and the slightly more subtle behavioral changes that require greater ongoing focused observation of the infant and greater observer's judgment. Moreover, when coding behavioral responses, the nurse needs to provide an estimate of the length of time that the infant was exhibiting the behavioral responses. Therefore, there is a potential higher likelihood of error based on subjective evaluation of behavior and duration. Differences may also reflect other subconscious biases of acknowledging behavioral responses. Sensitivity to behavioral indicators could be lost over time, and there could be a need for continuing education on scoring those over time-this is consistent with literature that asserts that e-learning modules are more effective when coupled with evaluation and follow-up training.45 Given that behavioral indicators of pain have been reported to be the most sensitive with respect to infant pain response,46 the current findings highlight the importance of targeted efforts related to the accuracy of training related to these indicators.

STRENGTHS AND LIMITATIONS

While the current findings provide valuable knowledge to support future interventions aimed at using e-health technology to improve pain assessment and treatment by healthcare professionals, there are some limitations that must be noted. The unit from which study participants were recruited places significant emphasis on the importance of optimal newborn pain prevention and treatment and already uses the PIPP tool, which is the predecessor of the PIPP-R. Given this, nurses may have had preexisting knowledge and competency related to infant pain assessment that could have influenced the results. As this study is a subset of a larger research initiative with a small sample size and was only conducted in one study setting, it is difficult to make statistical conclusions about the effectiveness of the e-learning module or its generalizability. However, the preliminary results can inform future studies that evaluate the clinical effectiveness of using e-learning modules and inform curriculum development and timing of use for education related to neonatal pain assessment. It is also possible that using 2 different videos for scoring purposes at each time point impacted consensus across scores. Another limitation was that we did not measure the amount of exposure the nurses had using the PIPP-R tool between evaluation periods, which may have affected our findings. Finally, in keeping with any study examining outcomes over time in a nonsimultaneous way, it may be possible that a unit contextual factor varied between the 1-week and 3-month study time points. Further examination of the influence of contextual factors such as unit culture, leadership engagement, time, and resources would add to a better understanding of the effectiveness of e-learning tools in the future.

References