As healthcare organizations strive to consistently maximize quality and reliability while minimizing costs, integration of health information technology (HIT) into practice has emerged as an increasingly important strategy. The effects of HIT implementation on clinician workflow and care processes are not benign and if ignored can lead to unintended consequences such as development of workarounds that may potentiate errors as serious as those the technology was intended to prevent.1-4 However, sociotechnical impacts of HIT are seldom considered during implementation or assessed in postimplementation evaluations.1-3 This article presents a framework for evaluation of bar-code medication administration (BCMA) technology that addresses these concerns.
Background
BCMA is a medication administration process that uses bar-code technology to support nurses administering medications by automating the process of the "5 rights": right patient, right medication, right dose, right route, and right time.4,5 Implementation of BCMA technology is associated with significant reductions in medication errors.4-9 However, BCMA implementations do not consistently meet these expectations; medication errors still occur on units with BCMA technology.1,4,5,8,10
BCMA workarounds, also known as deviations from policies and procedures, are well documented in the literature.1,4,8,10-16 When BCMA is implemented, nurses must modify medication administration workflows to meet the demands of the new technology.1,2,14 Nurses commonly compensate for poorly designed processes by relying on workarounds.2 Workarounds may also be a response to operational failures (eg, inefficient resource allocation, equipment malfunction).1,14 Regardless of the cause, workarounds negate BCMA safeguards and impose significant risk to medication administration safety.1 Workflow modifications associated with BCMA implementations can potentiate additional sources of error11-13,15,16 such as increased nurse confusion, decreased communication between physicians and nurses, task omission by nurses under time pressure, and decreased ability to deviate from routine sequences in emergencies.12
To avoid preventable medication errors, efforts must be taken to integrate evidence into practice and minimize negative impacts of BCMA implementation on nursing care.1,13 Maintaining a user-centric approach during design and preimplementation testing is important.17 However, continuous postimplementation evaluation is also needed because (a) the true impact of BCMA on practice can be realized only after implementation1-3,6; (b) practice impact varies based on technology-related, people-related, task-related, organizational, and environmental factors specific to the organization or unit where BCMA is implemented1-3; and (c) practice is a dynamic process that evolves over time.6,10,18 To achieve high reliability in medication administration processes and ensure safety, BCMA use and effectiveness must be continuously evaluated to identify and address workarounds and unintended consequences.4,6,10,14,18
The complexity of healthcare creates significant challenges to evaluation of BCMA.3 Health information technology, such as BCMA, is deployed within a context of multiple sociotechnical dimensions: hardware and software computing infrastructure; clinical content; human-computer interface; people, workflow, and communication; internal organizational policies and culture; external rules, regulations, and pressures; and system measurement and monitoring. Consideration of their collective impact is critical to evaluation of BCMA effectiveness.3 HIT evaluation best practices also emphasize frontline user input.10,17-19 BCMA evaluation programs that are based on the tenets of cultures of safety, ensure frontline staff involvement, aim to improve policy and workflow, address equipment failures, and provide continued end-user education have been shown to improve bar-code scanning rates and are associated with overall reduction in medication errors17 and scanning compliance.10,18
Harrington and colleagues10 developed an evidence-based BCMA checklist to evaluate administrative, clinical, and technological aspects of BCMA policies, procedures, and utilization. This article describes the expansion of their innovative approach into a comprehensive evaluation framework that addresses the degree to which current evidence is integrated into BCMA policies, procedures, and practices; identifies and addresses barriers to best-practice BCMA use; and modifies BCMA practice in concert with changes to the practice environment. Results of the program will be reported in a future publication.
BCMA Evaluation Program
Consistent excellence in healthcare delivery requires effective leadership and a culture of safety that empowers staff at all levels to continuously evaluate practice in order to identify and address safety threats using robust improvement methodologies.20 Based on these concepts, we developed a BCMA evaluation framework and program that provide an infrastructure for frontline nurses to partner with hospital leaders in a systematic process of ongoing evaluation, improvement, and integration of best practices in BCMA use (Figure 1). This program included unit-based medication safety champions who served as BCMA subject-matter experts, remaining current with evidence to ensure integration of newly developed BCMA into unit best practice. They also continually evaluated the medication administration process by directly observing nurses administer medications to identify workarounds and barriers to best practice and worked collaboratively with nurse leaders to track/analyze/address each identified workaround and barrier using the framework described below. Additionally, the BCMA evaluation program included monthly BCMA rounds with the information technology department to identify and resolve equipment-related issues.
Program Implementation
To implement the evaluation program, we assembled a multidisciplinary task force of nurse leaders, pharmacists, and frontline nurses from shared governance unit practice councils (UPCs) to gain widespread buy-in and provide accurate assessments of implementation barriers and facilitators. After obtaining leadership and UPC buy-in, the program was presented to nursing staff at change-of-shift unit huddles and staff meetings and a call for clinical nurse medication safety champions was made. Champions were selected based on expressed interest and clinical performance. Collectively, the champions and implementation task force formed the BCMA evaluation team, which met regularly to evaluate and monitor BCMA use.
Framework
Evidence Identification
The Harrington BCMA Checklist (32 evidence-based yes/no items evaluating administrative, clinical, and technological aspects of BCMA policies, procedures, and use) provided a basis for BCMA best practices.10 Before conducting evaluations, we used the methodology of Harrington et al10 to review the BCMA literature and identify new evidence or population-specific considerations for inclusion in our evaluations. We evaluated all checklist items for relevance to the practice environment and consistency with current evidence. At the time of our evaluations, all checklist items remained current with the literature.
Practice Evaluation
Organizational Evaluation
The 1st step in the evaluation phase was to determine the degree to which organizational policies, procedures, and technological workflows were consistent with BCMA best practice. To do so, we conducted a gap analysis-a process that compared current policies to best practices, identifying gaps or discrepancies.21 Data for this analysis were gathered by obtaining all hospital documents related to BCMA use, including written policies, procedures, practice guidelines, and instructional aids. Documents were reviewed and compared with the Harrington BCMA Checklist10 to identify discrepancies.
Practice Evaluation
The 2nd step in the evaluation phase was to determine consistency between practice and policy and to identify challenges to best practice use by directly observing nurses' use of BCMA to administer medications on all shifts and monitoring BCMA scanning compliance with the hospital-based computer-generated BCMA compliance report. These procedures were intended only to evaluate BCMA use and compliance with best-practice workflows and not to detect medication errors or identify violations to the 5 rights of medication administration. No information linking individual nurses to observations or the identification of medications given was collected.
Before conducting the observations, we informed nurses that these observations would be conducted to evaluate BCMA compatibility with nursing workflows and identify processes that negatively impact the nursing process. To capture observation data, we recorded all steps associated with each observed medication administration and conducted informal interviews with nurses on an ad hoc basis to provide additional information in order to clarify observations. Using this data, we created process maps (Figure 2) (visual diagrams depicting workflows) to enhance understanding, facilitate workflow comparisons, and show the impact of each step on the overall process.21,22 Each process map was compared with the Harrington BCMA Checklist to identify deviations in the BCMA process. Each observed deviation from best practice was defined as a workaround. After analyzing each process map, we described all identified workarounds in detail. All workarounds were categorized based on identified themes. These themes were used to track the frequency of various types of workarounds and evaluate the impact of practice changes on BCMA use.
Performance Improvement
To address observed BCMA practice shortcomings, we engaged hospital leadership and clinical nurses in a collaborative performance improvement process. Using identified discrepancies between organizational policies and defined best practices as a starting point, the team revised policies and procedures to reflect defined best practices, while accounting for the needs of each department and its respective patient population. The team evaluated each discrepancy to identify organizational and population-specific factors that might give rise to a clear mismatch between clinical priorities and defined BCMA best practices. These considerations were addressed in policy revisions to give nurses flexibility to use clinical judgment when administering medications and to clearly define situations in which deviating from BCMA workflows was clinically appropriate.1,10
The next step of the performance improvement process was to determine the causes of observed workarounds. Using the root-cause analysis (RCA) methodology of the "5 whys,"23 we conducted RCAs with frontline nurses to identify workaround causes and barriers to best practice by retrospectively posing a series of "why" questions related to the cause of each observed workaround.23 Factors considered in RCAs included interruptions, process delays, patient-related factors, patient diagnoses, application usability, technology failures, technological workflows, resource adequacy, unit design/layout, bar-code issues, medication preparation, unit rounding schedules, devices, and workflow. We mapped each identified cause to its subsequent effect to create cause-and-effect diagrams (Figure 3). We then superimposed cause-and-effect diagrams to visualize relationships between causes, identify workarounds that occurred as downstream effects of a co-occurring workaround, and identify workaround causes associated with multiple workarounds.
Based on this analysis, we determined the overall patient safety risk and impact on nursing practice of each workaround and its respective causes and prioritized performance improvement initiatives accordingly. High-risk, high-impact workaround causes that contributed to the largest number of workarounds were considered priorities for improvement. We addressed each identified priority through a user-centric approach that employed rapid improvement cycles. Coupling user-centric methods with systematic performance improvement methodologies ensures that solutions are generated by those best positioned to inform process changes; thus this approach has the greatest potential to yield sustainable, effective solutions.18-20,22
Discussion
Organizations cannot simply rely on BCMA technology to achieve desired improvements to medication safety. Instead, structures and processes that enable continual evaluation of BCMA use and impact on nursing practice are needed to ensure adherence to best practice and minimize the potential for error. The BCMA evaluation framework and program described here provide an evidence-based method for frontline nurses to evaluate BCMA use in collaboration with hospital leadership to identify barriers to best practice use. Although this framework was developed specifically to evaluate BCMA, its concepts and structure framework may be generalized to other health information technologies.
Involving frontline nurses in each phase of the BCMA evaluation process is critical to success.10,18,19 As end users of BCMA technology, frontline nurses provide valuable information related to causes of workarounds and challenges to best-practice BCMA use. They are best positioned to inform sustainable user-centric solutions to identified performance improvement needs. Creating an environment conducive to a user-centric, peer-to-peer evaluative process is challenging and requires an organizational culture consistent with a culture of safety. Cultures of safety are achieved through trusting relationships that encourage error reporting to drive visible system improvements.20 By empowering frontline staff to monitor practice and partner with leadership to address identified barriers to best practice, our methodology supports the development of such a safety culture. While this methodology aligns with the tenets of safety cultures, trust remains a prerequisite for program success. Leaders must create a supportive environment and act as facilitators, advocates, and coaches to provide the foundation necessary to instill trust and engage frontline nurses in this BCMA evaluation program.24 Furthermore, evaluations must be grounded in transparency and used to collaboratively identify and remove barriers to best practice.
Although a significant threat to medications safety, BCMA workarounds are valuable in that they provide insight into the shortcomings of technology. When discovered, workarounds should not be met with punitive action, but rather embraced as opportunities to improve care. Most commonly, workarounds are implemented by nurses as solutions to barriers to patient care1,2,13 and are so deeply ingrained into practice that they are neither perceived as workarounds nor seen as safety hazards.1,10 Thus, practice evaluations that use only self-reported methodologies (end-user interviews, focus groups, or event-reporting databases) in the absence of direct observation are unlikely to capture all workarounds. Similarly, evaluation strategies relying solely on computer-generated BCMA compliance reports cannot capture workaround data or provide the sociotechnical context needed to inform BCMA performance improvement initiatives. While BCMA compliance is important when evaluating practice, complementing compliance monitoring with efforts to identify and address sociotechnical technical factors that impede compliance has the greatest capacity to improve effectiveness. Given the complexity of the factors that impact its use, the effectiveness of BCMA is best evaluated from a multimodal systems perspective, which includes policy, procedure, and technology evaluations; direct observations; and end-user interviews.1,3,10,19
BCMA workarounds are caused by system defects and ineffective processes related to technology, people, task-related, organizational, and environmental factors.1 A workaround rarely occurs as an independent event with singular cause; it is far more likely to be associated with multiple upstream and downstream implications.1,3,10 Nurse leaders must consider the collective impact each of these domains has on nursing practice to ensure that BCMA policies, processes, and structures are consistent with best practice and align with demands of bedside nursing practice. Focusing on individual workarounds or causes in isolation precludes the ability to recognize the interrelated nature of workarounds and evaluate relationships among multiple causes of noncompliance.1,3 Moving away from linear causal analyses enables identification of the relationships among all sociotechnical dimensions that affect BCMA use.3 The "5 whys" RCA methodology provides an easy, effective, and efficient method to identify and understand the collective impact of multiple factors on BCMA workarounds.
Ensuring that organizational policies and procedures are consistent with current evidence may be a 1st step to reducing workarounds. Yet, BCMA best practices are not consistently integrated into practice.10 The Harrington BCMA Checklist10 provides an effective measure of best practice integration, as well as a method to evaluate BCMA use while identifying quality improvement needs. However, it does not include items on hardware and software computing infrastructure, human-computer interface, organizational culture/environment, and external rules, regulations, and pressures. Although these factors were considered during causal analyses of workarounds, our methodology is limited in that it does not provide comprehensive evaluation of each of the 8 sociotechnical dimensions of Sittig and Singh's3 Sociotechnical HIT Evaluation Model. More research is needed to determine how each dimension contributes to BCMA workarounds and affects overall BCMA effectiveness. As new BCMA best practices are developed, reevaluation and addition of checklist items will be critical.
Conclusion
Organizations must take a proactive approach to evaluate and improve BCMA through a systematic process that integrates new evidence into practice to ensure medication administration safety.1,4,10,14 This process must be adaptable to changing demands of nursing practice.10 As technology is increasingly integrated into practice, nurse leaders must work collaboratively with frontline nurses to evaluate technology use, identify and remove barriers to best practice, and continually adapt technology to complement nursing practice.
Acknowledgment
The authors thank Susan Rappaport, MSN, the 4 East UPC, and the 4 East clinical nurses for their contributions to the development and implementation of the framework and overall support of the project.
References