The Centers for Disease Control and Prevention (CDC)/Macro International Inc Evaluation Capacity Building Team developed an integrated capacity building framework that utilizes a multilevel approach and used this framework to address the multifaceted consideration of building organizational capacity through tailored training, individualized technical assistance, and follow-up. Data collected from 52 organizations that received capacity building assistance using the framework are used to identify factors believed to be critical to improve capacity building. The exploration of this framework also yields many areas fertile for further capacity building research.
In the context of human immunodeficiency virus (HIV) prevention, capacity refers to both the organizational arrangements and the technical capabilities that allow organizations to carry out functions related to HIV prevention and to accomplish the goals of insulating targeted populations from HIV infection or transmission. Although capacity building research offers several perspectives of the concept of capacity building, the literature agrees that capacity building is a dynamic and multidimensional process of organizational change.1
Achieving the desired change involves more than strengthening individual skills and abilities. The effect of formal policies, systems, and practices, as well as informal practices, symbolic actions, beliefs, values, and attitudes, must be understood and integrated into capacity building efforts.2 Facilitating change in the knowledge, skills, and attitudes among individuals can affect the culture of an organization; however, to achieve and sustain change, individuals need an appropriate environment and the proper mix of opportunities in which to use the acquired knowledge and skills in the context of formal and informal organizational systems.3
Organizations use a number of approaches to build their capacity for effectively planning and implementing HIV prevention activities. Common approaches rely on the simple transfer of knowledge and skills through tailoring individualized assistance, training groups of individuals, or providing information through the dissemination of materials and tools. However, knowledge transfer alone is insufficient to change the capacity of an organization, primarily because group trainings or information delivery does not consider the context in which the organization works, and why.3 Tailored individualized assistance, which may result in greater change than from information transfer, can be resource intensive, thereby limiting the number of organizations that can be served. In addition, successful utilization of the individual assistance often requires foundational learning of knowledge or skills as a prerequisite. In response to this dilemma, this article introduces and explores the use of an integrated capacity building framework.
The focus of this article is to describe the feasibility of employing a multifaceted framework that exhibits a three-stage process involving the integration of planning, monitoring, and evaluation in the field and discusses lessons learned from that experience. The "Methods" section explains the framework in detail, presenting an example of the framework used with community-based organizations (CBOs). Data from the 52 organizations that received capacity building assistance are presented in the "Results" section, factors to improve capacity building are identified in the "Lessons Learned" section, and the need for further capacity building research is presented in the "Conclusion" section. The intended audience for the utilization or piloting of this framework includes agency directors, managers, evaluators, consultants, and others responsible for capacity building aspects of HIV prevention programs and services.
Methods
An integrated capacity building framework uniquely uses evaluation planning as the anchor for capacity building. To improve the delivery of health prevention programs and services, organizations commonly focus their capacity building efforts either on program planning and implementation or on evaluation planning and execution. However, the two processes are interdependent in the context of HIV prevention interventions (Figure 1).
The first step in the evaluation planning process is a clear articulation of the program to be evaluated. To develop and execute a monitoring and evaluation plan for a program, an organization should first have common understandings of what will occur in the program and why those activities will be critical for achieving particular results. This process identifies anticipated outcomes, activities that yield those outcomes, and the resources required to implement the activities (ie, a logic model). Determining the appropriate outcomes to effect behavior change requires a comprehensive description of the target population, the behaviors that expose the group to HIV infection and/or transmission, and the determinants that sustain those risk behaviors.
An important step in program (intervention) and evaluation planning is identifying the environmental, personal, and societal factors (or mitigators) that contribute to behavioral risk. As these factors and the relationships between them (linkages) are identified, junctures of determinants (eg, exchange of sex for money, housing, and/or food)-which, if changed or influenced, will affect related factors and, in turn, affect risk behaviors-are also identified (Figure 1, B and C). The strategy of identifying risk determinants provides information on appropriate intervention points for ultimately affecting the HIV risk behavior (Figure 1, B and C). This methodology also specifies the outcomes needed and desired for the targeted group (Figure 1, D). The outcomes inform the combination of activities appropriate for achieving the desired ends, which then allow organizations to assess organizational resources and needs as they identify the resources necessary to engage in the activities (Figure 1, E and F). Collecting and using information (Figure 1, A through F) about the target population, community resources and conditions, and organizational resources and needs is an inherent part of this process.
Define and prioritize needs
Although the need for capacity building can be defined at any time, the point at which an organization begins to systematically assess its needs and resources is the ideal time to engage in the first step of the capacity building process-identification and prioritization of capacity needs (Figure 1, F). Contextual and organizational information must be obtained before plans for capacity building are undertaken, and the assessment must include appropriate organizational representatives (including both leadership and direct services staff) as partners in both the assessment process and the capacity building planning (rather than applying an "expert-to-nonexpert" approach). At this stage, the role of the "expert" or assistance provider is to help the organization conceptualize what it is attempting to accomplish-which informs the capacities that should be enhanced-in the context of the organizational culture and informal systems.
Using the framework, we helped define capacity building needs by interviewing organization staff members, project officers, and program consultants and by reviewing relevant documents. We developed interview guides to collect information about the interventions implemented by each organization, how the organization was structured, and the relationship of HIV interventions to that structure. Information was also collected on each organization's perception of evaluation, evaluation activities, and resources; perceptions of technical and capacity needs; barriers and facilitators to executing evaluation activities; how data were used; and perception of the need for additional resources.
We conducted telephone and in-person interviews with managers and evaluation coordinators of health departments funded by the CDC, followed by group interviews with staff from CDC-funded CBOs, AIDS services organizations, and local health departments. CBO staff members interviewed included program managers and coordinators, as well as direct services staff; separate interviews were conducted with either contractors or staff evaluators. As part of the needs assessments, organizations supplied relevant documents, including logic models or plans, descriptions of interventions, and data collection forms and templates. These assessments served as a reference point for the current functioning of the organization, what they were attempting to accomplish, and their perceptions of what those accomplishments would look like. Through the interviews and document review, we determined relationships between identified needs, the organizations' activities, and desired program and client outcomes.
Analyze and categorize needs
Organizational needs generally fall into three interdependent categories of capacity building: formal systems (infrastructure), resources (knowledge and skills), and informal systems and organizational culture (Figure 2). Formal systems make up the infrastructure necessary to plan, implement, and sustain HIV planning, implementation, monitoring, and evaluation. These systems include fiscal and human resource management systems; diversification of funding, staff recruitment, retention, and training processes; and appropriate policies, procedures, and protocols. Resources are the knowledge and skills of organizational staff and leadership; the availability of tools and processes necessary to carry out prevention program and monitoring activities; and appropriate staff in sufficient numbers, with sufficient time, funding, and materials.5 Informal systems and organizational culture are less tangible and more variable and include the perceptions of organization staff and leadership, motivation for accountability and efficiency, value systems, leadership and management styles, and internal relationships.6
The second step of this capacity building framework is the development of targeted capacity building strategies (Figure 3), which involves a determination of the categories in which the identified needs fall-What are the organization's needs for additional information or skills? What formal processes and systems should be adjusted or established that will allow the skills and/or information to be applied? Will the culture of the organization sustain the changes necessary for the organization to arrive at its desired destination?
Like the assessment, strategy development must also include and engage the organization. Determining and gaining support for appropriate strategies require working both top-down (eg, working with leaders and decision makers to create systemic opportunities) and bottom-up (eg, helping staff members determine the value to them of developing such capacity). Developing strategies also requires understanding what has to be done, how it should be done, and what it will take to build capacity within an organization, as well as addressing the fears and concerns that anticipated change may bring.3 During this process, additional needs will most likely be identified.
Develop and implement strategies
The third step of this framework is the development and implementation of effective capacity building strategies. This framework builds on the values of active participation, learning by doing, and respect for diversity7 by combining strategies that when applied singly will either have only limited effectiveness or may be cost prohibitive (eg, group training, individualized assistance, follow-up, linkages to other capacity building providers). The capacity building team used the needs assessment and analysis data to tailor training provided to groups of organizations with a common need. The training focused on knowledge transfer and skills building and reflected the identified needs of participating organizations. The training provided organizations with a foundation of basic concepts and principles of what they are attempting to accomplish, as well as a common understanding of the organizational changes that may be necessary to accomplish the desired ends. As individuals from organizations gain new skills and knowledge, they, in turn, affect the culture and informal systems of the organization.7
The evaluation training was based on principles of utilization8 and focused on the use of evaluation activities to plan, monitor, and improve HIV prevention programs. Each 2-day training included basic information on evaluation concepts, defining and describing target populations (including risk behaviors and mitigators of risk), generating clear descriptions of interventions (constructing logic models), and developing plans to monitor intervention activities. Emphasis and other topics (eg, instrument development, monitoring client outcomes) were driven by the identified needs of participating organizations.
Each training was followed by individualized assistance that provided organizations with an opportunity to generalize and apply the information and skills to the specific context and concerns of their organization. The capacity building team provided the individualized assistance in the same location in which training was given, over a 2- to 3-day period, with each organization participating in tailored sessions for an average of 4 hours. Capacity building consultants included Macro International Inc staff, CDC staff, independent contractors, and capacity building assistance providers (organizations funded by the CDC to provide assistance to other CDC-funded organizations). This approach capitalized on the lower comparative cost of group training while offering individualized, tailored assistance.
The availability of individualized sessions was discussed with organizations during the assessment stage, and tentative concerns to be addressed were identified. Recommendations generated during meetings with the capacity building team were shared with the organization prior to the training. Knowing the type and content of tailored assistance needed by organizations in advance allowed consultants to prepare for the individualized sessions ahead of time, including assembling or developing additional materials, information, or tools. Additional organizational needs were often identified during the training and clarified during the individualized sessions. Consultants worked with organizations to dissect the emerging needs and generate an action plan to address the needs.
Contingent on a determination of need for additional assistance made by the organization and consultant during the individualized session, follow-up assistance was provided to organizations either directly or through linkages to other providers (eg, infrastructure development capacity building assistance provider, behavioral and social science volunteers). Posttraining assistance took various forms, including electronic or hard-copy information and tools; telephone and e-mail assistance in selecting data analysis software; assistance in the revision of data collection forms and templates; site visits; additional training; review of revised logic models, program plans, and data collection plans; and assistance revising plans. Although only a small number of organizations needed more intensive assistance after the training, participation in the training and individualized sessions helped to clarify their needs, and when compared to the cost of providing onsite assistance to all organizations, the saving was substantial. The capacity building team made follow-up contact with all organizations between 1 and 6 months after the individualized sessions. Using follow-up interview guides, we obtained information on how the knowledge and skills were applied and subsequent changes that had occurred in the organizations. Depending on the defined needs of an organization, follow-up contact was made to develop or revise an action plan designed to build capacity. As organizations applied the knowledge and skills acquired through training and individualized assistance, they were able to reassess and reassign the needs capacity identified initially. In collaboration with organizations, consultants used this information to identify strategies to respond to either newly identified or redefined needs.
Results
Over a 3-year period, 156 individuals from 52 organizations received capacity building assistance through the organizational capacity building framework described. Twenty-seven agencies were CBOs directly funded by the CDC; the remaining 25 were CBOs, AIDS services organizations, and local health departments funded by CDC-funded health departments (state and city). An additional 664 individuals participated in training only. This section provides results from organizations that participated in the tailored training and individualized technical assistance.
Tailored training
A Likert scale was used to assess respondent perceptions of the training and achievement of training objectives. CDC-funded CBO staff members gave the training an overall rating of 4.43 on a 5-point scale (n = 87). Table 1 illustrates their ratings of the accomplishment of the training objectives.
Individualized technical assistance and follow-up
A Likert scale, in combination with open-ended questions, was used to ascertain respondent perceptions of the quality, utility, and user-friendliness of the tailored assistance. Open-ended questions were asked during follow-up to determine how organizations applied skills and knowledge, as well as the organizational changes made subsequent to receiving assistance. Participant responses and follow-up indicated changes in knowledge and skills after receiving assistance and changes within their organizations 6 months later. These changes included changes in the culture of HIV program planning, monitoring, and evaluation within organizations. Respondents reported that staff and leadership were more cognizant of the importance of identifying activities appropriate for achieving the desired outcomes and addressing the needs of their populations, consistent with the mission of their organizations.
On a 4-point scale, the overall rating of individualized assistance was 2.75 (n = 36). All organizations reported that quality, usefulness, and user-friendliness were excellent or good.
On a 5-point scale used to solicit participants' preassistance and postassistance perceptions of health department-funded organizations, perceptions of knowledge and skills increased from a mean of 3.61 to 4.64 (n = 38). Both versions of the assessment asked participants to assess their ability to articulate a clear program plan for their intervention, describe how evaluation data can be used to improve prevention programs, and explain how to access follow-up technical assistance.
At follow-up, organizations indicated that they were either applying or attempting to apply the information obtained. All of the organizations reported beginning to use logic models to describe their prevention interventions. They shared what they learned during the regional training with other staff by conducting teachbacks, reviewing materials, and disseminating copies of training materials. The organizations reported that the training enhanced their understanding of the role and value of evaluation in their organizations and the need for accountability. Having a better understanding of the relevance and context of evaluation led to a change in their motivation and willingness to engage in evaluation activities.
Participating in the individualized sessions prepared participants to use the SMART (specific, measurable, appropriate, realistic, and time phased) model to revise their objectives to be more measurable. Those organizations that had existing logic models were able to revise them to make them more specific. Some organizations revised their data collection forms, others were updating their evaluation plans, and others were considering more systematic methods of managing data. Directly funded health departments realized additional benefits-inconsistencies between interventions for identified populations and state-identified priority populations were replaced by interventions that targeted identified priority populations, resulting in more purposeful approaches to planning.
Lessons Learned
This article described a unique framework for building organizational capacity utilizing a three-stage process involving the integration of planning, monitoring, and evaluation. This framework is a useful planning tool for directors, managers, evaluators, consultants, and others responsible for developing and managing the capacity of HIV prevention programs and services. The findings from this experience suggest that capacity building is not simply the provision of training opportunities and workshops that lead to short-term outcomes. Effective capacity building involves combining strategies to ensure long-term sustainability. Gleaned from this experience are salient lessons.
Identify and implement additional capacity building strategies for long-term sustainability of organizations
Most capacity building strategies aim to make capacity changes at the individual and organizational levels. Training, for example, the most common capacity building strategy, addresses knowledge and skills gaps in both the individual and organization capacity. However, as indicated by the implementation of the framework discussed, providing organizations with opportunities to create usable products, such as an agency program logic model or a monitoring and evaluation plan during training, would make individuals more likely to realize greater benefit from the training. Training must be supplemented with the use of other capacity building strategies and follow-up and support over time to realize the greatest effect in improving the planning and implementation of HIV prevention programs.
Although it is beyond the scope of this article, the impact of systems within which organizations function cannot be overlooked. Table 2 provides examples of capacity building activities that can occur at the individual, organizational, and systems levels.
Create or develop mechanisms to measure changes in capacity
Conceptualizing organizational capacity building as a process implies the need to measure changes in capacity; however, indicators and measurable objectives must be identified to systematically measure such changes. Effective capacity building must involve ongoing, systematic, and planned processes with measurable performance objectives, defined outcomes and indicators, and strategies to track and measure those outcomes over time.4 Although indicators have been developed to measure performance, performance alone is an insufficient measure of the cultural changes that must occur within organizations for them to successfully engage in planning, implementing, monitoring, and evaluating HIV prevention activities. Beyond basic infrastructure needs (eg, governance, fiscal management, personnel management), there is a need to define the environmental context necessary to support successful implementation activities. With the exception of both domestic and international evaluation capacity building efforts, little attention has been given to the development of an integrated set of indicators to measure overall changes in organizational capacity. In addition, indicators would serve to specify the needs and gaps within organizations to prevent diversion of capacity building resources to low-priority concerns.
Table 3 presents examples of indicators, taken from monitoring and evaluation capacity building guidelines,1,4,9 that may serve as a reference point to begin a dialog on the development of integrated organizational capacity building indicators. Similar to the approach described in the framework, the values of active participation of the beneficiaries, learning by doing, and respect for diversity should guide the development of organizational capacity building indicators. General guidelines should also be considered in developing indicators, including keeping the number of indicators to a minimum, linking indicators to a particular goal or objective, and creating SMART indicators so that change can be measured.
Along with a need to move toward the implementation of comprehensive, complex strategies that require appropriate organization structures, skills, and culture, there is a need to ensure that capacity building efforts are targeted, delivered efficiently, and achieve the desired outcomes. These insurances require not only an articulation of indicators but also a consensus building effort to define capacity building.
Foster an internal demand or motivation for evaluation
Organizations realizing the greatest benefits were those that participated as active partners, being motivated to improve the efficacy of their interventions and willing to champion evaluation activities. Organizations mandated to participate in capacity building efforts, regardless of the format, were less likely to provide meaningful input at any phase of the process or execute needed organizational changes. This finding reinforces the notion that although external motivation can be used to "nudge" organizations, they must want to improve and enhance their systems and services before substantial changes in capacity can occur.
Explore more complex levels of building organizational capacity
This article presented strategies for building organizational capacity at the organizational and individual (personnel) levels. However, studies have shown that more complex levels may have an indirect effect on organizational performance and sustainability, such as health systems and community levels.1 The system level refers to the most complex level of capacity building, which illustrates collective resource pooling necessary for the institutionalization of the national HIV/AIDS prevention strategy. Capacity building activities at this level may include national policy making; legal regulatory action plans; management and accountability systems; and partnerships or networks linking national institutions with local agencies. Building capacity at these levels requires a more complex level of coordination of resources, stakeholders, and management systems. Capacity outcomes associated with this level include effective health policies, increased local financing of prevention programs, and development of formal and informal coalitions.
Another level of capacity building that is outside the scope of this article but should be part of the dialog on capacity building is the "demand" side or the community that, in addition to shaping health systems, "should" benefit from and participate in the prevention and healthcare system.1 Because community members, including clients and consumers, are the ultimate beneficiaries of successful organizational capacity building efforts (ie, programs and services effectively meet local needs and bridge identified service gaps), they need to be involved in the dialog.
Conclusion
Organizational capacity building is a strategic methodology or process that serves to enhance organizations and their staff members to perform or carry out their duties better. However, it is a dynamic multidimensional process that is influenced and affected by external and internal factors. A comprehensive organizational capacity building framework with complementing indicators may shed some light on how these factors impede or facilitate capacity building efforts. Well-planned and evaluated capacity building efforts will ultimately lead to improved organizational performance. In this study, capacity building teams identified the appropriate tools and strategies for assisting CBOs in achieving their stated goals and objectives and attaining long-term sustainability. Continual support for the implementation and evaluation of capacity building strategies is necessary to close the gap between those who have access to HIV prevention services and those who do not.
REFERENCES