The purpose of this article is to describe evaluation measures for a capacity-building program for HIV prevention program managers and discuss how the Institute for HIV Prevention Leadership (Institute) curriculum may have increased community-based organization (CBO) capacity. HIV is an important problem that is disproportionately affecting minority communities. It is estimated that between 1,039,000 and 1,185,000 persons were living with HIV in the United States by the end of 2003.1 The AIDS epidemic is not evenly distributed across the population. From 1999 to 2003, AIDS cases increased among blacks, Hispanics, women, and young adults. Blacks accounted for 50 percent of all HIV/AIDS cases diagnosed in 2003.2 In addition, although Latinos account for 13 percent of the population, they comprise 19 percent of new AIDS cases.3 As well, the number of persons exposed to HIV/AIDS through heterosexual contact increased between 1999 and 2003.2
Understanding the social, cultural, and economic context of HIV infection is critical to developing effective prevention programs, and programs developed at the local level should more appropriately capture the linguistic, cultural, and social norms of a targeted community. For this reason, CBOs-one of the principle frontline providers of HIV/AIDS services and prevention programs since the emergence of the epidemic in 1981-continue to hold special promise for reaching communities at risk for HIV, especially minority communities.4 More than 1,450 CBOs have received funding from the Centers for Disease Control and Prevention (CDC) and have provided more than 3,000 HIV prevention programs in the United States and its territories.5 Community involvement in defining and solving health problems is vital to the success of HIV/AIDS CBOs, especially those that serve racial, ethnic, and/or sexual minority populations.6-8 Thus, building community capacity or enhancing CBOs' capacity to address HIV/AIDS within communities hardest hit by the epidemic has become a central focus of current HIV prevention activities.9,10
HIV prevention capacity building is defined by the CDC as "a process by which individuals, organizations, and communities develop abilities to enhance and sustain HIV prevention efforts" with the goal of "fostering self-sufficiency and the self-sustaining ability" of CBOs "to improve HIV prevention programs, processes, and outcomes." CDC capacity-building efforts are currently focused in four areas: strengthening organizational infrastructure; enhancing HIV prevention interventions; mobilizing communities for HIV prevention; and strengthening HIV prevention community planning.9
One project that the CDC currently supports to build capacity of local HIV prevention program managers who work in CBOs is the Institute. In 1998, the CDC funded a needs assessment of the training needs and preferences of HIV prevention program managers working in CBOs. Those data revealed domains of instruction and supported a capacity-building format, whereby trainees would be sent to one national training site several times and be engaged for a 9-month period.11 The Institute was formed on this model and offered its first capacity-building programs to a national cohort of professionals from January 2000 to September 2000. For more on the curriculum development processes and curriculum itself see Richter et al.11,12
Methods
Design and measures
This is a longitudinal study involving measuring scholar capacity at three points in time: pre-Institute, post-Institute, and 6 months' post-Institute. The Scholar Capacity Survey was used to measure scholar and organizational capacity. For the purposes of the Institute's curriculum, and evaluation, capacity was defined as a set of knowledge and skills necessary for managing, planning, implementing, and evaluating effective HIV prevention programs. Capacity was measured across the following domains of skills building: HIV prevention practice (assessment, the use of health behavioral theories, program implementation, and evaluation), the use of strategic planning and human resources techniques, and the use of processes such as advocacy and social marketing to support HIV prevention practice. The Scholar Capacity Survey also incorporated measures of CBO learning environment and culture, as a supportive environment was found to be a critical to capacity building.13,14 Capacity in using electronic communication and information technology applications, although included in the Institute curriculum, was measured using more traditional learner methods, and results are not included here.
The development of the Scholar Capacity Survey instrument was conducted in a collaborative fashion, in consultation with an advisory body and via an iterative process. The advisory body consisted of Institute staff and public health faculty and included both evaluation and community HIV prevention experts. Together, this body drafted the survey instrument, and then made suggestions for further modifications and recommend changes. These changes were then incorporated into the instrument by Institute staff, and it was sent for review again until all modifications were finally agreed upon. The goal of this iterative process was to link Institute content and capacity-building strategies within an evaluative framework.15 More on the Institute's overall evaluation framework can be found in Richter et al.12 Sample questions from the Scholar Capacity Survey are listed by domain below and reflect the content of the Institute curriculum.
HIV Prevention Practice
* Assessing surveillance data
* Assessing attitudes, beliefs, risk, and protective factors of a priority population at the individual, interpersonal, community, and environmental levels
* Setting program goals that seek to reduce the priority populations' rates of HIV/AIDS cases
* Setting program objectives that directly impact the priority populations' attitudes, beliefs, and knowledge about HIV, and behavioral and nonbehavioral risk and protective factors
* Using behavioral theory in planning HIV prevention programs
* Developing interventions at individual, group, community, and institutional/systems levels
* Collecting process evaluation data to determine stakeholder involvement, representation of priority population, fidelity to implementation plan, and participant value of an intervention
* Collecting outcome data on participant attitudes, beliefs, and knowledge of HIV, and behavioral and nonbehavioral changes to risk and protective factors
* Collecting impact evaluation data on HIV infection and quality of life of priority populations
Strategic Planning and Human Resources Management
* Reviewing agency's mission statement
* Conducting environmental scan and situational analysis of the organization
* Involving key stakeholders in the strategic planning process
* Prioritizing a strategic issue and developing an action plan for a strategic issue
* Conducting human resources management activities
Processes That Support HIV Prevention Practice
* Meeting with elected and nonelected officials to discuss important HIV-related legislation and/or policies
* Becoming involved with policy development and policy advocacy
* Participating in HIV-related policy task forces
* Developing social marketing-based approaches to HIV prevention
The CBO Environment/Learning Culture
* Attending trainings
* Sharing knowledge and skills learned at trainings with others in the CBO
* Seeking advice from experts
* Assessing and providing feedback on internal HIV prevention activities and policies
* Having formal and informal mechanisms to share new information and get feedback on new ideas
* Having the tools and technology to support information exchange
For all questions related to HIV prevention programming and the processes that support HIV prevention practice, scholars were asked to rate the frequency with which they performed certain tasks or actions by selecting a score from 7 (always) to 1 (never). To measure the frequency with which strategic planning activities and human resource management activities were performed, scholars were asked to select the interval (ie, once every 3 months, once a year) that best represented how often they conducted each activity at their CBO. Scholars were also asked to rate their level of confidence in their ability to perform these same tasks and actions in all domains. A 7-point scale was used, ranging from 7 (completely confident) to 1 (not at all confident). To measure organizational learning environment, scholars were asked to rate the frequency with which certain activities occurred within their CBOs. Again, a 7-point scale was used, ranging from 7 (always) to 1 (never). To measure organizational learning culture, scholars were asked to rate their level of agreement ranging from 7 (completely agree) to 1 (completely disagree), with statements about their CBOs' learning culture.
Scholars completed the first measurement (T1) on the first day of week 1 of the Institute. The post-Institute measurement (T2) was conducted onsite at the Institute at the end of the last day of instruction (N = 105). For the third measurement, T3, these same scholars from the Institute were mailed a copy of the survey with a self-addressed return envelope 6 months' post-Institute completion. Only responses from scholars who completed all three surveys are included in this final analysis of the data (N = 94), representing an 89.5 percent response rate.
Analysis
The dependent variables in the analyses were (1) frequency of performing a given activity and (2) level of perceived self-efficacy. The independent variable was time (pre-Institute to post-Institute to 6 months' post-Institute). Most data were reported as scale scores. Frequency and confidence scales were derived for each domain by adding scholars' responses to all the frequency questions and all the self-efficacy questions separately. Repeated-measures analysis of variance was used to test differences between T1, T2, and T3. When several measurements are taken on the same person over time, the measurements tend to be correlated with each other, especially when these measurements can be thought of as responses to levels of an experimental factor of interest (in this case, the Institute). This correlation is taken into account when using a repeated-measures analysis of variance. Bonferroni multiple comparison method was used to find the difference between two points in time, and allowed us to better see the trend over time. All data were analyzed using SPSS 12.0 for Windows.
Some data were not collected as scale scores (strategic planning frequency and human resources management frequency). Consequently, they are reported descriptively as frequencies in this report and are compared to what Institute developers felt was the ideal time frame in which to perform each task. For most of the activities assessed, the preferred time period in which activities should be conducted is "every six months to once a year."
Results
Demographics
Table 1 presents the demographic characteristics of the 94 scholars who completed all three measurements of scholar capacity during the years 2002-2004 at the time of their matriculation into the Institute. Most respondents, as well as Institute participants, were female (59.6%) and from a racial/ethnic minority population (78.7%). Just over half (52.7%) had pursued educational opportunities beyond a bachelor's degree, whereas 24.7 percent had earned a bachelor's degree; the remaining 22.6 percent had a high school diploma or some college experience. On average, the scholars had worked just over 6 years in HIV prevention and had worked at their current CBOs for just under 5 years. However, as indicated by the wide range of responses, others had considerably more experience in both HIV prevention (up to 18 years) and within their current organizational environments (up to 27 years). All scholars were from minority-based organizations (data not shown in the table).
HIV prevention practice
Table 2 illustrates the changes between T1, T2, and T3 in the frequency of conducting and perceived self-efficacy in conducting tasks associated with HIV prevention programming in the areas of community assessment, writing program goals and objectives, using health promotion theory, developing and implementing interventions, and conducting process, outcome, and impact evaluation. A higher scale score in frequency or confidence in performing a given task indicated that Institute participants were more likely to perform each activity when designing an HIV prevention intervention and are more confident in performing those tasks. Higher scale scores indicated that scholars were more likely to follow evidence-based public health prevention practices in developing HIV prevention interventions.
Over the course of the Institute, and continuing 6 months beyond Institute participation, scholars significantly (P <= .001) increased the frequency and confidence with which they performed activities related to all domains of HIV prevention programming including community assessment, writing program goals and objectives, using health promotion theory, developing and implementing interventions, and conducting process, outcome, and impact evaluation. For most activities, these changes initially occurred during the Institute and were sustained at both post-Institute measures. Frequency of performing community assessments and goal and objective writing continued to significantly (P <= .05) increase during the 6-month period immediately following Institute participation.
For most of the activities related to the planning, implementation, and evaluation of public health prevention programs, at T1, scholars reported scale scores in the upper middle part of the scale range. These translate into scholars performing each activity "sometimes" to "often" when developing a new HIV prevention intervention. Likewise, at T1, scholars' mean confidence scores for most of these activities fell in the upper middle part of the scale range, and translate to feeling "somewhat confident" in performing these activities. In terms of using public health theories and methods, at T1, scholars' mean scale scores fell in the middle of the scale range. These mean scale scores translate to using theory only "rarely" to "sometimes" when planning an HIV prevention intervention and feeling "neither confident nor not confident" in using theory.
For all activities, significant change (P <= .05 for all measures and P <= .001 for 12 of 14 frequency and confidence measures) was seen between T1 and T2 (pre- and post-Institute). By T2, scholars' mean scale scores had increased, and these increases translate into scholars performing each activity (with the exception of using theory) closer to "often" when developing an HIV prevention intervention and feeling "mostly" confident in doing so. Scholars' mean scale scores for using public health theory increased as well, moving toward using theory "sometimes" and feeling "somewhat confident" in doing so.
Strategic planning and management
Table 3 details how frequently strategic planning activities were conducted by Institute scholars at T1, T2, and T3. Between T1 and T3, there was considerable movement in how often strategic planning activities were conducted. For the seven activities assessed, the preferred time period in which activities should be conducted is "every six months to once a year." Across all activities, more scholars reported performing the activity at the preferred time period at T2 than at T1, and for six of the seven activities these results were sustained at T3.
At T1, 40.4 percent of scholars reported reviewing their mission statement "every six months to once a year." By T2, 59.6 percent of the scholars reported conducting this activity during the preferred time period. At the start of the Institute, 41.5 percent of scholars "never" or "every two years or more" conducted environmental scans to assess external threats and opportunities. Another 15 percent did not respond to the question. Post-Institute, 58.6 percent of the scholars reported conducting environmental scans "every six months to once a year." Similarly, 37.2 percent of scholars at T1 conducted a situational analysis to identify internal threats or weaknesses at the most appropriate time interval. However, at T2, 59.6 percent of scholars reported conducting a situational analysis at the most appropriate time interval.
Before participating in the Institute, 43.6 percent of scholars "never" or only "every two years or more" involved key stakeholders in their strategic planning processes. By T2, 47.8 percent reported involving their key stakeholders on a more regular basis. Scholars' self-efficacy in performing these strategic planning and management functions increased over the measurement period, with most of the change occurring from pre- to post-Institute. Scholars moved from being less than "somewhat confident" pre-Institute to more than "mostly confident" at both post-Institute and 6 months' post-Institute (data shown in Table 2).
Table 4 details how frequently human resources management activities are conducted by Institute scholars at T1, T2, and T3. For the seven activities assessed, the preferred time period in which activities should be conducted is "every six months to once a year." For four of the activities, most scholars reported performing the activity at the preferred time period at T2, and, when compared with T1, had changed the interval at which these activities were performed to better reflect what was taught at the Institute. More scholars were also conducting formal job analyses, formally assessing employee training needs, evaluating employee performance, and reviewing or implementing employee performance reward or incentive programs at the recommended intervals. Scholars' self-efficacy in performing human resources management activities increased over the measurement period, with most of the change occurring between pre- and post-Institute. Scholars moved from being "somewhat confident" pre-Institute to "mostly confident" post-Institute and 6 months' post-Institute (data shown in Table 2).
Processes that support HIV prevention practice
Table 2 also illustrates the changes between T1, T2, and T3 in the frequency of and perceived self-efficacy in using advocacy and social marketing techniques as a way to enhance HIV prevention programming efforts and the management of those efforts. As with the HIV prevention programming activities, a higher scale score in frequency and/or confidence in performing a given task may indicate that Institute participants were more likely to perform each activity on a regular basis. Over the course of the Institute, and continuing 6 months beyond Institute participation, scholars significantly (P <= .01) increased the frequency and confidence with which they performed activities related to both advocacy and social marketing. The frequency with which scholars performed social marketing activities also significantly (P <= .05) increased in the 6 months following their Institute participation.
The CBO environment
Scholars also reported changes in their CBO learning environment and culture over the course of the Institute. The frequency with which scholars and their CBOs' HIV prevention staff performed activities related to improving the CBO learning environment such as attending trainings, sharing knowledge and skills learned at trainings, seeking advice from experts, and assessing and providing feedback on HIV prevention activities and policies significantly (P <= .001) increased from T1 to T2 and T3. In addition, their level of agreement that their CBO presented a positive learning culture significantly increased as well over this time period. The overall trend for the entire T1 to T3 time period was significant (P <= .001), with most changes occurring during the course of the Institute (T1-T2).
Discussion
Results of this study indicate that the effect the Institute has on the capacity of participants to conduct more effective HIV prevention programs as well as manage and sustain those programs at the program level is positive and significant. Moreover, Institute participants were more confident in their ability to perform activities related to planning, implementing, and evaluating HIV prevention programs, managing both program and human resources operations, and conducting advocacy and social marketing processes. Scholars also expressed positive changes in their CBO learning environment and learning culture.
That these changes occurred during the 9 months of Institute participation and were sustained (and in some cases improved upon) 6 months afterwards is impressive. Many capacity-building programs fail to evaluate retention of knowledge and skills among participants after they leave the program. The evaluation model of the Institute is unique in that it obtained 6-month post-Institute measures. The Institute maintains an electronic scholar resource room on the Internet as well as a scholar/alumni list-serve and conducts alumni seminars. These systems of support may help to maintain and reinforce improved capacity.
Although the data presented herein are not representative of all persons working in HIV prevention, they suggest that more widespread capacity-building programs for persons working on the planning, implementation, evaluation, and management of HIV prevention programs would be helpful. A limitation of these data is that the response rate varies among scales, ranging from 89 percent to 98 percent. For example, all questions on the theory frequency and confidence scales were answered by 79 of the 94 scholars (89%). It is unclear why persons chose not to answer these questions. However, Institute process evaluation data indicate that faculty covered the stated learning objectives, that the level and intensity of instruction was appropriate, and that the material covered during Institute sessions was learned by scholars.12
Current research findings suggests that self-efficacy16 is one of the key antecedents of learning transfer within organizations.17-20 High mean scores of self-efficacy provide evidence that scholars were able to transfer the skills learned at the Institute to their jobs in their CBOs. On a national level, these capacity increases may translate into more efficient and effective HIV prevention. Scholars are following evidenced-based practices in public health to a greater degree and are doing more to secure funding, as well as advocate for their programs and CBOs. Future research should investigate transfer of learning to others in the CBO, as this transfer would also contribute to enhancing the capacity of the CBO. At the organizational (CBO) level, these changes have translated to a more positive learning culture that is supportive of active learning processes.
REFERENCES