Measuring positive youth development (YD) outcomes in a population requires the assessment of both environmental influences experienced by individual young people and the extent to which these young people are experiencing healthy outcomes. Accomplishing this at the state level requires cooperation across numerous state, county, and local information systems, and may also require changes in these systems. This article presents the process used to develop a set of YD outcome indicators for youth-serving systems within New York State (NYS).
Background
Positive YD1-3 is an approach to adolescent health promotion that focuses on environmental supports for positive developmental outcomes and the active role of young people in promoting their own positive development. The Search Institute conceptualizes these two as external and internal developmental assets, or building blocks of healthy development.4
External assets are supports, opportunities, and services in the environment (eg, attentive and responsible mentor outside the family, caring and encouraging school environment), and internal assets are skills, competencies, and experience that the young person has (eg, doing at least 1 hour of homework per school day, placing high value on helping other people). A YD perspective thus expands the range of positive public health outcomes among young people beyond the absence of physical and behavioral health problems, to include civic engagement and community service, preparation for productive employment, and healthy personal and family relationships.
Youth development's emphasis on skill building and preparation for healthy and productive adulthood provides a good platform for pursuing national public health objectives regarding adolescents and young adults, which contain substantial behavioral components (eg, unintentional injury, especially related to substance use and motor vehicle accidents; violence including suicide; and inadequate physical activity).5 However, tracking increases in health-promoting behavior and healthy outcomes, rather than reductions in problems or deficits, often requires new data collection. This is not to say one set of outcomes is better than the other; both are important. But tracking YD outcomes on a population basis will require augmenting existing data collection systems.
Influences on adoption of YD indicators
Before a YD-informed approach can truly gain traction across public health systems, YD indicators need to be measured at state, substate, and community levels. In the words of Arthur Schwartz of the John Templeton Foundation, "What gets measured, gets done." However, simply developing a list of indicators does not assure that they will be used. Several issues are important influences on adoption.
First, users need to understand and begin to adopt YDlanguage and concepts. In many cases, the indicators will be a major influence in orienting new users to an approach that builds developmental assets. As users gradually incorporate more YD concepts into their work, they need support and a chance to periodically reflect. Promoting widespread understanding and application of YD concepts takes time, and it also takes systematic training and technical assistance efforts. Second, users need measurement tools. Cost and capacity are important considerations. Many YD measures are proprietary, thus adding to the costs for potential users. A community survey may provide useful information but a several-thousand-dollar cost can prohibit its use. Similarly, a client-specific assessment instrument that captures youth's strengths but which requires a degreed staff to administer may not be usable by programs without those staff resources. Third, adoption of YD indicators as performance measures requires consensus among public health systems or agencies that YD is an appropriate lens for disease prevention and health promotion and that it matches their organizational mandates. Fourth, the effect of multiple influences in multiple settings (eg, school, family, peer group, program setting) needs to be assessed. No one program addresses all aspects of YD, and program level data, although important, is not sufficient in and of itself. Family members, role models from outside the family, peers, and numerous other players such as teachers, other school staff, and community members also promote the healthy development of young people. The contributions of these figures need to be measured. Fifth, the ability to measure YD outcomes over time is central to a YD approach. The developmental process by its nature occurs gradually. Evolving data collection systems need to begin to incorporate longitudinal data about the consistency of supports, opportunities, and services young people experience on an ongoing basis. Finally, whatever means are used to collect data, training on use of data is important. Good data without good analysis can lead well-intentioned users to inefficient or ineffective decisions.
NYS state-level data systems
New York State has a well-developed information system for the collection and dissemination of statewide child and youth well-being data. NYS Touchstones, a common set of measurable goals and objectives adopted by the Council on Children and Families and its 12-member health, education, and human services agencies, outlines common goals, objectives, and health and well-being indicators for NYS health, education, and human services agencies. The indicators are presented at state, New York City (NYC), rest of state (NYS minus NYC), and county levels. This information, which is compiled from data contributed by the council's member agencies, is available as an annual NYS Touchstones/KIDS COUNT data book, and in an expanded, interactive, and user-friendly format through the Web-based NYS Kids Well-being Indicators Clearinghouse. Currently, the content is mostly deficit based and often reflects deficit-based national health objectives. The snapshot of well-being that these indicators provide would therefore benefit from the identification and inclusion of YD-oriented indicators. Furthermore, the incorporation of YD-oriented indicators into existing data systems would help encourage the statewide adoption of new YD-oriented approaches.
YD indicator use at substate levels
County, program, and community partnership settings are especially important potential users of consensus YD indicators. In NYS, county structure is very important. New York is a state-supervised, county-administered child welfare state; county governments are responsible for planning and operating services and allocating funding from more than 10 state agencies. Counties also have discretionary funding to allocate, and the required plans for state-administered funding streams give counties some flexibility about which outcomes and indicators must be tracked. Across counties, structures of county agencies, resources (fiscal and human), and capacities for planning and data analysis vary significantly. For example, many counties in New York are small and rural, in contrast to the large urban areas such as New York City, Buffalo, or Rochester. As a result, YD outcomes will be adopted differently, and to different extents, across counties. At the time of publication, the process of incorporating YD information into planning and funding decisions is well underway in NYS, and counties are asking for YD indicators and training in how to use them. As YD indicators become available and more commonly used by state agencies and other funders, their use by counties for planning and funding decisions is likely to increase.
New York State's many community partnerships represent another important group of information consumers. The efforts are funded by state, federal, and private organizations around a number of youth-related issues including drugs, teen pregnancy, child abuse, and others; and measurable results are expected from these coalitions. Although some community-level surveys are available now (eg, Communities That Care and Search Institute), these instruments are proprietary and costly to administer. The identification and/or development of credible public-domain measures would make measurement of YD indicators much more accessible on the community level.
At the program level, the collection and use of YD indicator information will be strongly influenced by federal and state funding policies, because programs are most likely to measure what they are paid to do. The move toward more YD-focused indicators will be influenced by the capacity of programs to collect data without unduly burdening staff and impacting service delivery. Presently, there is little consistency across programs in what YD information is collected. A key benefit that would result from consensus YD indicators is an increased ability to aggregate program-level data and use it in concert with other community-level indicators.
Identifying consensus YD indicators
Well-publicized, state-specific YD indicators would help state agencies incorporate YD outcomes into funding and reporting requirements, and would help augment YD-oriented policies (eg, an increased focus on community service and community action by youth, youth leadership, and/or membership on governing boards). Because of the influence of state systems on county, community, and program providers, increased measurement of YD indicators at the state level would influence systems at all levels.
The NYS Youth Development Team (YDT), a partnership of public and private youth-serving organizations in NYS, established a workgroup to develop a short list of strength-based indicators of the well-being of youth to supplement NYS Touchstones data. Intended uses included community planning; statewide planning; grant writing, both by funders and contractors; evaluation of individual programs and broader programmatic initiatives; and outcome monitoring in coordination with national YD-oriented initiatives. The short list was not intended to serve as a condensed YD framework or a full treatment of the topic; instead, it was intended as a consensus list of selected YD indicators that would be useful and credible in a wide range of settings.
Method
To identify a core set of YD indicators, the workgroup began with a review of YD and adolescent well-being indicators from published literature, existing instruments, and sets of state and local indicators from other states. In addition, YDT members nominated indicators. In this initial phase, items were included for consideration regardless of whether they were (1) strengths based, as opposed to risk- or problem focused; (2) measured by existing information systems; (3) linked by empirical research evidence with healthy outcomes; or (4) easily measurable with currently existing instruments. Although the ultimate intention of selecting strengths-based indicators was communicated to all respondents providing input into the process, no a priori decisions were made to exclude items designated as risk- or problem focused from consideration. This decision was made in order to promote statewide consensus among the different stakeholder groups about what types of indicators were sufficiently strengths based to be included in the core set.
Input on content, clarity, and measurability of potential indicators was solicited from workgroup members, YDT members, and national experts in YD from Chapin Hall, Forum for Youth Investment, and the Vermont Agency of Human Services. Ninety-one indicators emerged from this process. Input from varied NYS stakeholders was then used to identify a short list of 15 YD indicators suitable for use by a broad audience.
Respondents included three groups. The first was attendees at a statewide symposium in September 2004 to promote the use of YD indicators across state agencies and counties for planning, evaluation, and program implementation purposes. The 50 participants included staff from New York State, Maine, Vermont, and Connecticut agencies; US Department of Health and Human Services; and Child Trends. The second respondent group was composed of staff at agencies that worked directly with young people ("program providers"), and the third respondent group was composed of young adults (aged 18-21). Input from both these groups was collected via a Web-based survey. Although input from younger teens would also have been desirable, the process of securing parental consent for minors to participate in the survey was not feasible given the Web-based survey format and the resources at hand.
A concept-mapping6,7 exercise was used with symposium attendees to further conceptualize and prioritize the draft set of indicators. Concept mapping is a planning tool that allows groups to generate ideas on any topic of interest, prioritize the ideas, and represent them visually in the form of a map, with more similar concepts located closer together. Symposium attendees sorted the indicators into groups and assigned a name to each group. Attendees also gave each indicator a 1 ("not important") to 5 ("very important") rating in response to the question, "Think about the work you do to promote the well being of youth. Now, rate how important each of the following indicators is for accomplishing that work." Of 50 conference attendees, 41 (82%) completed the concept-mapping task and provided indicator ratings.
Nonrepresentative convenience samples of program providers and young adults (aged 18-21) were recruited to solicit wider input on the indicators. An invitation to rate the 91 indicators by Web-based survey was sent via electronic listserv to YDT members, with a request to forward the invitation to any potential respondents. Informal checks confirmed that the request reached program providers and young adults in a broad range of program types (eg, pregnancy prevention, peer health education) across the state. The program provider question was identical to the one used with symposium attendees. The young adult question read: "Imagine you are a youth program leader, a state leader (government, business, etc), or a community member. You are interested in making sure that young people (aged 12-21) in NYS have a good life and do well. What information would tell you how well young people are doing? Please rate how important each piece of information would be to you."
Data were analyzed using Concept System software.8 Statement-sorting data was analyzed using two-dimensional multidimensional scaling, which produced a map with individual statements in two-dimensional (x, y) space. Statements grouped together more often in the sorting task were located nearer to one another on the map, and statements less frequently grouped together were located further apart. After plotting the statement-sorting data in two dimensions, a hierarchical cluster analysis was performed to identify conceptually and empirically coherent statement clusters. Hierarchical cluster analysis blends agglomerative and divisive methods to identify clusters of observations within a dataset. Cluster configurations were derived using Ward's algorithm9 as the basis for defining a cluster. There is no simple mathematical criterion by which a final number of clusters can be selected; clusters were therefore iteratively examined and judgments were made at each iteration about whether possible mergers improved the thematic and empirical coherence of the clusters. This pattern of judgments resulted in acceptance of a nine-cluster solution. Finally, average ratings were calculated for each statement and cluster of statements.
Results
One hundred and twenty-one program providers and 91 young adults provided 1 to 5 ratings for each of the 91 indicators via a Web-based survey. The program provider ratings were analyzed together with the symposium attendee ratings for 162 program provider and symposium attendee respondents. Of these, 21.0 percent worked directly with youth, 17.3 percent were policy makers, 22.2 percent were directors or managers of agencies, 19.1 percent were program managers, and 20.4 percent specified other roles. Of 160 program providers and symposium attendees who gave a response about gender, 70.0 percent were female and 30.0 percent were male. Among the young adults, 40.7 percent were male and 59.3 percent were female; 17.6 percent were 18 years old, 36.3 percent were 19, 30.7 percent were 20, and 15.4 percent were 21. Of 90 who provided an answer about their race and ethnicity, 55.5 percent were White, 20.0 percent were African American, 5.6 percent were Asian/Pacific Islander, 4.4 percent were Hispanic, 2.2 percent were American Indian/Alaska Native, 10.0 percent specified multiple races or ethnicities, and 2.2 percent specified other races or ethnicities.
Nine indicator clusters were identified in the cluster analysis, and a label for each cluster was derived from the names generated by respondents (Figure 1). Clusters included School Climate, Prosocial Values & Skills, Community & Peer Relations, Family Connections, Safety, Physical Health, Risk Behaviors, Positive Use of Time, and Educational Achievement. In the figure, the height of each cluster's "stack" reflects the proportion of items within that cluster that received mean ratings above the mean of all 91 average ratings. Intergroup correlations in indicator ratings were computed among five groups: policy makers, agency heads, program managers, program providers who worked directly with youth, and young adults aged 18-21. All intergroup correlations were higher than 0.93, indicating a high degree of correlation among groups. Similarly, inspection of each group's rank ordering of the top items revealed a high degree of agreement. No further analysis by group is therefore presented.
Mean ratings for indicators within each cluster, along with the range of ratings within cluster, are shown in Table 1. Indicators related to School Climate received the highest mean rating (4.44), followed by Prosocial Values & Skills with a mean rating of 4.31, Community & Peer Relations and Family Connections with mean ratings of 4.25, Safety with 4.01, Physical Health with 3.69, Risk Behaviors with 3.88, Positive Use of Time with 3.83, and Educational Achievement with 3.38.
To select a list of core indicators, workgroup members identified the 15 highest rated of the 91 indicators (Table 2). One indicator was eliminated because it was redundant with another one in the top 15, and one ("used condoms at their last sexual encounter") was dropped because of duplication with an item on the statewide Youth Risk Behavior Surveillance survey. Finally, the lowest rated remaining indicator ("confident in their ability to accomplish their goals"), which was one of seven items originally selected from the Prosocial Values & Skills cluster, was replaced by an item ("caring relationships with community members") from the Community & Peer Relations cluster, which was regarded as conceptually important but which originally had only one item represented in the top 15.
On the resulting short list, the Prosocial Values & Skills cluster contributed six items: "have goals and aspirations," "say no to things that seem wrong or make them uncomfortable," "learn from their mistakes," "able to problem-solve," "believe standing up for their beliefs is important," and "believe telling the truth is important." Three items were taken from the School Climate cluster: "caring relationships within school," "high expectations in school," and "meaningful participation in school." Two items apiece were drawn from the Family Connections ("caring relationships within family" and "parents have knowledge of youth's activities") and Community & Peer Relations ("caring relationships with peers" and "caring relationships with community members") clusters, and one item apiece was selected from the Safety ("feel safe in neighborhood") and Positive Use of Time ("peers that follow prosocial norms") clusters. No indicators were selected from the Risk Behaviors, Physical Health, or Educational Achievement clusters because, after the item substitutions described above, they did not contain any of the highest rated items.
Discussion
The process of this collective cross-systems work on the YD indicators under the leadership of the YDT brought forth expected issues including the complexity presented by differing definitions of what YD is, the issue of how the goals of different agencies influence their focus on YD, the debate about whether to exclusively attend to positive indicators excluding all deficit ones, and the difficulty of aligning the measurement of indicators of positive development over time with the logistics of annual and targeted funding cycles. This process has also provided fertile ground for open discussions and mutual learning, which support the proposition that there is strong cross-systems and cross-jurisdictional interest in developing appropriate measures of New York's youth well-being and positive YD.
New York State started the process of identifying YD indicators by searching for asset- or strength-oriented indicators that would further help the state's YD work/agenda. As constituents nominated items for consideration, the list grew to include some deficit- or problem-focused indicators. This brought our work to confront one of the philosophical tensions within the YD movement as a whole. Many YD advocates firmly oppose framing youth issues from a negative or "deficit" perspective, whereas others are comfortable using and disseminating a combination of strength-based and problem-focused indicators in creating a picture of young people's well-being.
As expected, YDT members also had a range of opinions about the use of problem-focused data, driven both by personal views and organizational constraints. For example, some partners were committed to YD principles and had included a strengths focus in many of their policies and/or practices; at the same time however, their mandates required them to reduce problems. Many of our state agency partners, for example, have as part of their mandate to reduce problems or risks (eg, criminal and juvenile justice problems, drug/alcohol abuse). Ultimately, to encourage the widest possible range of input on this question (and to avoid becoming embroiled in a debate rooted partly in fixed organizational mandates), we included a wide range of indicators, including some risk- and problem-focused indicators. Through the concept mapping and rating process, we then let our respondents guide us as to which indicators seemed most useful and informative for a broad audience.
Indicators with typical YD content (eg, caring supportive relationships, safety, moral and ethical sense, social "soft" skills, confidence) made their way to the top of the list, despite a large proportion of problem-focused items on the list. Indeed, only one problem-related item was initially among the top 15 (condom use, which was removed because it duplicated a Youth Risk Behavior Surveillance item). Respondents evidently found the YD content of the final short list items appealing. Indeed, some clusters with more traditional content, such as Educational Achievement and Health Outcomes, did not contribute any indicators to the top-15 list.
Some of the indicators selected will be easier to measure than others. The workgroup was not assigned the task of creating instrumentation, and chose to focus on identifying relevant YD constructs, based on the assumption that ongoing work in conceptualizing and testing sound measurement tools will continue to expand the range of YD outcomes that can be validly, reliably, and easily measured. Tools, and training in data collection and analysis, will be needed for the range of end users to reliably and validly measure the YD constructs outlined here.
There is emerging evidence of exciting and innovative work already underway across NYS that supports the use of YD indicators. The YDT's efforts to develop an initial set of YD indicators to complement our statewide Touchstones/Kids Well-being Indicators Clearinghouse database are an important step. Work to promote the statewide understanding and application of YD principles will be needed to further advance this agenda.
REFERENCES