Components of the Evaluation
The five study components that comprise the national evaluation are discussed below. Sites are most directly involved in the collection of data for the Cross-sectional Descriptive Study, the Child and Family Outcomes Study, and the Practice Level Assessment. The remaining study components involve data collection that occurs primarily at the system level by the national evaluation team with support from site personnel. Additional information about each component, including data collection instruments used, can be located in later sections of this manual.

Two graphical representations of these components can be found in Figure 1-3 and Table 1-1. Figure 1-3 displays the multi-level nature of the evaluation design and shows how the comparison study comprises all five components. Table 1-1 summarizes, for each component, participants involved and the frequency of data collection activities.

Coming Soon - National Data and Local Data...

Cross-sectional Descriptive Study
The purpose of this study is to describe the children enrolled in the funded system of care in terms of their demographics, functional status, living arrangement, diagnosis, risk factors, and mental health service history. Family demographics, socioeconomic status, and composition will also be described.

These data are collected for all children and families as they enter services in systems of care and comparison sites throughout the program's funding period. In addition, for the children and families who participate in the Child and Family Outcome Study, the descriptive data elements that may have changed over time (e.g., family income legal custody) are collected again at follow-up data collection points.

Child and Family Outcome Study
The purpose of this study is to examine how participation in services in a system of care affects child clinical and functional status and family life. Outcome data on child clinical and functional status will be used to assess change over time in the following areas: symptomatology, diagnosis, social functioning, substance use, school attendance, and performance, delinquency and juvenile justice involvement, and stability of living arrangements. Family life will be assessed in the areas of family functioning, caregiver strain, and material resources.

These data are collected at all sites within 30 days of the child's entry into services and at 6-month intervals for the length of the evaluation. Some sites, in order to reach adequate sample size, will aim to recruit all families receiving services (and thereby participating in the Cross-sectional Descriptive Study) into the Child and Family Outcome Study. Other sites may need to sample families into the outcome study from the larger group in the descriptive study.

Practice Level Assessment
This study component is designed to assess the degree to which system of care values and principles have been incorporated into service delivery and practice. The Practice Level Assessment includes the collection of information on services received through different services sector and satisfaction with services. The Practice Level Assessment is a part of each follow-up interview conducted with families enrolled in the Child and Family Outcome Study. All families enrolled in the outcomes study participate in the Practice Level Assessment, and no additional eligibility criteria must be met for participation in this assessment.

System Level Assessment
The purpose of this study is to examine whether programs have been implemented in accordance with the system of care philosophy and to document how systems develop over time to meet the needs of the children and families they serve. Of particular interest is whether services are delivered in an individualized, family-focused, coordinated manner, and whether the system involves multiple child-serving agencies.

Data collection occurs primarily during site visits by the national evaluation team, which are scheduled to take place every 18 months beginning in the second grant year. Information will be collected through a combination of pre-site visit document review of randomly selected case records, and semi-structured interviews. Categories of respondents will include project directors, agency representatives, direct service providers, case coordinators/managers, and family members.

Services and Costs Study
The Purpose of this study is to describe the types of services used by children and families, their utilization patterns, and the associated costs. The relationship between service and outcomes will also be explored. This study draws on data maintained in fiscal (e.g., charge, billing) management information systems (MIS's) within grantee communities. After an initial mailed survey, a determination is made as to the appropriate level of involvement of each site in this study component. Depending upon the nature of the local MIS's, a site-specific plan will be devised for the transmission of appropriate data. Of interest are the types of services, the combination of services, continuity or gaps in care, the length of treatment, and service and system costs. This information will be analyzed with the data collected from children and family members to create a comprehensive picture of the services and supports families use.

Summary of National Evaluation Team and Local Evaluation Team Responsibilities
The grant program provides for evaluation at national and local levels through a partnership between the national team and the grantees' local evaluators. Data collected through the evaluation will be incorporated into both a grantee-specific and a national assessment of the program.