Previous |
GUIDELINES FOR DATA REPORTING TO FACILITATE META-ANALYSIS
This document is one of two products created under the auspices of the Evaluation Data Coordination Project (EDCP)—options documents providing measures for a number of constructs and informational papers describing guidelines for data reporting and the release of public-use data. The EDCP’s overarching goal is to develop common measures of constructs and reporting formats for selected evaluation projects to facilitate and improve the quality of potential future secondary analyses and cross-project syntheses. In this document, we provide guidelines for reporting findings to facilitate meta-analysis, secondary analysis, and cross-product synthesis. The evaluators of the nine U.S. Department of Health and Human Services/Administration for Children and Families (DHHS/ACF) studies are the primary audience; however, the information is applicable to a wide range of research and, therefore, to a broader audience. (For more information about the EDCP and the nine ACF studies, refer to the Options Document.)
This document is divided into two parts: recommendations for the type of information to include in evaluation reports and an illustration of how such information should be reported with an example study.
I. RECOMMENDATION OF INFORMATION TO INCLUDE IN REPORTS
Meta-analysis is a technique applied to examine the effects of a given treatment, program, or intervention on specific outcomes, using findings reported from multiple studies. To conduct a meta-analysis, certain types of information must be available from each of the studies. Drawing upon Lipsey and Wilson (2001) and American Institutes for Research’s work on the What Works Clearinghouse Project, we have developed a series of questions that illustrate the types of information needed to conduct a meta-analysis. These questions can be used as a checklist for ensuring that relevant data are included in reports (see table 1).1
The questions listed below fall into three categories. The first set is about the sample and research design of the study. The second set addresses the characteristics of the treatment examined in the study. These questions are intended to elicit information about the characteristics of the study; that is, information that will help readers interpret the nature and magnitude of the findings. Information related to these two sets of questions is usually provided in the methods section of study reports. The third set is intended to elicit information about the empirical findings of the study. Researchers must provide adequate information when reporting their findings so that the effect size for a particular treatment—the focus of the meta-analysis—can be calculated.
I. Sample and Research Design |
|---|
| 1. What was the age of the subjects? 2 |
| 2. What was the racial composition of the sample? |
| 3. What percentage of the subjects is male? |
| 4. What was the sample size of the treatment group at the start of the study? |
| 5. What was the sample size of the control group at the start of the study? |
| 6. What was the research design? |
| 7. Was the unit of assignment the same as the unit of statistical analysis? |
| II. Treatment Characteristics |
| 8. In what setting did the treatment take place? |
| 9. What was the duration of the treatment in days? |
| 10. What was the intensity of the treatment? |
| 11. What was the nature of the control group? |
| III. Effect-Size Data (To Be Reported Separately for Each Outcome of Interest) |
| 12. What is the outcome (e.g., standardized test, attitude)? |
13. How was the outcome measured (e.g., Likert scale, ranging 1-5)? |
| 14. What was the basis of the effect size? |
| 15. What was the sample size for the treatment group for the analysis of this outcome measure? |
| 16. What was the sample size for the control group for the analysis of this outcome measure? |
| 17. What was the treatment group mean on this outcome measure? |
| 18. What was the control group mean on this outcome measure? |
| 19. What was the treatment group standard deviation on this outcome measure? |
| 20. What was the control group standard deviation on this outcome measure? |
| 21. What was the level of significance? |
II. ILLUSTRATION WITH AN EXAMPLE STUDY
This section provides an illustration of the pertinent information needed to conduct a meta-analysis. This example is drawn from a study of the impacts of Early Head Start, a two-generation program designed to provide high-quality child and family development services to low-income pregnant women and families with very young children (DHHS, 2002). This evaluation involved 3,000 children and families in 17 Early Head Start programs. Half of these participating children received Early Head Start services, while the other half were randomly assigned to a control group that did not receive Early Head Start services. Parents and children were assessed at three different points while enrolled in the program, and families were interviewed about their use of Early Head Start services when they exited the program. The 17 participating Early Head Start sites were selected to represent the wide variety of programs with respect to geographic region, racial-ethnic status, urban-rural location, program auspice, and program experience in serving infants and toddlers.
| I. Sample and Research Design | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1. What was the age of the subjects? Children averaged 5 months at enrollment. They ranged from being unborn (up to 4 months prenatal) to 12 months upon enrollment; 39 percent of mothers were less than 20 years old at birth of focus child. Child outcomes were assessed when they were 14, 24, and 36 months; Parents’ well-being and parenting practices were also assessed at those times. Service-receipt and self-sufficiency data were collected at 7, 16, and 27 months after random assignment. |
||||||||||||||||||||
| 2. What was the racial composition of the sample? White Non-Hispanic: 37 percent of families Black Non-Hispanic: 35 percent of families Hispanic: 24 percent of families Other: 4 percent |
||||||||||||||||||||
3. What percentage of the subjects is male or female?
|
||||||||||||||||||||
| 4. What was the sample size of the treatment group at the start of the study? 1,513 |
||||||||||||||||||||
| 5. What was the sample size of the control group at the start of the study? 1,488 |
||||||||||||||||||||
| 6. What was the research design? Random assignment |
||||||||||||||||||||
| 7. Was the unit of assignment the same as the unit of statistical analysis? Yes, the family |
||||||||||||||||||||
| II. Treatment Characteristics | ||||||||||||||||||||
| 8. In what setting did the treatment take place? In 17 Early Head Start Programs, which offered either center-based care, home visiting, or a mix of the two. Programs were located in settings that ranged from small rural communities to large urban areas and were located across all 10 DHHS regions. |
||||||||||||||||||||
| 9. What was the duration of the treatment in months? Treatment group families participated in Early Head Start, receiving a variety of services, for an average of 21 months, with nearly half of the families participating for at least 2 years. |
||||||||||||||||||||
| 10. What was the intensity of the treatment? The intensity varied depending on the type of service received. Early Head Start Center-Based Child Care: Received 1,391 hours of Early Head Start center care (about 12 hours per week) Case Management Services: More than 80 percent of program families reported meeting with a case manager, and almost all of these reported more than one meeting. Nearly three-quarters of families reported meeting with a case manager monthly or more often during at least one follow-up period. |
||||||||||||||||||||
| Home Visits: 87 percent of program families received at least one home visit. Nearly three-quarters of program families received home visits at least monthly during at least one follow-up period. | ||||||||||||||||||||
| 11. What was the nature of the control condition? Free to assess similar programs and services in the community. |
||||||||||||||||||||
| III. A. Social-Emotional Development | ||||||||||||||||||||
|
||||||||||||||||||||
| III. B. Receipt of Any Child Care |
||||||||||||||||||||
|
||||||||||||||||||||
| III. C. Impacts on Emotionally Supportive Parenting |
||||||||||||||||||||
|
||||||||||||||||||||
| III. D. Impacts on Family Income, Resources, and Subsequent Childbearing |
||||||||||||||||||||
|
Study Title : Making a Difference in the Lives of Infants and Toddlers and Their Families: The Impacts of Early Head Start Volume I: Final Technical Report. Full report available at http://www.acf.hhs.gov/programs/opre/ehs/ehs_resrch/index.html
For this example study (DHHS, 2002) , the researchers provided detailed information about sample and research design and treatment characteristics (sections I & II of table 2) in the methods section and outcomes measured and effect-size data (section III. A–III. D of table 2) in the results section. These types of information are indispensable to high-quality meta-analyses based on this study and studies evaluating similar interventions. As Cooper (1998) notes, incomplete reporting may pose serious problems to the meta-analyst. Reports missing information on statistical outcomes may prevent the meta-analyst from estimating accurately the magnitude of the treatment effect, and reports missing information on study characteristics will prevent the meta-analyst from investigating how study outcomes are related to study characteristics. In the absence of the basic information as outlined here, the meta-analyst will have to downweight or exclude the study even if the study was methodologically impeccable, such a downweighting or exclusion is likely to compromise the validity of the research synthesis.
REFERENCES
Cooper, H. (1998) Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications, Inc.
Santa, C. & Hoien, T. (1999). An assessment of early steps: A program for early intervention of reading problems. Reading Research Quarterly, 34, 54-79.
U.S. Department of Health and Human Services. (2002, June). Making a Difference in the Lives of Infants and Toddlers and Their Families: The Impacts of Early Head Start Volume I: Final Technical Report. Washington, DC: Author. Available at http://www.acf.hhs.gov/programs/opre/ehs/ehs_resrch/index.html
1 The generic questions serve as the basis for developing treatment-specific coding guides. Depending on the nature of the treatment, more information may need to be reported for the relevant studies, and the wording of the questions may need to be revised to reflect the type of treatment under study. (back)
2 This illustration provides example of some sample characteristics of relevance. Depending on the study, other relevant characteristics may be important to include (e.g., income). (back)
| Previous |