Return to Previous page |
| PDF Version, B&W |  |
Panel 3: Considering effect sizes within the policy context
Panel discussion 1: What is best practice when using Effect Sizes to convey information about prevention and intervention research to practitioners and policymakers?
H. Stephen Leff, Human Services Research Institute
There are two types of research—basic and intervention. Basic research is primarily recognized in publications and is conducted primarily by academics. In intervention research, researchers try to address social policy and find answers to clinical problems of interest to stakeholders, such as policy makers or consumers of services. The following remarks are from a medical and primarily an adult mental health field perspective.
One interesting comment from the discussion regarded targeted outcomes and multiple outcomes. In medical literature, readers are taught to be very skeptical of multiple outcomes. In fact, the argument is made that the outcomes should be formulated in advance as the hypotheses and any additional testing is considered exploratory effects. To the extent that we talk about effect sizes and how one combines effect sizes and so on, we need to be very careful about synthesizing outcomes.
The hypothesis, whether it be an intervention or outcomes, should be theory guided. However, there was very little discussion of theory in the earlier sessions. It is clear that in the testing of medications and medical devices, an important part of the research is to sharpen the hypotheses and the outcomes—not necessarily in the exploratory phase, but as you move to product development. It is important to pay to these details and the context when researching how medications and medical devices are developed. In mental health, it was discovered that the interventions had to be more closely examined to determine the cause of null results. Not only is it important to examine the intervention group, but it is equally important to examine the control group. It is relatively easy to bias and manipulate a study by picking a control group that might be harming people. Another point to reiterate is that an intervention can have adverse events. In mental health, the field concedes that if any intervention is strong enough to do something good, it is strong enough to do something bad. Although adverse events tend to be rare, a longer period of time is generally required to identify them.
An effect size versus natural units is a very interesting and important discussion. However, whether qualitative measures are needed to make an effect size interpretable may not be as important. It seems a mechanism for grouping effect sizes, such as small, medium, and large, can be useful when comparing competing interventions. For example, having an intervention from category of large effect sizes would be better than one that is from the category of small ones. Effect sizes translated into percentages of people that can also be informative. Researchers under appreciate the importance of looking at the percentages for people who must choose among interventions.
Finally, there were two discussions today—one on effect sizes and one on meta-analyses. Meta-analyses using effect sizes are in part necessary because investigators use different outcome measures. If investigators focused on the same outcome measures study results might be synthesized without transforming outcomes into effect sizes.. We may have a “tyranny of policy”, that seeks ways of combining studies to make individual studies more applicable to decisions of all kinds. But there also can be a tyranny of academia that promotes the idea that important scientific studies need not be coordinated through the use of common measures since publication, not application is the primary goal. The FDA process usually supports the use of common measures because application, not publication is a primary goal. The notion of application—trying to determine whether an intervention works or not by having studies focus on the same outcome measures might eliminate the need to transform more direct measures of change into effect sizes. Nevertheless effect sizes are useful tools and the participants in this meeting have made many important points about how to best use them.
| Return to Previous page |