ACF’s evaluation policy establishes five principles: rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. This Teen Dating Violence Awareness and Prevention Month, we share how we balanced two of those principles -- rigor and ethics -- when collecting data from individuals affected by domestic violence and dating violence.
We aimed to pursue both rigor and ethics during the design and execution of a study of the National Domestic Violence Hotline (The Hotline) and loveisrespect (a help line targeted towards young people). The Hotline and loveisrespect offer crisis intervention and support to contactors by helping identify problems, priorities, possible solutions, and options, including making plans for safety and a plan of action. They offer information about resources, referrals, and direct connections to service providers. The evaluation of The Hotline and loveisrespect (conducted by the Milken Institute School of Public Health at The George Washington University) aimed to:
- describe the services and resources provided to those who access The Hotline and loveisrespect and identify whether the information and assistance received is helpful and sought after;
- document trends and patterns in the mode of service provision (i.e., calls, online chats, texts, and websites); and
- measure short-term outcomes of contacting The Hotline and loveisrespect and assess the level of helpfulness of the services provided.
From a rigor perspective, we wanted to collect useful and high-quality data. Specifically, we wanted to collect a representative sample and collect identifying information to link data sources. Our approach to collecting data involved following up with survivors, friends and family of survivors, and service providers who contacted The Hotline or loveisrespect for help, both immediately after their initial contact and two weeks later. Ideally, we would follow up with a representative sample of those who contacted The Hotline or loveisrespect and link their immediate responses to their follow-up responses two weeks later.
From an ethical perspective, we were concerned for the safety of all contactors, and survivors in particular. We wanted to be sure that perpetrators of violence would not be able to find out that survivors were contacting The Hotline or loveisrespect. We also wanted to make sure that those who were in crisis and/or needed immediate referrals to additional services were not asked to participate in a survey. And we wanted to protect the privacy and confidentiality of all of our study participants.
The study team arrived at the following approach to balance these goals. We decided to include only contactors who were not being directly referred to services. This did not allow us to study a representative sample, but it prioritized the safety of the contactors and their ability to get the services they needed. We also did not collect any identifying information from participants. Instead, we asked individuals to provide three pieces of information: any four-digit number, a pseudonym, and a code word. Participants could use the code word as a safe word to alert the research staff member that they did not feel safe to complete the survey. To match the immediate survey to the follow-up survey, the research team needed two of these three pieces of information. This information from participants allowed us to only match a percentage of the responses from participants two weeks after contacting The Hotline or loveisrespect. We did offer participants the option of providing a contact telephone number so that the study team could text or call to remind them to complete the follow up survey. To prioritize safety, the reminder text or message was vague and non-descript (e.g., “Have a nice day”) so as not to alert anyone about the nature of the study.
Ultimately, while these approaches kept participants safe, they did not yield a high follow-up rate. As highlighted in our report, Short-Term Outcomes for Users of the National Domestic Violence Hotline and loveisrespect (PDF), there was a low two-week follow-up rate (15% for callers and 3% for chatters). Also, because our sample was not representative, the results may not be generalizable to the larger domestic violence population nor truly representative of all those who utilize domestic violence services or who contact The Hotline or loveisrespect.
Despite these limitations, this study represents a proof-of-concept: we can conduct longitudinal research, and uphold both our ACF principles of rigor and ethics in anonymous contexts such as this one. In honor of Teen Dating Violence Awareness and Prevention Month, we will continue to think of innovative and creative ways to balance these principles. This will strengthen our efforts to advance rigorous research on intimate partner and teen dating violence and promote safety in the populations ACF serves.