Supporting Survey Response Through Tokens of Appreciation

The ICHQ project team conducted two experiments of tokens of appreciation with surveys of staff in CCEE centers to test the impact of the use and amount of prepaid tokens of appreciation to improve survey response and reduce the number of days it takes for a survey respondent to complete a survey after they receive an invitation to complete it (days to complete).

First experiment results (Spring 2021)

• There was a significantly higher response rate to complete a 15-minute time-use survey among respondents (including select center administrators and all teaching staff) who received prepaid and postpaid tokens of appreciation over those who received only a postpaid token of appreciation (81 percent compared with 61 percent). There were no differences between the two groups with respect to days to complete the surveys (about 11 days, on average).
• The gift card cost per complete for the group receiving both prepaid and postpaid tokens of appreciation was higher at $23.60 than the cost per complete for the group receiving only a postpaid token of appreciation at $20.00. The additional cost for this group was a result of needing to purchase physical gift cards for all sample members—not just those that completed surveys—and because there is an additional $1 processing fee for physical gift cards that is not incurred for electronic gift cards.

Second experiment results (Fall 2021)

• There was a significantly higher response rate among the group of teaching staff respondents who received the lower prepaid and higher postpaid tokens of appreciation to complete a 45-minute survey over those who received prepaid and postpaid amounts that were even (93 percent compared with 87 percent). There were not significant differences between the two groups with respect to days to complete the surveys (about 14.6 days, on average).
• The $10 prepaid/$40 postpaid token of appreciation structure offered a lower cost per complete ($51.85) than the $25 prepaid/$25 postpaid structure ($54.80).

In each round of the experiment, we categorized all the centers in our sample into strata based on their quality rating and improvement system (QRIS) rating and funding mix. We then randomly assigned centers in each stratum to one of the two experiment groups. For each experiment, we examined whether the treatment and control groups were balanced on observable center characteristics: QRIS rating, funding mix, age group of children served, and licensed capacity. We followed the approach recommended by the What Works Clearinghouse (2020), and calculated an effect size difference using Cox’s index with a correction for small-sample bias for each binary variable (age group of children served and licensed capacity), and each level of each categorical variable (QRIS rating and funding mix).

We conducted t-tests of the differences in overall response rates and days to complete between the two groups of respondents in the two rounds of experiments with tokens of appreciation; we did not conduct t-tests of differences between subgroups, given small sample sizes.

The results of these experiments show that a relatively small prepaid token of appreciation can significantly improve response rates among CCEE staff when in-person visits by study representatives are not possible. It also suggests that prepaid tokens of appreciation might remove the need for in-person visits to achieve a high response rate. This could generate significant cost savings on future projects when the design does not otherwise require an in-person visit, or the project is taking place in an environment where centers may not welcome on-site visitors, as during the COVID-19 pandemic.

S. M. Albanese, A. Edwards, A. Weiss, K Gonzalez, and G. Kirby. (2023). Supporting survey response through tokens of appreciation. White paper from the Assessing the Implementation and Cost of High-Quality Early Care and Education project. OPRE Research Report 2023-236. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.