|
[an error occurred while processing this directive]
Return to
OCSE Home Page Return to Table of Contents MID-IOWA COMMUNITY ACTION (MICA)
EXCERPT FROM SUMMARY OF FINAL EVALUATION FINDINGS FROM FY 1990 DEMONSTRATION PARTNERSHIP PROGRAM PROJECTS Agency: Mid-Iowa Community Action 1500 East Linn Street Marshalltown, Iowa 50158 Agency Director: Gary Stokes Evaluator: John Elsa, Ph.D. Contact Person: Arlene McAtee Telephone: (515) 752-7162 Project Type: Family Case Management Project Title: Family Self-Sufficiency Project Model: Matched Study Area Comparison Group Design Project Period: June 1989 to May 1991 I.Background
A.Purpose of the Program The MICA/DHS Demonstration Partnership Program was a joint activity of Mid-Iowa Community Action (MICA) and the Iowa Department of Human Services (DHS). The program was funded by the Office of Community Services, Family Support Administration, U.S. Department of Health and Human Services. An emerging concept in the field of social service is "family empowerment." The basic idea is to enable families to move toward self-sufficiency with interventions that emphasize families' strengths and encourage goal setting by families that will move them away from dependency, thus breaking the cycle of poverty. The purpose of this project was to demonstrate the effectiveness of a family empowerment strategy in increasing the self-sufficiency of Aid to Families with Dependent Children (AFDC) recipients. The strategy consisted of three developmental interventions: 149 nurturing interviews by DHS; 149 family development intervention by MICA; and 149 monthly joint staffings of DHS and MICA. These components were coordinated and implemented by DHS and MICA in four target counties in Iowa (Hardin, Marshall, Poweshiek, and Tama). The hypothesis tested in the project was that new AFDC recipients who participated in the Demonstration Partnership Program would require fewer months on AFDC and would consume a small proportion of the maximum possible AFDC benefits (based on family size) during assistance.
The first component of the MICA/DHS Partnership Program was the "nurturing interview" conducted by DHS Income Maintenance (IM) workers. The idea was to help the IM workers change their focus from AFDC eligibility to guiding the family's focus toward goal-setting and a life not dependent on AFDC. These nurturing interviews were designed to create an atmosphere of rapport and trust between the DHS worker and the family, thus allowing the worker to identify and validate the applicant's competencies and challenge the applicant's self-doubts and low self-esteem. The worker also explained to the applicant how participation in the DHS-MICA Partnership could facilitate movement towards self-sufficiency. The second component was MICA's family development intervention. This was based on a specific model of intensive case management characterized by an intense relationship between each family and a family development specialist. The specialists were trained in the following skills: 149 listening actively; 149 assessing family strengths and needs; 149 focusing family strengths; 149 teaching goal-setting skills; and 149 encouraging the family to achieve the developmental goals they set for themselves. Once the developmental goals were established, the specialists supported the families in the implementation of their plans and assisted in evaluating progress toward goal achievement and replanning if needed. The specialists were also trained to teach family members how to find and use resources and establish support networks with individuals, organizations, and institutions in the community. These intervention sessions occurred in the families' homes. Client participation was on a voluntary basis. The third component of this program was the joint DHS and MICA staffing. The monthly joint staffings were attended by the appropriate DHS and MICA staff members. Each agency shared information they had on each family's status, discussed their insights and provided feedback to the other agency, and suggested possible referral resources. The focus of the evaluation was on the group of families that participated in the Demonstration Partnership Program as provided by MICA, and two other groups of families selected to serve as comparison groups. Because the goal of the Demonstration Partnership program was to enable families to move toward self-sufficiency and thus reduce their dependency on the AFDC program, indicators of program success were identified in the program design as (1) repeated use of AFDC aid and (2) amount of AFDC aid received during the period of time a family was a participant in the Demonstration Partnership Program. This evaluation, however, covers only the 15-month period from April 1, 1990, when the treatment group was complete, to June 30, 1991. Needless to say, this is a brief period in which to be able to observe a program effect.
B.Description of the Program
Process Evaluation This section describes and evaluates the program's process as of August 1990. This evaluation was accomplished by treating each of the three intervention components as a separate subsection: 149 nurturing interviews; 149 MICA's family development intervention; and 149 monthly joint DHS-MICA staffing. Nurturing Interviews The first component of the Partnership intervention with which families came in contact was the DHS nurturing interview. Training of the DHS IM workers and supervisors in this interview technique was included in the original program design. The National Resource Center for Family Based Services (NRC) at the University of Iowa School of Social Work was selected to provide training in empowerment "nurturing interview" techniques. The specific format and techniques used in the Partnership empowerment interview training were jointly developed, planned, and coordinated by NRC, DHS, and MICA. NRC is nationally recognized for their family development specialist certification program. The NRC trained a total of 19 DHS staff, 3 county directors, and 16 IM workers. The NRC-trained staff members are employed in the DHS offices in the four study counties. The training of the DHS staff was accomplished early in the Partnership program. The first 2-day training session occurred on August 31, and September 1, 1989. Then NRC trainers conducted two 1-day sessions, which consisted of observing IM workers and training IM supervisors in how to supervise the IM workers in the nurturing interviews. Following the initial NRC training session, a DHS benefit payment administrator, who had observed the NRC's sessions, trained an additional nine IM workers and three county supervisors from the four counties included in the demonstration. Evaluation of Nurturing Interview Training In March 1990, the NRC evaluated the nurturing interview training and its implementation by sending a 31-item questionnaire to all DHS IM workers and county supervisors who had received the training. The questionnaire was jointly drafted by NRC, DHS, and the evaluator. In May, NRC provided a written report of the results. Twenty-one IM workers and six supervisors responded to the questionnaires (87 percent of 31 possible respondents). The questionnaire asked the respondents to assess the training, the effect of the nurturing interview, the frequency of use of the skills, the amount of supervision received, and frequency and usefulness of the DHS MICA joint staffings. (The discussion of the responses to DHS/MICA joint staffing is in the section on Joint Staffing.) A four-point Likert scale was used to measure satisfaction. For the analysis, these categories were collapsed into two: agreed and disagreed.
The DHS staff, regardless of work position, rated the usefulness of training highly. Ninety percent of the IM workers and 83 percent of the supervisors agreed the training had taught empowerment skills. Also, both levels of staff, 91 percent of IM workers and 67 percent of supervisors, believed the training improved their interview skills. And finally, 71 percent of IM workers and 67 percent of supervisors reported the training added new skills to their repertoire. When questions concerned the effect of interviews with families, both IM workers and supervisors reported very favorable results. Eighty-four percent of the IM workers and all the supervisors reported the empowerment interview increased their rapport and trust with the families. Additionally, 63 percent of IM workers and 60 percent of supervisors reported they found using the interview method increased the accuracy of information families provided. When staff was asked if they thought families felt validated by the use of the interview, 89 percent of the IM workers and 83 percent of the supervisors agreed. Eighty percent of the IM workers and 83 percent of the supervisors agreed the interview techniques helped them to be more focused on the families' strengths and resources. The two questions dealing with the usefulness of the interview in reorienting families toward hopefulness and toward future goals received positive agreement responses from 80 percent of supervisors, while IM workers reported 72 percent agreement. With regard to the use of the interview to enable families to identify incremental tasks, 79 percent of the IM workers agreed it helped, while only 40 percent of the supervisors agreed. Questions involving the use of interviews received lower ratings. Fifty-five percent of the IM workers and 40 percent of the supervisors said that IM workers did not have enough time to use the empowerment skills. When questioned about their use of the interview, 62 percent of the IM workers reported using it most of the time, 24 percent some of the time, 10 percent occasionally, and 5 percent rarely. The responses to the issue on length of time staff used empowerment techniques before establishing eligibility were grouped into 5-minute blocks: 39 percent reported 15 or more minutes, 22 percent reported 7-10 minutes, 17 percent reported 5 minutes, and 22 percent reported 1-3 minutes. The question on whether caseloads were too heavy to allow the time needed to engage the interview skills received agreement from 67 percent of the IM workers and 60 percent of the supervisors. The IM workers and supervisors differed greatly in their responses to a question about the frequency of supervision IM workers received from supervisors in developing their empowerment interviewing skills. Frequency of Supervision IM workers Supervisors Twice a month 0%20% Once a month 11%60% Less than once a month 83%20% Weekly 6% 0%
When asked if supervisors encouraged IM workers to use their interview skills, 100 percent of the supervisors, but only 47 percent of the IM workers, agreed. In response to the question of whether the supervisors helped workers to continue to develop their empowerment interview skills, 73 percent of the supervisors, but only 32 percent of the IM workers, agreed. General conclusions concerning the training sessions based on the questionnaire and evaluator's interviews were that both IM workers and supervisors found the training topics and training itself useful in increasing their interview skill level. The DHS staff reported they believed the overall goal of learning interview skills to facilitate clients' movement towards self-sufficiency was accomplished. DHS staff agreed the skills help to establish trust and rapport, validate the families, help identify both strengths and resources in the families, redirect self-doubt to hopefulness, reorient families' future goals, and to a lesser extent, identify incremental tasks for the family. IM workers and supervisors report that the heavy caseloads of IM workers make it difficult or impossible for them to spend the amount of time required to use the nurturing interview approach. This is a basic constraint to the implementation of the program design that was not adequately addressed in the planning process. Family Development Intervention MICA had been using the family development intervention method since 1984. The Partnership program is MICA's opportunity to demonstrate the effectiveness of this intervention on self-sufficiency in combination with two other interventions. MICA was successful in using the family development intervention with 125 families prior to this Partnership program. An essential aspect of this intervention is the family development specialist's relationship to the family and the specialist's ability to help the family acquire skills so that they can function more effectively among social support networks, independent from a relationship with MICA. Thus, an evaluation of the family development intervention should include an examination of the structure of the family development division, the specialists, and their skills set. The program was directed by MICA's family services manager, whose office was at MICA's central office in Marshalltown, Iowa (centrally located within the four counties). The family service manager holds a B.A. in communication studies and 13 years of experience in family development at MICA. She supervised the four county centers, each of which had a county coordinator. The county coordinators supervised the family development specialists. In all counties except Marshall, the coordinator had a caseload in addition to his/her supervision duties. (Marshall county's larger staff required a full-time coordinator.) There had been one change in county coordinator positions since the Partnership program began. Hardin's county coordinator resigned in May 1990, and in July, one of the family development specialists was promoted to fill the position. Two other county coordinators were also promoted from positions as MICA family development specialists; the fourth was hired from outside MICA. The three long-term county coordinators had served in those positions for 4, 8, and 13 years, respectively. Three of the four had their family development certification from NRC's specialized training program and the fourth had completed the training and was awaiting the results of her certification exam. One county coordinator had a Master's in Early Childhood Development, one had 2 years of college in general business, another was currently working toward an Associate of Arts degree, and the fourth coordinator had a high school diploma.
There were a total of eight family development specialists (5.8 full-time equivalents) involved with the Partnership program, distributed among the counties on the basis of the number of Partnership program families in each county. Of the eight specialists, two held those positions for 2-3 years, and six have held those positions for 6 to 9 months (three were promotions from other MICA positions and three were new employees with MICA). The family development specialist position was viewed by MICA as a paraprofessional position. The formal education of the specialists ranged from high school diplomas to master's degrees. The most recently hired specialists tended to have higher formal education. Once hired, all staff in this position were encouraged to become certified in family development through the NRC training (described below). There was a personal incentive for completing the certification, since it resulted in a 10 percent salary increase. The family development specialist certification program is a training and certification program for community action and other paraprofessional staff, whose job it is to support and provide services to low-income families. Specialists acquired the attitudes, skills, and knowledge they need to work developmentally with families to help them attain greater self-sufficiency. This is an 8-day certification program offered in three segments. Trainees must pass a written test on the course material for certification. Currently, three of the eight specialists are certified and the remaining five specialists have completed the training course and are awaiting the results of their written examinations. The salary range of the three current county coordinators was $26,400 to $28,020 per year. Family development specialists' salaries ranged from $16,320 to $19,920 per year. These salaries are very competitive with other human service positions in Iowa. Some of the family development specialists who worked with the families in the Partnership program also worked with families in other MICA programs. The standard caseload per family development specialist was 25 families for a full-time staff person. While most of the specialists began with a full caseload of Partnership families, some of these specialists had been assigned non-Partnership families because some Partnership families had dropped from the program. This shift of assignments seemed questionable when there were families in the Partnership program population who had not been served. Since the initiation of the program, much of the specialists' time had been devoted to contacting, recruiting, and developing rapport and trust with the families. Specialists expressed surprise at the number of families contacted who were not willing to participate in the Partnership program. This was in part due to some misunderstanding of the referral process. It was not clear to all the specialists that some of the families referred had not expressed a desire for the program, but had simply not refused referral. Some families were surprised to be contacted by the specialists, either because they were not adequately informed about the Partnership program by IM workers or because they did not hear the presentation of the Partnership program in the midst of the stress of the AFDC application process. The specialists indicated that an unusual amount of time and energy was spent on recruitment and developing the relationship before self-sufficiency skill development could be undertaken. The family development specialists operated out of a local center in each of the four target counties. Specialists interacted with the families primarily in their home setting. This allowed the specialists to observe how the families function within the home. The amount and frequency of contact varied with the needs and willingness of the family. The specialists completed the basic intake form and collected other data at the first visit. Then, during the early contact period with the families, the specialists attempted to collect all the informational data on the family and complete the self-sufficiency competency instruments. This information was recorded on forms MICA refers to as milestones, to be described below. The specialists attempted to complete one instrument per home visit. This was not always achieved, since families vary in their willingness to complete the instruments and in the amount of time they have available. To organize this information, MICA developed an MIS system with the assistance of the evaluators. This system contained MICA's normal intake forms, assessment information, and the five milestone forms. The milestone forms were newly developed standardized forms created to record the information required for the Partnership program. The forms contained the following information: 149 demographic data; 149 income and sources of income; 149 results of the self-sufficiency competency instruments; 149 training and educational activities, employment status; and 149 public assistance status. The information on these forms was gathered by the specialists and entered into the computer by the central office staff. The specialists found the use of these instruments and assessment tools valuable in the development of rapport and trust with the families. The specialists also found the instruments served as a useful tool to provide focus and purpose when they met with the families. It guided the specialists' interaction as well as gave them ways to bring families' focus back to task. Specialists felt families in this program were showing faster movement toward self-sufficiency than those in other programs due to these tools. Several specialists reported that the eco-map, genogram, and timeline have not only helped them to gain useful information, but, in some cases, helped participants gain new insight concerning themselves. Two specialists reported the timeline allowed participants a chance to share events that happened in their lives that were so painful they had never been able to verbalize them, i.e., childhood rape, abortion. The forms and instruments received such a positive response from the specialists that some began to use them with families in their general caseloads. Some of the specialists reported that they did not receive adequate instruction on the coding of eco-map (already subjective in nature), so consistency in scoring may be problematic. Since specialists varied in their styles of presentation and administration of the self-sufficiency competency instruments, reliability of the scores may be questionable. There was a great deal of enthusiasm among the specialists about the family development intervention, their jobs, and the support they get from all levels of management. They were very optimistic about the success of the program and are pleased with the achievements some families have already made. The staff in each center appeared to work closely as a team and were supportive of one another, which positively affected their attitudes and performance.
Joint DHS-MICA Staffings The joint DHS-MICA staffings were designed to allow for monthly review of each family's participation in the Partnership program. The monthly meeting allowed for feedback and brainstorming between the two agencies. The hope was these sessions would lead to insight and information regarding appropriate referral sources for the families and impress the IM workers with the importance of their work and the effectiveness of the family development approach. In order to facilitate rapport and positive relationship building between DHS and MICA staff at the county level, visits to each other's offices occurred before the first staffing. These meetings provided an opportunity for staff to observe and gain a clearer understanding of the other agency's work. Marshall, Hardin, and Tama counties' first staffing took place in December of 1989, while Poweshiek began in September of 1989. The staffings were held monthly in all counties, with the exception of March in Tama county. Most joint staffings in Marshall and Hardin counties occurred at DHS's offices, while the staffings in Poweshiek and Tama usually alternated between DHS and MICA offices. In addition to these meetings, during the program start up, there were frequent, less formal contacts beyond the staffing concerning referrals and contacts with the families. The format of these staffings varied. In Marshall county, the IM worker and the specialist assigned to a family met one-on-one and updated each other on changes or events concerning the family. The length of time spent discussing each family varied based on the issues involved. Most staffings observed by the evaluator lasted an average of 5 minutes. Usually, the specialists reported what had occurred during their contacts with the families. The IM workers usually did not have much input to make, since they had limited contact with the families. In the other three counties, Hardin, Poweshiek, and Tama, the staffings were group discussions involving all the staff members from both DHS and MICA. The discussions only covered those families whom staff members believed warranted discussion at the monthly meeting based on changes in, or events related to, the family. These meetings were usually brief -- 10 to 15 minutes in duration. Neither IM workers nor specialists found the monthly staffings to be time well spent. Members of both staffs would like the freedom to consult by phone and on an "as needed" basis. Both DHS and MICA staff stated that it was not due to any negative feelings towards the other's organization. IM workers' primary reason for desiring the reduction of the joint staffing was based on their belief they have no need for this in-depth knowledge of the families. They also saw no relevance of any information beyond that required to determine AFDC eligibility. A review of the NRC's questionnaire administered to the DHS IM workers corroborated the statements of IM workers about the staffings. Asked if the staffings were a good use of staff time, those who met 2-3 times a month with specialists agreed, while those who met only once a month disagreed. In response to the question asking whether the monthly staffing increased IM workers' hopefulness about potential self-sufficiency among AFDC recipients, 78 percent of the IM workers indicated that it did not. The specialists stated that during the staffings, they learned about the eligibility aspect of AFDC and that IM workers were helpful in providing families' addresses for the initial contact. This type of information was beneficial when the program was beginning. The staff support for the staffings appeared to decrease over time. It appeared that both DHS and MICA staff members needed some clarification of what the purpose of these meetings was and what an effective agenda should be to make these staffings effective. It appears the staffing did not achieve the goal of brainstorming for insight into appropriate actions and referrals as it was intended. C.Target Population The original plan for selecting participants for this demonstration was changed. The details concerning the necessary design changes and the reasons underlying these changes are described in a later section of this report. The revised sampling procedures produced three groups of families: one treatment group and two comparison groups. Study Group (N = 98). The treatment group was a convenience sample of all eligible families seeking AFDC benefits who: 149 applied for AFDC benefits between September 1, 1989, and March 31, 1990; 149 lived within MICA's geographic region (Hardin, Marshall, Poweshiek, and Tama counties); 149 agreed to participate in the Partnership Program; and 149met the program intake criteria by having an adult member who was 18 years of age or older, by having an employable adult member, and by having been off AFDC for at least 2 months prior to this AFDC approval. Comparison Group 1 (N = 120). Group one was a comparison group of families that received the nurturing interviews from DHS IM workers, but did not receive the family development intervention or the joint DHS-MICA staffing. The counties in which these families live -- Benton, Iowa, Jones, and Washington -- are in the same DHS district as were the treatment group families. Comparison group 1 consisted of 120 randomly chosen families from all families approved for AFDC benefits between September 1, 1989, and March, 1, 1990, who lived in these counties. Comparison Group 2 (N = 120). Comparison group 2 was drawn from four counties that most closely matched the study counties on population, demographic factors, and characteristics of the AFDC population. This comparison group did not receive DHS's nurturing interviews, MICA's family development intervention, nor the DHS-MICA joint staffing. Thus, in these four counties --Mills, Webster, Cedar, and Crawford -- there were no changes made in the AFDC delivery system. Once the matched counties were selected, all families approved for AFDC benefits between September 1, 1989, and March 1, 1990, who lived in these counties formed the population from which 120 families were randomly selected. In summary, the three groups of families analyzed in this study were drawn from 12 Iowa counties. The total study population was 338 families. TABLE 1 Demographic Characteristics for Study Group and Two Comparison Groups in Demonstration Partnership Program
* p <.05. ** p <.01. NSS = Not Statistically Significant. D.Research Significance The evaluation of this Demonstration Partnership Program was based on an analysis of two sets of variables. The first contained two outcome variables, which are described below. These two outcome variables were used in the analysis of the treatment group (the MICA families) and both of the comparison groups. The second set consisted of a number of intermediate variables that were used in the analysis of a subgroup of the treatment group. The intermediate variables were intended to act as an underlying explanation as to why the program had the desired effect. To be a member of the treatment subgroup, a family had to have been a participant in the family development intervention for at least 1 year prior to the end of the demonstration program.
Outcome Variables. The study and comparison groups were compared using two outcome variables: a.Number of months a family received an AFDC grant during the project period (April 1, 1990, to June 30, 1991), and b.Amount of the family's AFDC grant during the last 12 months of the project. Intermediate Variables. In addition to the analysis of study and comparison groups using the two outcome variables, this evaluation analyzes a subgroup within the treatment group population composed of those who received family development for at least 1 full year. This analysis tests the families' growing empowerment and their movement toward self-sufficiency. Measures of these intermediate variables were taken because enhancing specific self-sufficiency skills is one of the assumptions of family empowerment. Five self-sufficiency competencies believed to be essential were measured in this study. They are outlined below, together with a description of the instruments that were used to measure each of these variables. These instruments were administered at the time of admission and after 1 year in the program. 1.Self-esteem - family members' level of self-respect. Self-esteem was measured by the Index of Self-esteem. (Walter Hudson, The Clinical Measurement Package: A Field Manual, 1982). 2.Problem-solving - a family's ability to identify and define problems that it would like to solve, to set specific achievable goals, and to plan actions needed to achieve these goals. Problem solving was measured using both the Nowicki-Strickland locus of control instrument and a planning form that MICA regularly uses to teach families problem-solving skills. 3.Locus of control - the tendency of family members to feel that they govern their own lives, as opposed to being controlled by external forces. This facet of empowerment was measured by the Index of Internal Control (Walter Hudson, The Clinical Measurement Package: A Field Manual, 1982). 4.Social relations - the ability of family members to establish and maintain a positive, supportive network of individuals, organizations, and institutions in the community. Networking was measured with two instruments, the index of Social Relations (Walter Hudson, The Clinical Measurement Package: A Field Manual, 1982) and the drawing of Eco-maps. 5.Family relations - the ability of family members to establish and maintain positive, supportive relationships with other family members. Family relationships were measured using the Index of Family Relations. (Walter Hudson, The Clinical Measurement Package: A Field Manual, 1982). 6.Parental attitudes - the ability of the parents to assume the parental role with confidence and satisfaction. Parental attitude was measured with the Index of Parental Attitudes. (Walter Hudson, The Clinical Measurement Package: A Field Manual, 1982).
The family development specialist administered the instruments to the head of household during their second visit with the family. The specialists found that the rate at which families completed the instruments and the amount of information they were able to gather varied, depending on the frequency of visits, number of family crises, and the level of family cooperation. Thus, complete data on the intermediate variables (self-sufficiency competency scales) was obtained from only 20 of the 98 families in the treatment group. E.Participant Characteristics To study the effects of demonstration programs, the practice in program evaluation is to compare change in a randomly selected sample of clients who participated in the program with change in a randomly selected sample who did not receive the program. In order to be able to attribute a change to the program rather than to some differences with the groups, it is necessary that the groups be as similar as possible. Personal and environmental characteristics that might be influencing the change are thus held constant across the groups. The most important characteristics of the three groups of families in this study are presented in Table 1. In addition to providing an indicator of the validity of the study, these demographic data furnish us with a fairly comprehensive profile of the families in these groups. Unfortunately, unlike the information on the outcome indicators, demographic data were not available from DHS for all of the randomly selected families. Table 1 shows that there were no statistical differences between the three groups in terms of age of the head of the family and the grant amount that they had received in January of 1990 for food stamps and for shelter. There were, however, significant differences in regard to gender and marital status of the head of household, as well as family size and number of family members who were over the age of 18. In general, we can say that these families were predominantly headed by single young women who had one or two children, and who were either unemployed or underemployed to such an extent that they met the criteria for AFDC eligibility. The largest difference in these data is in gender of the head of household, and this is such a large difference that it is problematic. Undoubtedly, there is an environmental explanation for the difference, such as a large lay-off in the four-county region of Comparison Group 2, and this factor could influence an interpretation of the study findings. The Treatment SubGroup (N = 20). A sub-group of families in the MICA Partnership Program agreed to be assessed on intermediate outcome measures. At their entrance into the program and at 1 year, the head of household took six written tests that measured a number of intermediate outcomes that are of interest. The treatment subgroup was not very different from the total group of MICA program participants, although there were some data items available for this subgroup that were not available from DHS for the full groups. In terms of gender, the subgroup was also predominantly female, but they were older on average than were the other groups.
This brief profile of this group of 20 families shows the majority having at least a high school education. They have, on average, one child per family who appears to do reasonably well in school. During the year under study, the children's schools reported that only three children received disciplinary action, and, of these, only one child received more than one disciplinary action. II.Discussion of Findings A.Impact of the Program on Clients There were 98 families that agreed to participate in the Demonstration Partnership Program during the period from September 1989 to March 1990. The assessment of program outcomes required, therefore, an examination of these families' records of dependence on the AFDC program between April 1990 and June 1991. If the hypothesis of the demonstration was to be supported, then the proportion of families receiving AFDC grants in the Study Groups would decrease at a faster pace than would the proportion of families in the other two groups. Table 3 summarizes this information. TABLE 3 Comparison of Study Group and Two Comparison Groups Dependency on AFDC Across 15 Months
NSS = Not Statistically Significant.
The data from DHS on AFDC grant amount by month for these 338 families were analyzed to determine whether there is a statistically significant difference in the rate of change in grant amount. This analysis attempts to answer the question regarding whether or not the Demonstration Partnership Program succeeded in reducing the Treatment Group's reliance on AFDC below the levels of dependency of the Comparison Group families. The answer, in general, is that it did not. Across the 15 months, there were no statistical differences in the mean amount of the grants that were received by the families in the three groups. In other words, although the proportion of families receiving grants and the average amount of the grants for the three groups decreased across time, they did so equally. It can be seen in Table 3 that over the 15 months, all three groups of families experienced a decline in the proportion of their numbers that were dependent on AFDC and, indeed, the average amount of the grant also declined. However, none of the differences in these reductions are significantly different. It can also be seen, however, that the average grant amount for the study group was considerably less than for Comparison Group 2, the group that received no intervention. It may be that if a study were to look at the next 30 months, this trend would continue and would be significant. Further study of these groups will have to take into account the gender and family size differences in the groups in order to better understand program effects. Increasing Self-Sufficiency Competencies Measures of Intermediate Outcomes. As described above, a subgroup of the treatment group agreed to participate in an assessment of whether the family development intervention was able to create change in families along a number of dimensions thought necessary for self-sufficiency. The competencies are in the areas of self-esteem, problem-solving, locus of control, social relationships, family relationships, and parental attitudes. Table 4 reports the results of the tests used to measure the change in the subgroup families' abilities in these areas. Table 4 provides a summary of the scores that represent the extent to which program participants improved in basic self-sufficiency competencies, as measured by the instruments administered by the staff. The scores used for this analysis were obtained from the head of household. Twenty parents took these written tests at intake, and the average of the 20 scores is reported in column 1. Likewise, the average scores for this group of parents after 1 year of participation in the program are listed in column 2. The t-test, a measure of statistical difference between the average scores of groups, is shown in column 3, with its index of statistical significance in column 4. Overall, and based on these measures, participating families improved in all but one of these self-sufficiency competency areas. All of the post-test averages were lower (or higher as the case may be), and most of the 1-year scores were significantly different from the initial scores, and all were statistically significant at p < .10 or lower. Only in the area of parental attitudes do the scores indicate a worsening of the problem. Not only are the raw scores extremely low, but the 1-year average scores are higher than the initial ones. In these families, parents' attitudes toward their children grew increasingly negative as the year went by. This is not an unexpected outcome of welfare-to-work programs, however, and has been found in other program evaluation studies. TABLE 4 Differences in Average Scores between Pre- and Post-Test Measures of Self-Sufficiency Competencies (N=20)
* p < .05. ** p < .01. The three areas in which these families improved the most were in the areas of self-esteem, problem solving, and locus of control. These scores show statistically significant improvement (self-esteem t=5.07, p < .000; problem solving t = 2.63, p < .004; and locus of control t = 2.12; p < .048). 5 The ability of these parents to establish firm and useful social relationships with peers and other family members also improved, but the scores were not statistically significant. B.Institutional Impacts Several other insights, gained from the evaluator's observations and interviews with IM workers and DHS supervisors, are worthy of reporting. First, the IM workers found the interview training valuable. An overwhelming number of IM workers only have high school diplomas. Many found the nurturing interview valuable professional training. The IM workers' major job function is that of a "technician," determining if the families' income level qualifies them for AFDC grants. This does not mean to imply that IM workers are uncaring and nonprofessional in their jobs. In fact, the IM workers interviewed were glad for the training opportunity, since it aided in their contacts with families. The training allowed them to develop and refine skills beyond those of pure eligibility determination. The IM workers' training appears to have been beneficial not only to this project, but also in enhancing the IM workers' skill set. The second finding was that there was considerable confusion regarding the eligibility criteria and referral process. This resulted in ineligible families receiving referrals to the Partnership program. It appears this problem began from the initial presentation of the Partnership program to DHS staff. The Partnership program was presented to MICA management staff and DHS directors (supervisors) in a July 17, 1989, meeting, followed by an August 2, 1989, presentation meeting to IM workers. The purposes of these meetings were to provide information, foster support, and address issues or concerns. In addition to these meetings, DHS and MICA staffs organized local meetings to develop support for, and understanding of, the Partnership program. It appears that these meetings were not effective in achieving their purpose. IM workers expressed frustration that some issues and concerns at the August meeting were not resolved before the program began. They raised questions about confidentiality, what IM workers viewed as conflicting job roles (nurturing interviews versus corrective actions), concerns of increased work demands, and basic "how to do" questions. IM workers left with many questions and concerns, and a feeling that the program required further refinement. This played a part in IM workers' understanding and enthusiasm for the program. DHS supervisors were more accepting of this ambiguity as a reality in program development, stating that clarity about all aspects of any program could not be expected until programs are underway. The evaluator asked the IM workers about their understanding of the eligibility requirements for the Partnership program. The IM workers responded that the only eligibility requirement was for families to have been approved for AFDC; none mentioned the additional eligibility criteria. The third evaluation finding was that the amount of support within DHS for the Partnership program varied by organizational level. It appears that upper management was very supportive and enthusiastic about the program. The individual county supervisors seemed torn between their support for the program and empathy for their staff. Though they were enthusiastic about the program, they do not seem to have provided the intended intensity of supervision of the IM workers in their use of the nurturing skills. One supervisor indicated that she reduced the amount of supervision to lessen the stress on the IM workers. Proper and continuous supervision of the interviewers was an essential element of the program design. The NRC trainers stated that to change IM workers' focus from eligibility to empowerment interviewing would require constant reinforcement. Based on questionnaire findings and the evaluator's interviews, this reinforcement does not seem to have occurred in most cases, thus reducing the effectiveness of this intervention.
The fourth finding was the resistance of the IM workers to the program for a variety of reasons. They perceived the nurturing interview as inconsistent with their primary responsibility of AFDC eligibility determination. Furthermore, they saw the nurturing interview as requiring more work when they were already overloaded. Three changes that occurred in the policies regarding the referral process during the course of the program (due to the events surrounding an Assistant Attorney General's opinion) provided another reason for resistance, since it appeared the program was not well thought out. On the other hand, IM workers stated they did want to help families and were not opposed to referring families to sources for help and benefits, as they have always done. Nor was there resistance to referring families to MICA. Participation of IM workers during the planning process probably would have reduced this resistance. Early involvement would have given the IM workers a sense of ownership rather than the experience of having additional work added to their already heavy loads. IM workers' perceptions, reinforced by the videotaped interview used in the NRC training, were that planners had little understanding of the IM workers' job duties. Also, a follow up meeting with the IM workers to address the issues they raised at the August meeting would have been beneficial in lowering anxiety and increasing program understanding. C.Expected Versus Actual Outcomes To test the effectiveness of these combined-interventions methods, 98 newly approved AFDC families in the four target counties were assigned to the treatment group receiving the family development intervention. The self-sufficiency achievements of this group were to be compared with two comparison groups of newly approved AFDC families from other counties. In the original evaluation design, the study population was to be composed of the first 120 newly approved AFDC cases in the four target counties. The sequential selection of cases would terminate when the population distribution of cases across the counties were 15 families each in Hardin, Powesheik, and Tama counties, and 75 families in Marshall county. Due to a series of unanticipated and unfortunate events, the plan for a sequential distribution had to be revised. The original concept was that, since this was a partnership with DHS, the MICA family development services would be considered a DHS program and thus a signed release form would not be necessary prior to MICA's contact with the family. The IM worker would briefly present the Partnership program to the family, using the nurturing interview approach. Then, the MICA worker would contact all new AFDC approved families, explain the family development program in greater detail, and ask the family whether they would like to participate. If the family said yes, MICA would ask them to sign a formal release of information. However, before the program started, the assistant Attorney General for DHS issued an option that DHS could not authorize referrals of recipients' names to MICA without a client's written release. This opinion changed the design for the creation of the study population, since it allowed families to opt out of the program without a thorough explanation of the program by MICA staff. Furthermore, the Assistant Attorney General's opinion was not made until early November, so new AFDC approved families from September 1 to November 6 were not given an opportunity to sign written release forms authorizing referral to MICA during their AFDC intake interviews. As soon as the Assistant Attorney General issued his opinion, release forms were sent to all families approved between September 1 and November 6. After November 6, all AFDC applicants who met eligibility requirements for the Partnership program were offered the option to participate if they were approved.
The change in the program design and the delay in the Assistant Attorney General's opinion resulted in a very low number of referrals during the first 3.5 months of the program. Of 116 new approvals between September 1 and December 19, only 34 (29 percent) agreed to participate. Since applicants for AFDC are under tremendous stress at the time of application, and since the IM workers varied in their understanding of the MICA family development program and in the quality of their presentation of the benefits of the program, the change in evaluation design significantly reduced the expected rate of "up-take" of the program by potential participants. The evaluator discovered this significant change in referral process on December 11, 1989, and noted that it represented a major change in the mutually agreed-upon evaluation design. This resulted in a conference on December 21 involving the evaluator, MICA staff, DHS regional and central office staff, and the Assistant Attorney General for DHS. Once the assistant Attorney General was clear on the nature of the program and partnership, an agreement was reached to return essentially to the original research design effective December 31, 1989. Thus, in January 1990, DHS began telling newly approved AFDC families that they would be referred to MICA for a visit by a family development specialist unless they specifically refused. In order to correct some of the error that had occurred during the September-December period and to recover for participation in the program some of the families that had not been referred, letters were sent to all AFDC approvals from September through December stating that unless the family objected, their name would be referred to MICA. Some families did not respond, and so were referred to MICA, but other families refused participation. This series of events created a dilemma regarding how to construct the study population. It was no longer accurate to consider the first 120 new AFDC approvals as the study population, as in the original design. Many of the new AFDC approvals, especially in the first 2 months, did not receive a thorough presentation of the family development program, and a high proportion were not even referred because of the constraints created by the Assistant Attorney General's opinion. A revision in the design of the study and comparison populations was essential. The compromise the evaluator adopted was to keep all new AFDC approvals in the study population, and to assure that there was a reasonable subpopulation size actually receiving family development services. Thus, in early April of 1990, the evaluator obtained a complete printout from the DHS Bureau of Research and Statistics of all new AFDC approvals from September 1, 1989, through March 31, 1990, who met the program eligibility criteria. (The guidelines for family program eligibility were that the adult member must be 18 years of age, or older, have been off of AFDC for at least 2 months prior to the current approval for benefits, and be an employable adult). The method used to create the list of names was to compare the names of those persons with "effective dates of approval" for AFDC in September of 1989 with the name of persons who received AFDC the 2 previous months. If the name did not appear, it meant the person was a newly approved recipient. This comparison was repeated on data through March 1990. This computation was to ensure that families were not receiving assistance in the 2 previous months, thus meeting the program requirements. The computer program also eliminated persons who did not meet the other program criteria, i.e., the head of household must be 18 years of age or older.
D.Summary This evaluation of outcomes has been a limited look at a program that was funded for a very brief period of time. The results outlined above should, therefore, not be taken out of context and generalized beyond the confines of this demonstration. Even though statistical differences could not be detected in the proportion of the treatment group families who received AFDC grants during the 15-month evaluation period when compared with the other groups, there were interesting findings that, at a minimum, beg for further support of the family development approach. The program clearly was able to improve parents' self-esteem, problem-solving skills, and the quality of their social relationships in their social networks. These are important findings. |
