Exploratory research in public social service agencies: as assessment of dissemination and utilization.

The purpose of this study was to investigate how nine exploratory research studies were disseminated and utilized by social services agencies in four California counties. It is based on in-depth interviews with sixteen key social service agency staff members in four counties who were involved in the planning and implementation of the research projects. While reports were disseminated internally to agency management staff, the results revealed that fewer were shared with supervisory and line staff. All of the studies influenced agency thinking and, in some cases, specific agency decision-making processes. The key factors influencing the utilization of research included: (a) characteristics of the findings and recommendations, (b) specific project situations, (c) practitioner and researcher characteristics, and (d) communication. Implications for future research and practice are identified in relationship to improving the scope of work, strengthening the agency-researcher partnership, developing a research networking program, and increasing media strategies for bringing research to the marketplace.

**********

While the primary purpose of research dissemination is to get new knowledge into the hands of those who could use it for the good of society (Cronenwett, 1995), studies of research utilization in public social services reveal limited use of findings by practitioners (DeMartini & Whitebeck, 1986; Gingerich, 1982; Millman et al., 1990; Mutschler, 1984). The picture is only slightly more favorable when assessing research dissemination and utilization among agency administrators and policy makers (Beyer & Trice, 1982). While Rothman (1980) reported considerable use of research in planning, achieving objectives and correcting false assumptions among social service administrators, McNeece, Dinito, and Jonson (1983) found that only 25% of mental health directors who used research indicated that it influenced decisions regarding program changes. Weiss (1980) reached similar conclusions in a study on the utilization of social science research by 150 mental health directors.

Not only is the extent of research utilization low within social services agencies, dissemination efforts by academicians are also quite limited (Feldman, 1986; Mizrahi, 1992). For many years, research dissemination and utilization were viewed by researchers as simply reporting findings/recommendations through reports and articles based on the assumption that they would be applied to programs and practice. Over the past few decades, researchers found this approach to be inadequate and have identified an array of research dissemination and utilization strategies (Pelz, 1978; Rich, 1977; Beyer & Trice, 1982).

The dissemination and utilization of research by agency practitioners is a topic of considerable complexity (Anderson, 2001; Reid & Fortune, 1992). As Staller and Kirk (1998) have noted, the complexity of the practice environment along with the characteristics of the knowledge or research being disseminated are structural factors that can impede effective research utilization. To add to this complexity, the challenges faced by researchers in assessing dissemination and utilization are also substantial. As Larsen (1981) noted, the methods for evaluation are complicated by: a) differing definitions of utilization (e.g. is reading a study the same as implementing one of the recommendations?), b) differing research methods impacted by a seemingly endless array of variables which interact with one another to influence the ways in which practitioners utilize information, let alone interpret and utilize specific research findings, c) the lack of a systematic catalogue of contextual factors and situational variables that can impact dissemination and utilization (e.g. knowing how to insert research findings/recommendations into an agency environment of continuous change and demanding caseloads), and d) the impact of temporal factors on research utilization (e.g. "timing is everything" especially when it comes to assessing practitioner readiness discuss findings and utilize recommendations).

With these caveats in mind, Bullock et al (1998) in their pioneering research on social service practice noted that the primary forms of dissemination are through in-service training, on-going supervision, and word of mouth where the process of sharing is more important than the conference presentation or the published report. They found that practitioners were most receptive to research when the team of practitioners was stable (no major changes), staff morale was high, staff shared a common outlook on their work, and agency resources were reasonably good. Given the unusual occurrence when all of these factors are present, it is not surprising that research dissemination and utilization are such complex processes, especially when practitioners often regard research as a threat, an irrelevance, or both. Bullock et al (1998) found that there appears to be "no structural basis for the relationship between research and practice ... (where practitioners tend) to dismiss criticism from without as a failure on the part of the critic to understand the social work task (and yet) in research, such scrutiny is the engine of progress" (p. 85, 86). To address this situation, they suggest several important steps: 1) considerable restructuring of research findings will be needed to make them user friendly, 2) a new breed of translator is needed because neither the researchers nor the practitioners may have the time or skills, and 3) new forms of practitioner-researcher collaboration will need to be found to effectively promote dissemination and utilization. This paper focuses on this third dimension.

Given the current state of the art of research dissemination and utilization in the human services, this study is an assessment of the dissemination and utilization of nine exploratory research projects conducted in four county social services agencies in the San Francisco Bay Area. The topics of all nine projects were proposed and funded by each of the four agencies. The scope of each research project was negotiated between the university research team director and agency representatives. The impetus behind this follow-up study of nine projects is the shared interest in outcomes by the agency representatives and the university research team.

This follow-up study was sponsored by the Bay Area Social Services Consortium (Austin et al., 1999), funded by the Zellerbach Family Fund, and conducted by the Center for Social Services Research at the University of California, Berkeley, School of Social Welfare. The overriding goal of the study was to learn about how research reports were disseminated and utilized in the four social service agencies. The specific objectives of the study were to (a) gain feedback from research consumers about the dissemination of the studies as well as their utilization, (b) use the feedback to modify the research program, and (c) contribute to the body of knowledge related to research dissemination and utilization in the field of public social services. It is important to note that the term social services "practitioner" is defined in this study as middle and senior program managers who were the primary audience for the nine exploratory research studies. A much broader practitioner audience could have been defined but that would have been beyond the scope of this study.

Overview of Literature on Research Dissemination and Utilization

This brief review focuses on literature related to: 1) factors that influence research dissemination, 2) organizational factors that influence research utilization, and 3) theoretical types of research utilization.

Factors that Influence Research Dissemination: Applied research is considered valuable to the extent that it contributes to the development and implementation of policies and programs, as well as the extent to which it addresses strategies to deal with social problems. Social services practitioners face many challenges when they are on the receiving end of research dissemination (Stricker & Keisner, 1985). Often, practitioners are not familiar with research language and methods (Cronenwett, 1995), and therefore, it is important for researchers to present research in a clear manner (e.g., avoiding technical jargon and advanced statistics). Even when practitioners are capable research consumers, the time pressures of service delivery pose a serious challenge to keeping abreast of new, relevant information (Funk, Tornquist, & Champagne, 1989; Pettengill, M, Gillies, D., & Clark, C., 1994). While practitioners may lack a mastery of research methodologies, researchers may lack the practice experience/expertise needed to report information in an accessible manner (Holland, 1998).

Depending on whether the research is "decision-driven" (i.e., it leads to a new practice or intervention) or "conceptual" (i.e., it is intended to influence thinking rather than action), different methods can be used for effective research dissemination. The key aspects of "decision-driven" research include accessibility of the research to practitioners, the practitioners' ability to interpret the literature, and the utilization of conferences and integrative research findings (Cronenwett et al., 1995; Wicox, Hadley, & Bacon, 1998). Conferences and print materials (e.g., workplace newsletters) have been found to be most effective when disseminating "conceptual" research (Cronenwett et al., Ingram, D., 1998). The new information technologies, such as websites, also offer many opportunities to disseminate decision-driven and/or conceptual research (Ingram et al., Holland, 1998.)

Green and Johnson (1996) identify three possible roles that researchers might play in the process of disseminating research. One role is to continue replicating and re-testing past research to assess the generalizability of findings. Another role is to put more time and effort into presenting research findings and implications in a format relevant to their consumers. A third approach is to conduct more meta-analyses where findings and applications from numerous studies are analyzed and presented in a format that builds a common knowledge base relevant to practitioners and researchers. In summary, the literature suggests that the way research is presented to practitioners can be one of the most significant factors that can influence the dissemination of research.

Types of Research Utilization: In contrast to dissemination, research utilization by practitioners, according to Reid and Fortune (1992), has at least five dimensions:

* "Instrumental utilization" occurs when practitioners alter their practice methods after reading research findings, thereby impacting specific decisions or problem-solving processes (Rich, 1977).

* "Conceptual utilization" occurs when research provides agency staff with greater insight about social problems without necessarily directly influencing specific decisions. These insights are applied at a later time when practitioners combine their own experiences and beliefs with research findings and apply them in specific situations (Rich, 1977).

* "Persuasive utilization" occurs when practitioners use research findings to support a position, as reflected in the work of lobbyists, advocates, policy makers, and administrators (Leviton & Hughes, 1981).

* "Methodological utilization" occurs when practitioners adopt specific research tools which can be used as either standardized tests for service evaluation or diagnostic tools for client assessment (Tripodi, Fellin, & Meyer, 1983).

* "Indirect utilization" occurs when practitioners employ theories, practice models, or procedures that are based on research but do not require any contact with research findings (Reid & Fortune, 1992). Reid and Fortune (1992) suggest that practitioners are probably informed by research indirectly more often than they would think (e.g., knowledge gained during graduate school, reading books and articles that draw on research, working with program directors who are influenced by research literature).

Because methodological and indirect utilization were not a focus of the current study, only the concepts of instrumental, conceptual, and persuasive utilization were used.

Organizational Factors that Influence Research Utilization: Rosen (1983) identifies four organizational factors that influence research use: (a) characteristics of the knowledge to be utilized; (b) practice situations; (c) practitioner and researcher characteristics; and (d) communication channels as follows:

* For research to be utilized, the knowledge generated by research must be relevant to critical dilemmas and decisions facing practitioners (Weiss & Bucuvalas, 1980).

* Practice situations calling for immediate action may not be easily reconciled with the gradual and cumulative pace of research.

* The education and predisposition of practitioners (more humanistic and interpersonal interests with less experience or interest in conducting or utilizing research) usually differs significantly from those of researchers who pride themselves on distance and objectivity and have less interest in client involvement. These two perspectives intersect when the reputation and credibility of the researcher can influence the use of research findings, especially if practitioners perceive the researcher as having an inadequate understanding of the service sector. Even with a high level of research credibility, research findings are more likely to be used if they are consistent with the beliefs and expectations of practitioners and minimally conflict with other available information (Cousins & Leithwood, 1986).

* The nature of communication channels between researchers and practitioners will influence the likelihood of research utilization (e.g., oral and non-technical written presentations of results and broadly-flamed recommendations). In addition, ongoing and/or prior communication, as well as geographic proximity between the researcher and the users, are related to increased research utilization (Cousins & Leithwood, 1986). Finally, the extent to which users participate in the planning and implementation of the research can influence utilization and future research (Casey, 1983).

In summary, other studies confirm the relevance of the four organizational factors in a variety of settings such as politics, mental health, and education (Apfel & Worthley, 1979; Cohen, 1977; Huberman, 1994; Nilsson & Sunesson, 1993).

Method

This study assesses the degree to which nine exploratory research projects (noted in Figure 1) were disseminated and utilized by four county social services agencies. These nine studies constitute the first phase of research collaboration between local social service agencies and a university school of social welfare (the second phase is devoted to multi-county studies, not reported here). The nine studies were conducted in four agencies (two urban and two suburban) that were self-selected based on their ability to fund research studies of up to $25,000 per year. It is important to reiterate that the topics of all nine exploratory research studies were selected by agency staff and structured in the form of a research scope of work in collaboration with university researchers.

Figure 1

BASSC Research Projects & Objectives

1. Homeless Needs Assessment. Initiated by the county to better understand the service needs of homeless adult individuals and their families to plan for county-wide homeless services.

2. General Assistance Client Demographics Study. Initiated by the county to better understand the demographic characteristics and service needs of GA applicants and recipients in order to develop program plans and policies.

3. Assessing Quality of Care in Kinship and Family Foster Care. Initiated by the county to examine various dimensions of quality of care in kinship foster care and family foster care in order to develop guidelines for child welfare workers to use when making initial placements for children.

4. Considerations Relating to the Placement of Children in Gay/Lesbian Foster and Adoptive Homes. Initiated by the country at the request of Superior Court judges to assess the existing literature relating to gay and lesbian parenting in order to more thoroughly consider issues pertaining to the placement of children in gay and lesbian foster and adoptive homes.

5. Developing a Public Information and Community Relations Strategy. Initiated by the County to examine how public relations is currently carried out in the public and private sectors in order to assist the Social Service Department in developing a formal public information function.

6. Factors Associated with Family Reunification Outcomes: Understanding Reentry to Care for Infants. Initiated by the County to identify various characteristics (child, family, service, environmental, court, and caseworker) associated with success or failure in family reunification in order to further inform the planning of child welfare services.

7. Foster Care Recruitment, Retention and Rate Setting. Initiated by the county to compare and assess the various approaches used by states to recruit and retain foster parents and to understand how state and county governments determine payment rates for foster parents.

8. Service Use and Unmet Needs Among Long-Term AFDC Recipients. Initiated by the County to better understand the service needs of long-term AFDC clients in order to develop programs to help promote their financial self-sufficiency.

9. A Review of Managed Care as a Tool for Child Welfare Reform. Initiated by the County to investigate approaches for reforming the delivery, management and financing of child welfare services, by critically assessing current managed care principles and practices.

The interviews for the assessment of the nine studies were conducted in-person and by telephone with key social service agency staff involved in the planning, dissemination, and utilization of the nine research projects. The interviews with staff focused on three areas of inquiry: (1) the ways the agencies disseminated project reports; (2) the impact of the projects; and (3) factors influencing the dissemination and utilization of project reports. The goal of the study was to learn more about how to foster the utilization of research findings and improve dissemination strategies.

Study participants were selected in consultation with the social service agency directors who were the first people to be interviewed regarding the research projects conducted in their respective counties. The convenience sample consisted of a total of 16 respondents (12 in-person and 4 telephone interviews) who are middle and senior managers in the four counties. These respondents were involved in the design of the study, the data collection, and/or the data interpretation process leading to articulating clearly understandable and relevant recommendations. These respondents were also the primary consumers of the exploratory studies and therefore in the best position to assess dissemination and utilization.

The senior author of this paper conducted all the interviews and recorded all the data. The data from the interviews were categorized and content analyzed for major cross-cutting themes using key concepts from the literature. For example, questions about dissemination were analyzed in terms of their "decision-making" or "conceptual" orientation. Similarly, the utilization concepts of "instrumental," "conceptual," and "persuasive" were used to categorize intervention question responses. And finally, the organizational factors impacting research utilization were utilized in the data analysis process.

The design limitations of this follow-up study are important to note. A more rigorous design would have replaced a convenience sample with a more randomized sampling plan. Second, given the limited resources to carry out the study, there were several threats to both external validity (multiple factors impacting the frenetic pace of agency life) and internal validity (self-interests of respondents or the complexity of cross-site, cross-study comparisons). As a result, this follow-up study focuses more on identifying and understanding research process improvements than on easily generalizable findings.

Results

This section provides general findings on dissemination and utilization of the research reports. The concepts of instrumental utilization, conceptual utilization, and persuasive utilization were used to organize the findings. The section concludes with an examination of the three organizational factors that influence research utilization: (1) research relevance, (2) practice implications, and (3) communication patterns.

Agency Dissemination

Research dissemination is a set of activities designed to make research findings and recommendations available to relevant audiences. Agency personnel identified 12 types of stakeholders (organizations and individuals) who were affected directly or indirectly by the recommendations of the research projects. The most commonly cited categories of stakeholders to receive research reports were internal agency staff, external service networks and providers, community groups and government and legislative representatives.

All of the reports were disseminated internally to agency management staff, including executive team members and/or senior managers. Three reports also were shared with supervisory and line level staff because of their relevance to service delivery and/or participation in the study (Family Reunification Outcomes, General Assistance Demographics, and Gay/Lesbian Foster & Adoptive Parent Policy Analysis). The majority of the studies were primarily of interest to management staff since they provided basic data about service delivery or administrative processes.

The majority of agencies reported no specific or formal plan for internal dissemination and no specific staff meeting to present the report other than listing it on a regular staff meeting agenda. Without a dissemination plan, agency staff reported some confusion about whose role it was to distribute reports beyond those members closely involved with the study. As a result, dissemination varied widely across the nine studies from senior staff only to staff inside and outside the agency. Only limited senior staff attention had been given in each study to the possible array of stakeholders who might have an interest in the study. Similarly, none of the scope-of-work statements included any reference to a dissemination plan other than preparing copies of the report for dissemination by agency senior staff.

When asked about what they would do differently to disseminate the results of their study today, staff closely allied with the study identified additional stakeholders, including the statewide Child Welfare Directors Association, community groups, and agency supervisors who might have benefitted from the information included in the report or whose relationships would be strengthened through the sharing of information. In general, study respondents recommended the implementation of a more deliberate and systematic dissemination process in the agency as well as regional discussions of findings and recommendations.

Research Utilization

In contrast to research dissemination, research utilization is reflected in the different ways the studies were designed and implemented. The major categories for organizing the utilization findings include (a) instrumental utilization, (b) conceptual utilization, and (c) persuasive utilization.

Based on an analysis of the interview data on utilization, the findings were categorized using the following criteria:

1. Each study met the definition of "conceptual utilization" if one or more of the following responses were noted: a) "the research project helped us think more clearly about the issues (goals, process, outcomes, etc.)," b) "it helped us see the issues in a new light", c) "it encouraged us to dig deeper with a follow-up investigation," and/or d) "it generated new questions and ideas to be pursued."

2. Each study met the definition of "persuasive utilization" if one or more of the following responses were noted: a) "we used the findings/recommendations with legislative bodies (county or state) or judicial bodies (Superior Court)," b) "used findings to educate staff inside and outside the agency," and/or c) "used findings to strengthen the work of local planning groups."

3. Each study met the definition of "instrumental utilization" if one or more of the following responses were noted: a) "used the study to expand/modify services," b) "used the study to increase local community collaboration," c) "used study to establish a new agency function," and/or d) "used the data collection instrument for on-going use in service monitoring."

While some of the studies reflected all three types of utilization (instrumental, conceptual, and persuasive), that was not the case for all of them. While there were no working hypotheses about types of

utilization, it appears that the degree to which the findings matched or complemented the agency's current plans, the more likely it was for the utilization to reflect all three types (conceptual, persuasive, and instrumental). As noted in Figure 2, this was the case for the first four studies. However, studies numbered 8 and 9 had either controversial or complex findings and therefore were utilized more conceptually than instrumentally or persuasively. Over half of the studies (5 out of 9) had one or more of the research report recommendations implemented as noted in the "instrumental" column in Figure 2.

Instrumental Utilization. Instrumental utilization involves the extent to which the recommendations of each study were implemented. For example, study #1 (assessing the needs of the homeless) illustrates instrumental utilization with respect to the impact of the study on decision-making. The agency made considerable progress in implementing the recommendations as a result of hiring a new Homeless Coordinator who collaborated with the community and built a coalition of agencies to develop: (a) a year-round shelter, (b) an extended rent-broker service providing four month rent credit to employed homeless persons (funded by U.S. Department of Housing and Urban Development), (c) expanded housing for homeless families leaving a violent situation or those recovering from substance abuse, (d) increased employment and health screening services tailored to the special needs of homeless people, and e) a successful HUD grant that involved different stakeholders in building continuum-of-care programs. Conceptual Utilization. Conceptual utilization refers to the use of research to influence thinking about issues related to future program and policy development (Freeman & Rossi, 1993). While all nine studies had a conceptual impact on the agencies, the managed care in child welfare policy study best illustrates conceptual utilization.

The managed care study was initiated to gather information regarding restructuring the delivery, management, and financing of child welfare services with respect to the applicability of managed care principles and tools. The sponsoring agency is now using the concepts to explore the feasibility of increased contracting of child welfare services. Other departments within the agency also are interested in exploring the application of managed care practices, such as contracting out welfare-to-work employment services based on specified incentives and outcomes.

Persuasive Utilization. Persuasive utilization refers to using research results to support or refute policy positions. Six of the nine research studies were used in this manner as illustrated by the studies on foster care rate setting and building a public information capacity. The foster care rate setting study was designed to assess various state-wide approaches to developing payment rates for foster care. The sponsoring agency and its foster family association (independent providers of foster care services) had been concerned about the loss of foster and adoptive families due to low payment rates and wanted to pursue a legislative remedy. When the research report was finished, it was distributed to legislative staff advocates at the state and county levels where it successfully contributed to the Agency's legislative agenda to increase the rate of foster care reimbursement by six percent.

Another study that focused on public information examined how public relations are currently carried out in the public and private sectors in order to assist the sponsoring agency in exploring the feasibility of establishing its own formal public information function. As a result of this study, a new public information office was established. The new public information officer was able to convince staff (traditionally reluctant to use the media) that it was in the best interest of clients and the agency to share information about agency services with the community. The report served as an educational document to help orient and train staff on the public information functions of a social service agency.

As a result of implementing the study recommendations, more staff members understood the importance of the public information function and were involved in public information roles such as distributing flyers in the community (e.g., PTA, churches, and chamber of commerce). In addition, the agency developed a Speakers Bureau with 34 agency staff members making presentations on topics such as implementing welfare reform. As a result of these activities, the agency director has received positive feedback from the Board of Supervisors, business people, and other community members who have taken time to notice the good work of the agency.

Organizational Factors Affecting Utilization

In this section the findings on utilization reflect the following organizational factors: (a) relevance of the research to agency operations, (b) situational factors (timeliness and potential conflicts), and (c) communication patterns (scope of work and written/oral communications).

Relevance of the Research

As noted in the review of the literature, research utilization is related to the relevance of the study's findings and recommendations to decision-makers. Administrators are generally interested in concrete recommendations that can be implemented and evaluated within existing agency operations. One respondent summarized some of the reasons why research is viewed suspiciously by management staff: (1) the inability of research findings to address external policy and funding constraints, (2) most research reports do not translate findings and recommendations into specific action steps, and (3) insufficient time and/or interest to either analyze the findings or utilize the research recommendations. These concerns may relate to unrealistic expectations about the direct relevance of research for current practice. However, these concerns are very important for those flaming research recommendations and promoting utilization by embedding outcomes of the study within the current flow of agency operations.

Despite these concerns, two studies were especially relevant to the interests program managers involved with the placement of children: (a) Quality of Kinship Care and (b) Gay/Lesbian Foster and Adoptive Homes. The Kinship Care study was distributed to key managerial staff (executive team, program managers, director of foster parent recruiting, children's shelter director and members of the foster parents committee). In contrast, the Gay/Lesbian study was shared with all levels of agency staff as part of a campaign to educate them about gay and lesbian families.

It is important to note that a few of the research studies reflected some level of controversy, such as managed care (are such cost-saving strategies relevant to child welfare?), homelessness (how large is the population and have we done enough?), and placement of children in gay and lesbian homes (will these families corrupt the morals of innocent children?). In many instances the research reports led to policy and program changes that helped to lower the level of controversy and increase informed decision-making.

And finally, the relevance of the studies was greatly enhanced by the involvement of an operational manager in the research design and implementation of the results. Of the four most utilized studies, all had an operational staff member involved in the research and/or the implementation of the recommendations. Staff expertise related to the research study was a critical element in the successful utilization of the study recommendations. Similarly, the involvement of a community group or staff committee increased research study ownership and the potential for increased dissemination and utilization. The studies that involved only the agency directors and research staff were utilized less.

Situational Factors

Several situational factors were identified by agency personnel and research staff such as limited financial resources (all projects were budgeted for $25,000 and some included two components for a total of $50,000), time constraints (all studies needed to be completed in 6-12 months), and agency or university delays in processing the scope of work or gaining access to data sources. This section expands on these findings by examining the issues of timeliness and potential conflicts.

While only one agency reported significant time delays involving the University's Committee for the Protection of Human Subjects (potential client vulnerability to the misuse of data), most of the delays were caused by difficulties experienced by agency managers and university researchers in specifying the primary research question of interest to the agency (delays in receiving follow-up information, rescheduling meetings, etc.). In addition, the cumbersome approval procedures within agency and university bureaucracies made it difficult to undertake studies in a timely manner.

Since exploratory research can surface conflicting needs and interests, it is important to identify and address potential conflicts. For example, one agency wanted their research project to support their lobbying of the state legislature. Instead of conducting a piece of advocacy research which is not in the purview of university-based research, the researchers framed the scope of work to include a concluding section on "Policy Considerations" rather than specific recommendations. In another example, the study findings and recommendations conflicted with the agency's new service directions. The data from the study called for increased training and support services for clients prior to employment while the agency had recently adopted a "work first" welfare reform strategy whereby training and support services would follow the securing of employment. Since the agency had become more interested in identifying the factors that lead to successful transition from welfare to work, the study was not widely disseminated.

Communication

A third organizational factor that was found to influence utilization was communication. Most agency staff described communications with the research staff as open and effective. If there was any perception of communication problems, agency staff were quick to point out their own responsibilities in this area. For example, sometimes agency staff were slow in reviewing draft reports or were rushed and did not review them critically. Other busy staff needed oral presentations prior to reading the report to determine how they could use the findings. While agency directors actively sought input from senior management on their ideas for research projects, some noted that it was difficult for many key managers to think through how to use research data for policy and program development. Specific communication issues described in this section relate to the scope of work and the presentation of the final report.

Sharing information during the development of the scope of work was seen as essential for both researchers and agency staff in order to ensure that all sources of expertise, not just the literature, were consulted. While the research staff had a clear understanding of the issues to be addressed, the research methods to be used and the product to be delivered, there was a need in many of the studies for a clearer specification of the roles to be played by agency staff. This proved to be critical when staff assignments changed and new staff members were given responsibility for working with the research team. Similarly, the involvement of multiple departments within an agency can significantly delay the research process. So it is important to determine which staff, departments, agencies and stakeholders should be involved in clarifying the research objective, interpreting the findings, and assessing the recommendations.

With respect to presenting the final report, most agencies found the reports to be well written, informative, easy to read, and reflected a high level of analysis and clarity. Most of the respondents were especially impressed with the literature reviews. However, it was often noted that the final reports could have provided more connections between the recommendations and the necessary action steps for implementation. In only one study (Homeless Needs Assessment) did the agency staff invest considerable time in framing the recommendations within an overall action plan that helped to foster successful dissemination and utilization.

Understanding the audience for a report, as well as its external or internal use, are important factors for improving both the dissemination and utilization processes. The agencies often used their reports as parts of presentations to government and community bodies. Some developed Power Point presentations of the research findings and recommendations. Others developed brief, two page abstracts with bullets, articles for agency newsletters, and oral presentations using visuals and handouts at staff meetings and/or "brown bag" lunches. In general, most agency staff members who were not directly involved in the research project read only the Executive Summary that included recommendations. Several agencies suggested that oral presentations become a part of the scope of work in future studies.

Summary of Organizational Factors

After reviewing the organizational factors related to research relevance, the situational aspects of timeliness and conflict, and communication related to the scope of work and the presentation of findings and recommendations, the following highlights of major findings emerged:

Positive Results

1. Staff participation, especially the involvement of a senior manager, in the planning and implementation of research projects positively influenced research dissemination and utilization.

2. The research studies helped to increase the level of informed decision-making and lower the level of controversy within the agencies as well as the community.

Areas for improvement

3. Shared responsibilities across a number of different departments or stakeholders can significantly delay the research process, calling for increased efforts to coordinate by agency staff as well as the researchers

4. Successful dissemination of reports requires that the researchers invest more time and energy in identifying and understanding the interests of internal and external audiences.

5. Successful utilization of research reports is based on the realization that agency managers have different levels of understanding when it comes to using research findings and recommendations for policy and program development.

6. Oral presentations, beyond the distribution of the written research reports, were seen by busy staff as the most valuable way to present research results and explore the feasibility of implementing the recommendations.

7. Increased investments in communications are necessary to: (a) clarify the nature of the study and expectations for the outcome, (b) specify types of research and dissemination methods to be used, and (c) identify methods for addressing key issues (cooperation, shared accountability, conflict, and the translation of findings and recommendations into action steps).

Implications

The findings indicate that research dissemination and utilization varied across studies due to several organizational factors. The two most important are: (a) establishing clarity in the early stages of defining the scope of work, and (b) strengthening communication in the agency-researcher partnership before, during, and after the research project.

With regard to developing the scope of work, several activities could strengthen the research collaboration:

1) Include a plan for dissemination and utilization that involves staff in translating findings and recommendations into action steps.

2) Include opportunities for regularly scheduled review meetings to facilitate communications and promote increased understanding of agency issues by the researchers.

3) Expand the literature review process to include consultation with experts inside and outside the agency as a way to promote an increased understanding of the research topic and the organizational issues by the researchers.

4) Assist researchers in acquiring a first-hand familiarity with agency service programs and the array of relevant stakeholders.

5) Discuss the scope of work with all relevant parties within the agency.

The second major implication of this study relates to strengthening communication in the agency-researcher partnership. In this regard, the following suggestions emerged from the data:

1) Involve an agency staff member from the program area under study in all aspects of the research project coordination.

2) Identify areas of potential conflict between agency staff and research staff as early as possible in order to avoid compromising the study or its dissemination and utilization.

3) Identify opportunities to share the research results widely within the agency as well as with other interested parties in the community and region.

The implications of this follow-up assessment of nine exploratory studies suggest an agenda for future research in the following four areas: (a) scope of work, (b) the agency-researcher partnership, (c) the research dissemination process, and (d) media strategies.

Conclusion

This follow-up study identified some of the organizational factors that influence the ways in which social services agencies disseminate and utilize exploratory research findings. The three most salient organizational factors were: (a) establishing clarity in the early stages of defining the scope of work, (b) strengthening communication in the agency-researcher partnership during the entire research process, and (c) finding multiple methods for disseminating findings and increasing utilization. Future research is needed to assess the process of developing a scope of work while strengthening agency-researcher partnerships, as well as strengthening the research dissemination process.

University facilitated agency-based research is a team process that requires careful planning and communications. Given the time and resource constraints on completing these research projects, additional resources may be needed to assist agencies and/or researchers in disseminating and utilizing their research reports. It is clear that both researchers and agency stakeholders share responsibility for maximizing the use of the results and recommendations of exploratory research.

Figure 2
Types of Research Project Utilization

                                Conceptual
                                "Insights    Persuasive  Instrumental
                                for later     "Used to     "Altered
Research Projects              application"  advocate"    practices"

1. Homeless Needs Assessment        X            X            X

2. General Assistance Client        X            X            X
   Demographic Study

3. Gay/Lesbian Foster and           X            X            X
   Adoptive Homes

4. Developing a Public              X            X            X
   Information Strategy

5. Assessing Quality of Care        X                         X
   in Kinship Foster Care

6. Family Reunification             X            X
   Outcomes

7. Foster Care Rates                X            X

8. Long-Term AFDC Recipients        X

9. Managed Care as a Tool for       X
   Child Welfare Reform

Total                               9            6            5

References

Anderson, S. G. (2001). The collaborative research process in complex human services agencies: Identifying and responding to organizational constraints. Administration in Social Work, 25(4), 1-19.

Austin, M. J., Martin, M., Carnochan, S., Duerr Berrick, J., Goldberg, S., & Weiss, B. (1999). Building a comprehensive agency-university partnership: The Bay Area Social Services Consortium. The Journal of Community Practice, 6(3), 89-106.

Apfel, J., & Worthley, J. (1979). Academic technical assistance: The university and state government. Public Administration Review, 9/10, 408-414.

Beyer, J., & Trice, H. (1982). The utilization process: A conceptual framework. Administrative Science Quarterly, 27, 591-622.

Bullock, R., Gooch, D, Little, M., & Mount, K (1998). Research in practice: Experiments in development and information design. Aldershot, UK: Ashgate Publishing.

Casey, A. (1983). Developing dialogue between rehabilitation practitioners and researchers. Rehabilitation Literature, 44, 7679.

Cohen, L. (1977). Factors affecting the utilization of mental health evaluation research findings. Professional Psychology, 11,526-534.

Cronenwett, L. (1995). Effective methods for disseminating research findings to nurses in practice. Nursing Clinics of North America, 30, 429--437.

Cousins, J., & Leithwood, K. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56, 331-364.

DeMartini, J., & Whitebeck, L. (1986). Knowledge use as knowledge creation. Knowledge: Creation, Diffusion. Utilization, 7, 383-396.

Feldman, R. (1986). Group work knowledge and research. Social Work with Groups, 9, 7-14.

Funk, S., Tornquist, E., & Champagne, M. (1989). A model for improving the dissemination of nursing research. Western Journal of Nursing Research, 11, 361-367.

Gingerich, W. (1982). Teaching students to use single-case evaluation methods. Paper presented at the Council of Social Work Education Meeting, New York.

Green, L., & Johnson, J. (1996). Dissemination and utilization of health promotion and disease prevention knowledge: Theory, research and experience. Canadian Journal of Public Health, 87, S11-S17.

Holland, A. (1998). Some guidelines for bridging the research-practice gap in adult neurogenic communication disorders. Topics in Language Disorders, 18, 49-57.

Huberman, M. (1994). Research utilization: The state of the art. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization, 7,13-33.

Ingram, D. (1998). Research-practice relationships in speech-language pathology. Topics in Language Disorders, 18, 1-9.

Larson, J. K. (1981). Knowledge utilization: Current Issues. In R. E Rich (Ed.), The knowledge cycle. Beverly Hills, CA: Sage Publications.

Leviton, L., & Hughes, E. (1981). Research on the utilization of evaluation: A review and synthesis. Evaluation Review, 5, 525-548.

McNeece, A., Dinito, D., & Jonson, J. (1983). The utility of evaluation research for administrative decision making. Administration in Social Work, 7, 77-87.

Millman, J. (1990). The dissemination of psychological research. American Psychologist, 5, 668-669.

Mizrahi, T. (1992). The future of research utilization in community practice. In A. Grasso & I. Epstein (Eds.), Research utilization in the social services. New York: Haworth Press.

Mutschler, E. (1984). Evaluating practice: A study of research utilization by practitioners, Social Work, 29, 332-337.

Nilsson, K., & Sunesson, S. (1993). Strategy and tactics: Utilization of research in three policy sector contexts. The Journal of Applied Behavioral Science, 29, 366-383.

Pelz, D. (1978). Some expanded perspectives on the use of social science in public policy. In M. Yinger & S. Cutler (Eds.), Major social issues: A multidisciplinary view. New York: Free Press.

Pettengill, M., Gillies, D., & Clark, C. (1994). Factors encouraging and discouraging the use of nursing research findings. Image, 26, 143-147.

Reid, W., & Fortune, A. (1992). Research utilization in direct social work practice. In A. Grasso & I. Epstein (Eds.), Research utilization in the Social Services. New York: Haworth Press.

Rich, R. (1977). Uses of social science information by federal bureaucrats: Knowledge for action versus knowledge for understanding. In C. Weiss (Ed.). Using social research in public policy making. Lexington, MA: Lexington Books.

Rosen, A. (1983). Barriers to utilization of research by social work practitioners. Journal of Social Service Research, 6, 1-15.

Rothman, J. (1980). Social R & D: Research and development in the human services. Englewood Cliffs, NJ: Prentice-Hall.

Stricker, G., & Keisner, R. (1985). The relationship between research and practice. In G. Streicker & R. Keisner (Eds.), From research to clinical practice: The implications of social and developmental research for psychotherapy. New York: Plenum.

Staller, K. M., & Kirk, S. A. (1998). Knowledge utilization in social work and legal practice. Journal of Sociology and Social Welfare, 25(3), 91-113.

Tripodi, T., Fellin, P., & Meyer, H. (1983). The assessment of social research. Itasca, IL: Peacock.

Weiss, C. (1980). Social science research and decision making. New York: Columbia University Press.

Wilcox, M., Hadley, P., & Bacon, C. (1998). Linking science and practice in management of childhood language disorders: Models and problem-solving strategies .Topics in Language Disorders, 19, 10-22.

TERESA DAL SANTO
SHERYL GOLDBERG
PAMELA CHOICE
University of California at Berkeley
Center for Social Services Research

MICHAEL J. AUSTIN
University of California at Berkeley
School of Social Welfare

Teresa Dal Santo Center for Social Services Research University of California 120 Haviland Hall Berkeley, CA 94720-7400 [e-mail]