A synergistic program evaluation model partnership between a state agency and a university program.

Public rehabilitation programs and university rehabilitation counselor education programs have been historically linked since the passage of the Amendments to the Vocational Rehabilitation Act of 1954 (P.L. 83-565). These amendments provided federal support to university programs to facilitate the development and sustainability of rehabilitation counselor education programs in order to provide the public rehabilitation program, and related practice settings, with a continuous flow of master's level educated rehabilitation counselors. Although the occupational status of rehabilitation counseling was established in the 1920's, it was not until the mid 1950's with the passage of this landmark legislation (Amendments to the Vocational Rehabilitation Act of 1954) that the discipline embarked on a series of significant and ongoing professionalization initiatives, which included at the forefront, the development of pre-service educational programs (Wright, 1980; Leahy & Szymanski, 1995; Rubin & Roessler, 2001; Leahy, 2004; Fabian & MacDonald-Wilson, 2005).

In the years that followed, rehabilitation counseling programs developed at a rapid pace, based in part through the ongoing support provided by competitive grant funds available through the Rehabilitation Act Amendments (Title III), and the partnership arrangements established with the public rehabilitation programs in each of the states where these academic programs were initiated (Wright, 1980). The nature and value of these partnerships however has been varied over the years in both effectiveness and satisfaction, as both the public rehabilitation programs and educational programs have evolved (IRI, 1999). In the late 1980's and early 1990's there was a national initiative launched to strengthen these partnership relationships through the design and implementation of an annual training conference that would bring educators and state agency personnel together on a national platform to collectively address training issues in the public rehabilitation program. This partnership, formed at the national level by the Rehabilitation Services Administration (RSA), the National Council on Rehabilitation Education (NCRE) and the Council of State Administrators of Vocational Rehabilitation (CSAVR) continues today. With the inclusion of the Comprehensive System of Personnel Development (CSPD) in the 1992 and 1998 Amendments to the Rehabilitation Act of 1973 that mandates specific educational requirements for qualified personnel, the partnership between the educational community and the public rehabilitation programs became an even more critical issue (IRI, 1999).

Today, the typical relationship between a state agency and educational program includes agency membership on advisory councils, and input into the graduate training program, serving as adjunct instructors, making practicum and internship sites available to students, and employment options for program graduates. It also includes course offerings for CSPD in order to upgrade agency staff educational qualifications, and in some cases, continuing educational offerings for the professional development of agency staff.

One substantive area that is not typically an integral aspect of the partnership is research and program evaluation. While both partners may individually participate and benefit at some level from research sponsored by National Institute on Disability and Rehabilitation Research (NIDRR), unique studies funded through RSA (e.g., Longitudinal Study of the Vocational Rehabilitation Services Program), and other funding sources, the partnerships that have evolved over the years usually do not include this component. Given the level of accountability for outcomes, the complexity of the process, and limited resources of most state agencies to effectively devote time, energy and expertise to program evaluation, this appears to be an ideal place for university programs to further contribute to the developing partnership.

The purpose of this article is to describe a program evaluation partnership that has been developed and sustained in Michigan for the past four years by the Department of Labor and Economic Growth - Rehabilitation Services (herein referred to as MRS) and the Office of Rehabilitation and Disability Studies at Michigan State University. We will describe the evolution of the general relationship that led to the design and implementation of the program evaluation partnership, and describe the status of the project, including scope of work and how the partnership functions. Finally we will conclude with the impact of the partnership on the state agency and university program, and describe the lessons learned from the design, implementation, and maintenance of this unique partnership model.

Evolving Partnership between MRS and MSU 

The Office of Rehabilitation and Disability Studies (ORDS) at Michigan State University (MSU) and Michigan Rehabilitation Services (MRS) have had a long-standing, productive, and mutually beneficial relationship that extends back some 45 years or more. During this time, the two organizations have collaborated on numerous projects, training grants, and professional development activities. All of the typical activities described earlier, which are associated with a meaningful and collaborative partnership between a state agency and a university program, were well in place by the early 1990's. In addition, MSU began working in collaboration with MRS in relation to dissertation research initiatives (e.g., access to MRS delivery system for research subjects; design, implementation, and analysis of data for descriptive and intervention studies), and establishing unique research apprenticeship opportunities for a few doctoral students within the in-house MRS Program Evaluation Unit.

Problems Viewed as Opportunities

One of the problems that MRS was experiencing during the 1990's was a lack of stability in their ability to staff the program evaluation function of the agency. Frequent turnovers, and difficulty finding and retaining qualified personnel were persistent problems. In addition, the agency was very active in implementing a series of new strategic directions and programs and were unable to adequately measure the impact of these initiatives due to lack of capacity and consistency in the research and program evaluation area. In addition, in late 1999, MRS implemented a new client case management system, encompassing all the RSA 911 data fields, agency specific data fields, and all case notes in a readable format, available in real time. To have quality research with timely information and consistently evaluate programs over multiple years became a greater need and objective of the agency. The idea of partnering with MSU's Rehabilitation Counseling program for program evaluation, as well as providing the faculty with information to be used in the classroom regarding daily issues confronted by MRS counseling staff was an intriguing proposition that the agency began discussing internally.

With these needs in mind, the agency contacted MSU to explore whether the existing partnership could be extended through a funded partnership to include program evaluation. These discussions began in the late spring of 2001. For MSU, this represented an opportunity to provide a rich, real-world, field-based environment to enhance the training of doctoral students in research and program evaluation. It also provided a unique revenue stream to fund doctoral students as Research Assistants as they pursued their studies in rehabilitation counselor education. For MRS, this collaborative effort represented an opportunity to stabilize, and greatly expand its research and program evaluation capacity to meet both the routine regulatory reporting requirements, and to use program evaluation, for the first time, as a major component of its continuous improvement strategy for the agency.

The proven track record of joint accomplishments over the years, and the specific knowledge and trust each of the organizations gained through previous partnership activity were essential beginning elements in this new expanded partnership arrangement. MSU was in an informed position, through these previous experiences with the state agency, to provide both the type and level of program evaluation and research services required by the agency.

Project Excellence: Design and Scope of Work MSU and MRS designed and entered into this funded partnership in the fall of 2001, with the goal of providing long-term stability to the existing program evaluation function of the state agency. In addition, the partnership expanded capacity to address new related evaluation and research questions that provide the agency with data (quantitative and qualitative) and analysis regarding the impact of the rehabilitation services provided to citizens with disabilities in the State of Michigan. The title of this partnership is Project Excellence: A Program Evaluation Partnership. The title was selected as an ongoing reminder that the ultimate purpose of the program evaluation is to improve the excellence of service delivery and outcomes. The specific project goal was as follows: Project Excellence will further develop and improve the application of qualitative and quantitative data analysis with respect to Michigan Rehabilitation Services culture and programs while prioritizing regulatory responsibilities and program excellence values. Prior to the initiation of Project Excellence in fiscal year 2002, MRS and MSU developed a data matrix to provide a framework for the principle objectives for the ongoing project.

Based on this matrix, the first prioritized project was an in-depth review and analysis of the agency's RSA 911 data for fiscal years 1999 through 2001. This was not only the logical place to begin, but critical to developing a common foundation of knowledge about the current status of the agency. The findings of the analysis set the baseline for identifying and tracking trends in the data related to a wide variety of issues, and most importantly, the impact of new initiatives on customer outcomes. The findings of this project also identified additional areas of interest that the agency wished to explore, which in turn became the next set of prioritized projects using the RSA 911 data.

Both prior to and following the analysis of the RSA 911 data, an area of concern for both partners was the reliability of the data. When the new case management system was implemented in 1999, data entry responsibilities shifted from rehabilitation assistants to rehabilitation counselors. This was not only a new responsibility for counselors, but required new skills as well. Although extensive training on the system was provided for all staff, the question that remained was whether the data entered was accurate or not. Project Excellence conducted a qualitative study that identified data fields that were most challenging for counselors, counselors' recommendations for improving the system, and on-going technical training needs of the counselors.

One of the surprising findings of the study was that while some counselors knew that data were required by RSA, the majority did not know why the data were being collected or why accuracy was important. As a group, they stated that if they knew what was important and why, they would make sure it was correct. Based on the findings of this study, the agency took immediate steps to revise the case management system to better meet counselors' needs, and developed and implemented training programs about the how the data were used by RSA (e.g., the performance indicators) and by the agency. The impact of these efforts was then monitored and findings indicate that while there were random errors in the data, the overall reliability of the data has improved significantly.

As the partnership continued to develop, the scope of work expanded as planned to include agency initiatives such as business services, youth transition services, and services to minority populations. Project Excellence staff began working with each workgroup to develop an understanding of the initiative, the current status in terms of implementation, and to identify any program evaluation needs. Because Project Excellence was often entering the process after the workgroup had already been established and working for several years, it was often initially challenging to determine how to integrate Project Excellence into the process. For some initiatives, such as youth transition services, Project Excellence provided consultation services. For other projects, such as business services, the workgroup had already identified some program evaluation objectives. At the time that Project Excellence joined the services to minority populations workgroup, they where already moving forward with a plan to evaluate the multicultural competencies of all agency staff. This project happened to dovetail perfectly with the dissertation research interests of an advanced doctoral student on the Project Excellence staff. The workgroup and the student, under the supervision of the principal investigator for Project Excellence, worked collaboratively to develop the research protocol, select appropriate instruments, and to notify staff about the study. As a result, both parties benefited and the study had more depth than it would have had if they had worked independently of each other.

With the growing awareness of Project Excellence as an available resource during the first three years of the partnership, the requests for support and the scope of work also expanded. To date, 30 projects have been completed and an additional 14 are in progress. Areas of investigation include: customer satisfaction, best practices in serving minority populations, employer satisfaction, development of process and outcome program evaluation goals and objectives for a federally funded High School High Tech grant project, an interdepartmental adjudicated youth program, a statewide comprehensive needs assessment, evaluation of the impact of an organizational restructuring process, as well as numerous studies to investigate findings identified in the annual review of the RSA 911 data.

Working as Partners

Although extensive discussions were held prior to the project start up, it was impossible to fully anticipate the issues and needs of each partner that would arise in the development phase of the project. The first year of the project was consumed with the details of getting the new project implemented as quickly and efficiently as possible. The tasks of training Project Excellence staff on the database, overcoming security obstacles to enable access to MRS data systems, analyzing the agency's previous evaluation reports, establishing tasks, protocols, and project lists while simultaneously getting the first projects completed required good communication, creative problem solving and often patience on everyone's part. This collaborative effort to achieve a common goal, however, laid the foundation for the strong working alliance, based on mutual trust and respect, necessary for the success of this project.

The working alliance was tested in 2003 when the agency reorganized due to a State of Michigan early retirement program. As a result of the early retirements, the majority of the agency staff that had been involved in the planning and initial implantation of the Project Excellence model was leaving the agency. Although there remained a strong commitment to the project, where and how Project Excellence would fit in the new structure had to be determined. Following extensive discussion, the program evaluation function was placed organizationally with the division director responsible for technology, the client case management system, budget and audit. This was a logical placement because audit follow-up, similar to evaluation, requires coordination with the database support staff as well as training and policy units within the agency. The system for audit follow-up was already in place and working well.

This placement stimulated the next stage of the partnership development process. From the beginning, a critical common goal was that program evaluation would be integral part of program development. Up until this point, however, a systematic, structured model for integrating project findings into the day to day agency activities had not been developed. Following the organization restructuring efforts, a team composed of Project Excellence staff, the MRS division director, the staffs supporting the client case management system and preparing the 911 Report, the training supervisor, and a staff liaison (hereafter referred to as Implementation Team) met to establish a plan for soliciting project ideas and recommending which to pursue, disseminating project findings and developing an implementation plan to address recommendations from completed projects. This process was called Project Excellence Work Flow (see Figure 1) and ongoing and special projects were filtered through this Implementation Team approach. After consulting with the agency director, the plans were implemented. This same model is still in place today.

[FIGURE 1 OMITTED]

Annually, the agency puts a call to all staff to solicit program evaluation ideas. Each submission is discussed and organized by the Project Excellence Implementation Team. Suggestions are sorted as to: a) required by federal, state, or department; b) ongoing from previous year; c) agency priority or new initiative; d) nice to do and; e) does not apply. Projects are prioritized within each category, with rationale. The full list is referred to agency executives, who meet with the project's principal investigator to discuss feasibility of the projects and timeframes. Each year's approved project list is included in the contract.

Following the timeline and prioritized project list, Project Excellence staff meets with the MRS Division Director, the liaison, and the manager responsible for each project to be evaluated. A plan of action is developed and the work begins. When possible, Project Excellence staff begins working with the MRS project work group from the inception of the project. In this way MSU staff can ensure that the qualitative and/or quantitative evaluation methodology, including any necessary data elements in the case management system, are in place to evaluate the key performance indicators when the project is implemented. Once the evaluation is completed, Project Excellence submits a draft report to the work group to review and receives feedback from the work group about the report format and terminology and works with the work group to generate recommendations based on report findings. This step is essential to ensure that the report findings are presented in a "user friendly" way for MRS staff at all levels of the agency and that the recommendations are comprehensive. The final report is then issued to the work group and the Implementation Team.

Critical to continuous improvement and quality customer service is the Implementation Team phase of each project. While reviewing the project report, the Implementation Team identifies items requiring follow-up. For example: If a statistic reported in the Annual Analysis and Review of the RSA 911 data is incongruous to other data or what we know to be true, software changes may be needed for clearer understanding by persons doing input, additional training to clarify a data element, and also clarification and data support/help desk notification to assure accurate response to inquiries. As each action is identified, a responsible party is named to ensure that the action is completed. Our staff liaison notifies the responsible party. The party self-identifies a due date, and the liaison follows up to confirm the item's completion and to obtain supporting documentation. Finally, the follow-up analysis in subsequent years confirms resolution or indicates additional action steps are needed.

Impact on State Agency

Project Excellence staff is involved in the design and data needs of every agency initiative. This has provided us with the ability to have the data elements needed for quality evaluation in place during program implementation.

Timely and comprehensive program evaluation has enabled MRS to evaluate service delivery using both quantitative and qualitative data. As service goals are discussed, we are able to establish a baseline, actions needed to improve, and finally confidence that the goal is met. We better understand training needs, and are able to directly address them and confirm training effectiveness.

Projects during 2005 included establishing a method to evaluate a new client orientation program prior to implementation; sub-recipient grant evaluation; employer satisfaction; customer satisfaction; services provided to and program effectiveness for minority populations; and continued analysis and review of annual RSA 911 data. Action items range from improving underreported data, assuring correction of a decline in a performance factor, understanding why disability types and characteristics are increasing and decreasing within the population of persons we serve, and understanding changes in our referral sources. We are delighted that in two years we have reduced the action items associated with the RSA 911 data from 20 issues with over 60 actions needed to eight issues with 22 actions needed.

Impact of the Project on the University Program

There is a current trend toward partnership development between universities and local communities in which the resources and needs of both interrelate. On the forefront of this trend, our formed partnership has resulted in mutual benefits to both parties. At Michigan State University, we see the apprenticeship model for graduate training at MSU flourishing and doctoral student research opportunities expanding. We have also seen significant benefits for faculty and the program in general.

The apprenticeship model for graduate training matches a graduate student with a faculty advisor who can guide the student in the multiple tasks required in completing research (Zanna & Darley, 1987). According to Leavitt (1993), the apprenticeship process is conceptually right, but founders in practice because it tends to isolate the doctoral student from the outside, relevant world. Within Project Excellence, the criticisms of the apprenticeship model are challenged because students are given the opportunity to work in a mentoring relationship with faculty, while engaging in activities highly relevant to the current undertakings of the rehabilitation counseling profession. The partnership has modified the traditional apprenticeship model in such a way as to provide senior graduate students with the opportunity to mentor newer students who recently join the project. Thus, senior students have the opportunity to experience tasks typically observed in faculty positions. Early exposure to teaching, advising, and research properly introduces these students to future careers in academe.

As stated earlier, the Project Excellence partnership provides a unique revenue stream to fund doctoral students as Research Assistants as they pursue studies in rehabilitation counselor education. Doctoral students have reported that involvement in Project Excellence has exposed them to literature, various methodologies, and data-all which have provided practical experience with which to connect ideas learned from activities in their research classes. The ability to access and utilize state agency databases and to make comparisons with longitudinal or Federal datasets provides a richer pattern of information from which students draw understanding and gain knowledge about outcome research and its accountability within systems and organizations.

The positive impact on faculty and the program in general are also clear benefits attained from our partnership model with the agency. Faculty, as a result of their participation with PE, has a much more contextualized knowledge of issues and challenges that the public rehabilitation program faces each day. This knowledge and direct contact with the field is brought into our teaching, advising and curriculum decisions.

Lessons Learned in Designing, Implementing and Maintaining the Partnership

With mutual trust and a long history of collaborative efforts at the core, we have learned that we can continue to implement and maintain the partnership amid changing priorities and evolving improvement goals. As the partnership identifies ongoing program evaluation needs, we draw upon our experiences together to create possibilities for a sustainable future. Success factors that advance us toward these possibilities are: creative leadership; common values; and shared goals.

Creative leadership has served the purpose of motivating staff, developing capacity where limitations emerge, and monitoring performance of all parties involved in a particular project (e.g. workgroup performance). For example, when a project is not progressing as planned, this indicates to the partnership leadership that it is time to reevaluate and identify methods to ameliorate the process. At times, intervention at the leadership level is needed to support or facilitate the work groups planned activities. For example, if a project plan includes data collection from field-based staff, we have found that if the agency director informs staff about the purpose of the study and requests their support, the response rate will be higher. If the difficulty is at the work group level, leadership will challenge work group members to develop action steps that help the workgroup move forward. Such incremental advancement toward completion of tasks provides motivation and a sense of responsibility for being part of a group (De Pree, 2001). Wherever limitations arise in the complex process, management utilizes its position and/or the shared competencies of members within the partnership to expand restrictions, thus accelerating the flow of operations.

Relying on common values has also been a significant factor in maintaining the partnership, and both MRS and MSU view our adherence to "learning organization" values as a critical component of program evaluation enterprise. According to Senge, Kleiner, Roberts, Ross, Roth, and Smith (1999), learning organizations are places where people expand their capacity to create truly desired results and are continually learning to see the whole together. Many partnerships fail because individuals who play an important role in the organization are invisible to leadership and partnership stakeholders (Bernal, Shellman, & Reid, 2004). Within Project Excellence, all individuals across all levels of the organization are involved in program evaluation efforts, and total employee involvement is valued. For instance, MRS counselors and rehabilitation assistants use a case management tracking system that directly implicates program evaluation endeavors performed through the partnership. Without engaging in and valuing information gleaned from dialogue (Senge, et al., 1999) between Project Excellence and MRS staff, accurate data input and collection may inhibit the reliability of client and programmatic information used in many evaluation projects. As we apply the knowledge learned across all members within the organization and across the partnership, we are more likely to reach our evaluation goals.

Additionally, we have learned that our shared goals of improving service delivery and providing a variety of research opportunities to doctoral students has made the partnership both successful and effective. Zanna and Darley (1987) indicate in their research on apprenticeship models that as doctoral students genuinely engage in working on research, their participation invariably makes the study better. In our partnership, students are expected to be involved and have input at every step in the program evaluation process. All doctoral students routinely complete literature reviews. When a student involved with Project Excellence completes a literature review, the purpose is to supplement what MRS currently understands about specific project (e.g., Project Excellence student literature search is additive to MRS history of service provision to minority populations). Information from both sources provides an objective and subjective perspective that serves to enrich the design of or methodology for a proposed project. As the more developed quantitative or qualitative approach unfolds, and as students actively engage in data collection, analysis and report writing, MRS' accountability for its outcomes is supported. Hence, as the partnership assists the organization in accounting for its outcomes, the more readily MRS is able to demonstrate whether its goal of improvement of service delivery for citizens with disabilities in Michigan has been attained.

Conclusion

Based on experience with this expanded research and evaluation partnership over the past five years, we believe this type of model relationship between a state agency and a university not only has a great deal of merit, but should be pursued in other states where rehabilitation counseling or related programs reside within research intensive universities, particularly those with doctoral preparation programs. Obviously, this type of unique partnership arrangement requires a significant commitment on the part of both parties to be successful, and the funding mechanism to make it happen. It has been our experience that for this type of project to be successful, the program evaluation functions and related activities provided need to be imbedded in the fabric of the service delivery system. Although this requires an incredible commitment of time and energy by both parties, the end result is well worth the investment.

References

Bernal, H., Shellman, J., & Reid, K. (2004). Essential concepts in developing community-university partnerships. Public Health Nursing, 21, 32-40.

De Pree, M. (2001). Creative leadership. Leader to leader, 20, 10-13.

Fabian, E.S. & MacDonald-Wilson, K.M. (2005). Professional practice in rehabilitation service delivery systems and related system resources. In R. Parker, E.M. Szymanski, J.B. Patterson (Eds.) Rehabilitation counseling: Basics and beyond. Austin: Pro-Ed.

Institute on Rehabilitation Issues (1999). Achieving employment outcomes through VR counselors who meet the comprehensive system of personnel development requirements. AR: University of Arkansas.

Leavitt, H.J. (1993). The business school and the doctorate. Selections, 9, 12-24.

Leahy, M.J., & Syzmanski, E.M. (1995). Rehabilitation counseling: Evolution and current status. Journal of Counseling and Development, 74, 163-166.

Leahy, M. J. (2004). Qualified providers of rehabilitation counseling services. In D. R. Maki & T. F. Riggar (Eds.), The handbook of rehabilitation counseling. New York: Springer.

Rubin, S.E. & Roessler, R.T. (2001). Foundations of the vocational rehabilitation process. Austin: Pro-Ed.

Senge, P., Kleiner, A., Roberts, C., Ross, R., Roth, G., & Smith, B. (1999). The Dance of change: The challenges of sustaining momentum in learning organizations. New York: Doubleday/Currency.

Vocational Rehabilitation Act of 1954, 68 Stat. 652.

Wright. G.N. (1980). Total rehabilitation. Boston: Little Brown.

Zanna, M.P., & Darley, J.M. (1987). The Compleat academic: A practical guide for the beginning social scientist. New York: Random House.

Michael J. Leahy

Michigan State University

Virginia Thielsen

Michigan State University

Darlene A. G. Groomes

Michigan State University

Lori Shader-Patterson

Michigan Department of Labor and Economic

Growth-Rehabilitation Services

Jaye Shamsiddeen

Michigan Department of Labor and Economic

Growth-Rehabilitation Services

Michael J. Leahy, 463 Erickson Hall, Michigan State University,

East Lansing, MI 48824-1034.

Email: [e-mail]