Sense-making of institutionalizing assessment.

Abstract

Interviews and focus groups were held with key college constituents (administrators, faculty members, program coordinators and department chairs) to better understand the process of implementing and sustaining assessment practices in research-intensive commuter institution in the mid-south. Processes that facilitate or hinder institutionalization of assessment are discussed, in light of recent theoretical frameworks.

Introduction

Discussions pertaining to institutional change abound (duToit, 1996; Fullan,1991; Pantake, 1998). Some believe that change is best described as a process rather than an event and that there is a need for ongoing management of organizational culture in order to sustain a change. This article will review observations about the process of implementing assessment practices in two different colleges within a research-intensive commuter institution in the mid-south. Key elements of successful implementation processes, as well as barriers to institutionalizing assessment, will be discussed. A useful definition of assessment will be shared to set the context for this work:

   Assessment is the systematic collection, review, and use of
   information about educational programs undertaken for the
   purpose of improving student learning and development
   (Palomba Banta, 1999, p. 4).

Institutionalization of assessment, thus, further requires that information be systematically collected, reviewed, and used to improve learning (Palomba & Banta, 1999) as a given part of an institution's culture. Suskie (2001) recognizes institutionalization as a "culture shift," and Barr (1998) echoes the notion of significant change, which involves a fundamental transformation of structures and processes related to assessment. Huba and Freed (2000) note two related but distinct dimensions involved in moving to an assessment culture at the organizational level. First is the importance of the mission, which emphasizes student learning as opposed to providing instruction, and second is the shift to functioning as a learning organization. The literature is rich with prerequisites for successful implementation: a clear and consistent vision with a defined process; faculty involvement and shared ownership; and long-term support, resources, and a recognition or reward system for those who embrace the new culture (Huba & Freed, 2000; Palomba & Banta, 1999; Suskie (Ed.), 2001; NAAL, 2003).

With regard to the first prerequisite, a clear vision toward assessment is an institution-level responsibility (Barr, 1998; Doherty et al., 2002; Sewall, 1996). This responsibility begins with, and must be driven by, the university mission (NAAL, 2003; Sewall, 1996, p. 331). Furthermore, there is an ongoing responsibility to keep the plan as visible as possible on campus so that all parties understand the process as it is occurring (Doherty et al., 2002; Tierney, 2001). The second prerequisite is faculty understanding, buy-in, and involvement (Huba and Freed, 2000; Palomba & Banta, 1999; Suskie, 2001). Two extremes demonstrate the importance of this prerequisite. In one instance, Palomba and Banta (1999) suggest that perhaps no single factor is more influential than faculty involvement in the successful implementation of an assessment culture, and that faculty resistance can be the greatest challenge. In addition, Davies and Hadden (2002) stress the need for the cooperation between administrators and faculty in developing an assessment culture. Long-term support and resources, the third prerequisite for successful implementation, is largely the responsibility of administrators. The support must be continuous, come in a variety of forms, and must meet the needs of the users (Pankake, 1998).

This study sought to achieve two major goals: first, to further the understanding of the institutionalization process, and second, to determine whether or not the processes were similar between two different colleges within the same institution.

Methodology

This qualitative study, at a research-intensive commuter institution in the mid-south, relied on focus groups and interviews with three constituent groups: college administrators, including deans and associate deans; program coordinators and department chairs; and, faculty members, some of whom were members of the college assessment committees and readers' panels (teams of faculty assigned to read the annual assessment reports for each academic department). The voices heard were those of first-hand implementers involved in the process of assessment, representing very different positions within the organization. Phenomenological exploration was the specific model of qualitative inquiry used by which the investigators attempted to "see things from the faculty and administrator's point of view" (Bogdan & Taylor, 1975, p. 14). Focus groups were conducted during both spring and fall semesters of 2003, with individual interviews held at approximately the same time of the academic year. At this time, processes to implement assessment had been underway 6-7 years.

Six focus group interviews were conducted with twenty-three faculty members (average group size ranged from two to seven participants). Five individual interviews were conducted with administrators (e.g., the deans and associate deans of the colleges). Each focus group included like constituents, e.g., faculty members, members of the assessment committees and the readers' panel, department chairs, and program coordinators. The focus groups and interviews lasted about one hour each and were conducted in conference rooms within each building assigned to the specific colleges, while the individual interviews were generally conducted in the administrators' offices. Due to circumstances, however, one interview was conducted via telephone.

Data Collection and Analysis

Data were collected over a one-month period during spring semester 2003 for the first college and over a two-week period during the fall semester 2003 for the second college. Sessions were audio-taped and the two facilitators each took notes. A five-question, open-ended response interview protocol, designed for this research, was used. The questions were developed to solicit the perceptions of faculty and administrators in the following areas: the implementation process; perceived supports or resources that served to facilitate implementation; perceived challenges or limitations that impeded implementing assessment practices; perceived measures to support the change process; and any additional comments the constituents wished to share.

Initially, all audio-taped sessions were transcribed. Tesche's eight-step qualitative process (Tesche, 1990) was used to ensure a systematic process for data analysis. Data were reviewed for topics, which were then clustered, and coded. The primary units of analysis were statements made by the various focus group and interview participants. A second review was then conducted to detect trends and identify themes.

Findings

As suggested by Suskie (2001), institutionalization requires a "culture shift." Furthermore, according to Barr (1998, p. 19), assessment "is no ordinary change. (lt) ... is a transformation [involving the altering of] fundamental structures and process." Based upon analysis of the data collected through focus groups and interviews, the two researchers found exactly this type of transformation in one of the colleges, but not in the other college. The researchers wondered how such a transformation was accomplished. The analysis began with the transformed culture.

Based upon ninety-three different statements by the study participants, it appeared that the major influence on implementing and sustaining assessment in the first college was primarily the involvement, vision, and deliberate leadership of the current dean of the college. This was exemplified by the following statement from one of the faculty members: "Right now I don't think there's anything that we're doing in assessment, that if tomorrow the Chancellor said, "we don't care about assessment anymore," it would just go away. It's become part of our organization ..."

A heightened awareness and attention to assessment actually began at a university-wide level in 1995-96, when a plan was developed and submitted to the North Central Association of Colleges and Schools. In order to implement the plan at the collegiate level, the former dean of the college "commissioned a team that had a representative from every department, and spearheaded efforts from each unit." Comments from the faculty indicated, however, that the administrative view of the previous dean was "fairly hands-off," although "he gave a charge every year at the state of the college reminding us of how important it was." The previous dean also worked with "seven or eight" of the faculty to "set up some deadlines, to read each others' reports and then ... to use that form that PAAG (Provost's Assessment Advisory Group) provided for giving each other feedback and providing a categorization rating."

Based upon seventy-five additional comments, the researchers noted a second important finding: that the first college had persevered and overcame, to a large extent, early resistance to the transition. Both administrators and faculty focus group members acknowledged early struggles. The resistance did not appear to come from people's unwillingness or inability to do assessment, or even a negative reaction to assessment per se. Rather, resistance flowed largely from faculty members' perceptions that university assessment had been superimposed on their existing accreditation requirements, and thus, there was a mismatch between assessment and their "real jobs." Administrators' comments indicated that they had anticipated such perceptions and "[had] done a number of things to try to make assessment be seen not as an add-on, but part of what we do as professionals." Even so, a member of one focus group felt that "... the people who are out there doing their jobs don't necessarily see the value of assessment. They think of it as just more paperwork they have to do."

Comments on resistance fell into three categories. For example, one faculty member described the resistance as the "push-pull of assessment." In addition, to the "push-pull" between accreditation and assessment, researchers noted perceived conflicts between qualitative versus quantitative needs, and between the superficial aspects of "hoop-jumping" and "add-ons" versus the "real jobs" of the loyal faculty. Assessment vied for the attention of faculty members who were already steeped in accreditation as a means to accountability. Many departments in the college already faced a stringent accreditation process and viewed assessment as less rigorous than what they were already doing to maintain compliance with accreditation standards. An assessment committee member noted that "... one of the frustrations I've had goes all the way back to the matching up of what the University needs from us, with what our accreditation wants." Another noted frustration with the report structure that required "... data defined differently from what our accreditation process is. [That's] really, really tough on some of our programs." Finally, when feeling forced to choose between accreditation and assessment, respondents said that, "Sometimes I feel that I could meet the University's [assessment] requirements and lose my accreditation probably. That would do me NO good! [It] would be devastating for us to get disaccredited and to have a great [assessment] report, which would have been not only an inefficient and ineffective, but counterproductive, allocation of resources."

In some cases, assessment was reported as seeming superficial, as seen in the terms "jumping through hoops," and "an add-on." In fact, the words 'jumping through hoops' came up very often during the focus groups: "That we were jumping through hoops to please a group of people that [knew] nothing about our program." Another group member noted the "... resistance to what was perceived as one more hoop to jump through and not integral to what they were there to do." Since assessment was not

recognized as integral to their work, some respondents also saw assessment as an "add-on" to their workload. An administrator noted that "... [faculty members saw assessment as] somehow divorced from what we do as teachers." An assessment committee member commented that "I think to begin with a lot of faculty thought, 'Well, I do assessment. I give grades. What more do I need to do? Why do I need to add something else on top of that?" However, an administrator clearly stated of assessment, "It's not an add-on. It's not an additional thing. It's not other duties as assigned. It's what we do." From the multiple comments made by both administrators and faculty members, it became clear to the researchers that resistance had been a force to be reckoned with during the process of implementing assessment. Equally clear was the perception that this college had evolved over time and assessment ultimately was seen as an integral part of the routine.

Findings in the second college were somewhat different from the aforementioned description. Researchers found the participant comments grouped into the following categories: 1) "mired in the mud" or "moving the stone," 2) event-versus-process or "the stone skips or sinks," and 3) sustaining-the-process or "there's an art to skipping a stone." One's perception of the institutionalization of assessment appeared to depend upon the "waterline perspective," or in other words, where one is in organizational relationship to the "water's edge"--those who were actually doing program assessment. Specifically, the faculty comments were predominantly in the first category, "mired in the mud" (41%), and the next largest number of comments appeared in the event-versus-process category (35%). By comparison, the administrators' comments were distributed only 22% and 16% in those two categories, respectively. By comparison, the administrators favored the third category of sustaining the process (62%) as compared with 42% for the faculty members.

The researchers found shared understanding among faculty and administrators on many aspects of institutionalization of assessment, as demonstrated by the same range of comments and the same three categories. The difference appeared to be seen in perspective, that is, one's position within an organization. Faculty members cited top-down expectations and initial confusion with assessment. They believed they were expected to serve too many masters and that the process was cumbersome at best. They felt overwhelmed by the time investment and resented the time assessment took away from teaching and serving students. In general, the faculty believed that they were still in a short-term rather than long-term mode regarding assessment. Faculty saw assessment as an "add-on" as opposed to an integral piece of their work. Administrators acknowledged the initial difficulties, but believed that assessment was an evolving phenomenon and that the college leadership had played a significant role in the implementation process. In particular, they cited the addition of a halt-time assessment coordinator for the college and their belief that the college was at the forefront of the campus with the addition of such a position. Administrators cited how much progress had been made, believing that the college was moving forward, as evidenced by the annual assessment reporting that was part of their calendar of activities.

Discussion

This study, designed to understand the process of institutionalizing assessment during the implementation phase, found that all participants articulated similar trends that either encouraged or blocked the implementation process. The important themes identified from both colleges were: 1) early implementation struggles, 2) refining the process, and 3) supporting and sustaining the process. However, the emphasis placed on any given theme varied considerably across groups. That is, administrators and faculty members placed markedly different emphasis on the importance of each of the three themes. Faculty members in the second college continued to emphasize the view that assessment was an "add-on" activity rather than part of their work, whereas administrators in both colleges, and faculty members in the first college, had moved to understanding assessment as "part" of their work.

The researchers recognize the limitations of this study. First, the findings are primarily based on research conducted at a single institution thus it is unknown how events specific to the timing of the focus groups and interviews might have shaped the data. The researchers also may have selected colleges and administrators that were predisposed to share the results of implementing assessment. Second, due to the limited scope of the study, the findings may not apply to other colleges in research-intensive commuter institutions in the mid-south. Nonetheless, these findings may have transferability to other related settings (Lincoln & Guba, 1985). The findings are consistent with the literature about implementing change in higher education organizations. The study extends what we know in other ways as well. For example, researchers believe that the reflections shared were influenced by position in the institution, level of personal awareness, and experiences at the university and with the assessment process.

Participants in the study were faculty members and administrators in two colleges nested within a research-intensive institution. Tenure of the faculty participants varied; some had been at the institution prior to the implementation of assessment on the campus and within the college, while others had only been involved since coming to the university within the past two or more years. Similarly, two of the administrators had joined the university prior to the mid 1990s, while the remaining administrators joined the university during the implementation process. Faculty members working in smaller programs (one or two person programs) seemed particularly affected by the expectations surrounding assessment. They remarked repeatedly that there was too little time to teach the many courses, advise the students, and report results annually for their assessment reporting.

The faculty in the first college appeared to utilize assessment as a focus on student learning, which, according to Palomba & Banta (1999) must be the outcome of collecting, reviewing and using assessment data. Further, the Dean used monies made available through the campus assessment budget to support faculty attendance at AAHE's National Conference on Assessment and to support an annual assessment day for the college which includes a nationally known speaker on assessment. These "rewards" were mentioned by faculty members in the first college when asked if there were indicators of on-going support, resources, and a recognition or reward system on the college (Huba & Freed, 2000; Palomba & Banta, 1999; Suskie (Ed), 2001). It appeared from the findings that the administrators in the second college failed to gain faculty "buy-in" or shared ownership, which is essential to successful change process (Huba & Freed, 2000; Palomba & Banta, 1999; Suskie (ed.), 2001; NAAL, 2003). Also, researchers noted that there seemed to be no recognition or sustained reward system for early change adopters in the second college| another key ingredient to successful change processes.

Conclusions

The major challenge for college administrators is to understand the culture of the college and to engage faculty members in ways that support and sustain change as progress is made in the process of institutionalizing assessment. Davies and Hadden (2002) stress the need for the cooperation between administrators and faculty in developing an assessment culture, and addressing the barriers to faculty involvement. Effective management frequently consists, in part, of gaining faculty buy-in. Rodrigues (2002) states that "[by] understanding the culture of your institution, a culture that both creates and reinforces faculty values, you can engage faculty members in ways that they will respect and respond to." Clear vision, faculty involvement and continued support to sustain the transition are key elements to successfully create a culture of assessment. Administrative accomplishments can serve as confirmation that following such a model can lead to a successful transition from perceiving assessment as an "add-on" to becoming "part of daily business."

Recommendations

Since the results were so divergent, one possible recommendation is for the first college to consider reviewing successful institutional change processes to inform future endeavors in the area of assessment. Administrators in the second college strongly felt the creation of a part-time assessment coordinator would assist faculty, particularly those in smaller programs, with further implementation of assessment practices. Therefore, it may be appropriate to consider re-interviewing faculty members to ascertain the effect of the assessment coordinator on the culture of assessment. Finally, further research in additional colleges at the institution is recommended for the near future, especially in light of the upcoming self-study process preceding the 10-year accreditation visit.

References

Banta, T. W. (1997). Moving assessment forward: Enabling conditions and stumbling blocks. New Directions for Higher Education, 25 (4), 79-91.

Barr, R. B. (1998, September-October). Obstacles to implementing the learning paradigm What it takes to overcome them. About Campus, 18-25.

Davies, T. G., & Hadden, C. (2002). From innovation to institutionalization: The role of administrative leadership in the assessment process. Community College Journal of Research and Practice, 23 (3), 243-260. EJ645737

Denzin, N. K. & Lincoln, Y.S. (Eds.). (1994). Handbook of qualitative research. San Francisco, CA: Jossey-Bass.

Doherty, A., Riordan, T., & Roth, J. (Eds.). (2002). Student learning: A central focus for institutions of higher education. Milwaukee, WI: Alverno College Institute.

DuToit, C. M. (1996). Transforming and managing the organizational culture of a university to meet the challenges of a changing environment. South African Journal of Higher Education, 10 (1), 96-104. EJ546111

Fullan, M. (1991). The new meaning of educational change (2nd ed.). New York, NY: Teachers College Press.

Gray, P. J. (1997). "Viewing assessment as an innovation: Leadership and the change process." New Directions for Higher Education, 25 (4), 5-15.

Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses: Shitting the focus from teaching to learning. Needham Heights, MA: Allyn and Bacon.

Lincoln, Y.S. & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. Miles, M.B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.

The National Academy for Academic Leadership. (2003). Factors for successful change. Retrieved September 2, 2003 from www.thenationalacademy.org/Ready.requisites.html Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Thousand Oaks, CA: Sage.

Palomba, C. A., & Banta, T. W. (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco, CA: Jossey-Bass: Pankake, A. M. (1998). Implementation: Making things happen. Larchmont, NY: Eye on Education.

Ratcliff, J. L. (1999). The rudder and the sail: Assessment for staff, program, and organizational development. Journal of Staff; Program & Organizational Development, 16 (4), 171-181.

Rodrigues, R. J. (2002, October 6). Want campus buy-in for your assessment efforts? AAHE Bulletin.com. Retrieved Sep. 2, 2003 from http://aahebulletin.com/member/articles/ 2002-10-feature02_pf.asp?pf=1

Sewall, A. M. (1996). Education, 116 (3), 325-332.

Suskie, L. (Ed.) (2001). Assessment to promote deep learning: Insight from AAHE's 1999 and 2000 assessment conference. Washington, DC: AAHE.

Tierney, W. G. (2001). Overcoming obstacles to reform. About Campus, 6 (2), 20-24.

SueAnn Strom, University of Arkansas at Little Rock, AR Linda J. Hemminger, University of Illinois at Urbana-Champaign, IL

Strom, Ph.D., is an Associate Professor of Higher Education in the Department of Educational Leadership at UALR, and Hemminger, Ph.D., is Director, Foreign Language Teacher Education, College of Liberal Arts and Sciences at UIUC.