Principles for more deliberate assessment.

Abstract

In an era in which the federal government is going to spend an unprecedented 53.1 billion dollars on public education, the level of accountability and pressure has increased for both students and teachers (U.S. Department of Education, 2003). The College of Education at North Carolina State University is developing a more deliberate process by which program assessment will be undertaken and reported annually. The process is supported by the use of specific technologies to assist in the design, collection, management, and reporting of assessment data. This paper reports on how design principles and technology tools were employed to support emerging progress for continuous program improvement.

Introduction

North Carolina State University's College of Education is dedicated to continuous program improvement and has developed a more deliberate process by which program assessment will be designed, implemented, and evaluated. The process is supported by the University's assessment-based Undergraduate Program Review (UAPR). The reporting procedure will be supported by the use of specific technology for the collection, storage, and reporting of assessment practices, findings, and program changes occurring as a result of the findings. The assessment process was initially implemented using the nine undergraduate academic programs in teacher education in the fall of 2002 and were extended to graduate programs in the fall of 2003. The purpose of this paper is to document that deliberate process and how it relates to two specific questions: (1) How can a more deliberate process of program assessment design and implementation be initiated? (2) How can the use of specific technology impact the manner in which assessment is conducted in a college of education?

Undergraduate Assessment-Based Program Review

University Context North Carolina State University (NCSU) is a research-extensive land-grant university located in Raleigh, North Carolina. The university has a present day population of over 29,000 students and over 1,600 faculty members. It is home to 10 colleges and offers bachelor's degrees in 92 fields, masters degrees in 101 fields, and doctoral degrees in 58 fields (NCSU, 2003). All undergraduate programs at NCSU participate in the UAPR process. The UAPR process includes submission of an assessment plan and portfolio for each program. The process allows each program to determine objectives, outcomes, and measures appropriate for that program and promotes program improvement by asking a series of questions: (1) What are we trying to do and why are we doing it? (2) What do we expect the student to know or do as a result of our program? (3) How well are we doing it? (4) What evidence do we have to support our findings? (5) How do we use the information to improve? (6) Do the improvements we make work? Four specific outcomes for programs are a direct result of the UAPR process: (1) increased knowledge about the reality of program quality; (2) evidence-based decision making; (3) greater accountability to stakeholders; (4) evidence for meeting requirements of accrediting institutions (NCSU Provost, 2002).

College Context The College of Education at NCSU is composed of four departments: Adult and Community College Education; Curriculum and Instruction; Educational Research and Leadership and Counselor Education; and Math, Science, and Technology Education (NCSU College of Education, 2003). The College has made a commitment to effective program assessment and to innovation in undergraduate teacher education and graduate programs. The College affirms this commitment in the following propositions: (1) Encourage dialogue on the content of assessment. (2) Recognize that assessment should be public. (3) Assess the cognitive and behavioral outcomes as well as personal and social dispositions. (4) Link assessment back to the conceptual framework. (5) Develop multiple and varied assessments. (6) Utilize technology to enhance the assessment process. (7) Distribute meaningful feedback on assessment results to all stakeholders. These seven propositions relate to the college mission, conceptual framework, and the National Council for the Accreditation of Teacher Education (NCATE) standards. NCATE assessment is conducted through an evaluation of the institutions' adherence to a specified set of six standards. Standards one and two evaluate Candidate Performance, and standards three through six evaluate Unit Capacity (NCATE, 2002).

For the purposes of assessment and program improvement, the focus is on standards one and two, Candidate Performance. These standards mandate that the College of Education provide evidence that teacher candidates are developing the knowledge, skills, and dispositions necessary to become effective practicing teachers. According to NCATE, standards one and two require the following: "systematic assessment, which encourages more attention to program design, increased faculty collaboration, and greater alignment within and between programs" and that "teacher education programs engage in self-assessment and continuous improvement" (NCATE, 2002).

Selected Review of Literature

Assessment is the means used to determine the progress of students toward reaching a program objective (Wilde & Sockey, 1995). At NCSU, assessment is a form of research that allows for systematic evaluation of the extent to which the program meets its objectives and outcomes (NCSU Provost, 2003). It has also been suggested by Palomba and Banta (1999) that assessment be purposeful and systematic with clear goals and objectives. The explicit emphasis of assessment for higher education is to provide a basis for accountability (Yorke, 1998). Institutions are accountable to all of their stakeholders who include, but are not limited to, accrediting organizations, students, the professional community, and taxpayers. A more implicit reason for conducting assessment is continuous program improvement (Reiff, 1982). Assessment is a means by which faculty can professionally and critically assess their programs in order to change in response to current conditions (Gray, 1997). Within the College of Education, the purpose of assessment is to determine how programs are contributing to the learning and development of teacher candidates. Programmatic assessment places the emphasis on programs rather than individual students. It allows the program to assess the overall experiences of the students involved and determine the cumulative effects of the program (Palomba & Banta, 1999). The units that exist within an institution of higher education are highly autonomous. This autonomy makes it more difficult to implement overarching modifications, thus the need for assessment at the program level (Gray, 1997).

It has been stated that "there is probably more bad practice and ignorance of significant issues in the area of assessment than in any other aspect of higher education" (Yorke, 1998). Consequently, this results in faculty resorting to the easiest option or most traditional methods with respect to assessment. A more systematic and deliberate approach can offer an effective and efficient means to conduct this process. A well-constructed system can help to ensure that what is expected to occur actually does occur (Yorke, 1998). In a time when all governmental institutions are expected to be fiscally conservative because of budget cuts and shortfalls, programmatic and systematic assessment of public colleges and universities is even more necessary. Direct costs to prepare for an accreditation visit can exceed $250,000, and the indirect costs in faculty and staff time can be twice as much (O'Neil, 1997). In the NCATE accreditation process, as many as 70 to 80% of the institutions may be required to submit a rejoinder, which requires additional time and cost. Programmatic assessment shifts the process from the institution as a whole to the programs that are most impacted and instrumental in making changes in response to assessment data. Systematic assessment allows for the development of a series of procedures by which programs can conduct continuous improvement, which minimizes the time and cost associated with the development of a new assessment plan with each cycle of accreditation.

Process: Essential Questions

The challenge is how to systematically assess a program's ability to document changes in teacher candidates' development of knowledge, skills, and dispositions. All teacher education programs should be in a continual state of examination and evaluation for the primary purpose of program improvement and secondary purpose of accreditation renewal. A deliberate program would have a set of principles in place that define, document, and report findings based on evidence collected throughout the year. Reports generated on as annual basis provide clear evidence on program quality and progress toward defined objectives and outcomes. The annual report is also a means by which programs can show changes that were made in response to the assessment procedure findings. The question then is to determine how a more deliberate process of program assessment design and implementation be initiated. Also, how can technology impact the manner in which assessment is conducted?

Developing a More Deliberate Assessment Process

Clarity of assessment mission The successful development of a more deliberate assessment process was contingent upon seven key factors. It was first necessary to begin with the development of a clear assessment mission. This was accomplished through cooperation between the UAPR and the College of Education. The assessment mission is based the following propositions: (1) Encouragement of dialogue on the content of assessment. (2) Recognition that assessment should be public. (3) Assessment of both the cognitive and behavioral outcomes as well as personal and social dispositions. (4) Linking assessment back to the college conceptual framework. (5) Development of multiple and varied assessments. (6) Utilization of technology to enhance the assessment process. (7) Distribution of meaningful feedback on assessment results to all stakeholders.

Building a common language The building of a common language included defining specific terms such as objective and outcomes. Objectives are broad, general statements reflecting what programs expect students to be able to do or know. They can be evaluated directly or indirectly. The related outcomes are more specific statements based on the objectives. Outcomes are observable and measurable and are used to evaluate specific behaviors or knowledge the program determines the student should know or be able to demonstrate (NCSU Provost, 2003). The number of objectives and the related outcomes needs to be limited. The programs in the College of Education tend to have roughly five objectives with related outcomes. By limiting the number of objectives, a program can prioritize those mensurable outcomes that are most important to the program, the College, and its students without overburdening the individuals collecting the assessment data.

Coaching in design of outcomes-based assessment plans The development of a more deliberate process for program assessment in the College of Education began in the fall of 2002. Initially, a dialogue was held between program coordinators and the assessment coordinator about the ways in which to better assess the knowledge, skills, and dispositions using specific objectives and outcomes. Program coordinators for each teacher education program then submitted assessment plans that included the following elements that were identified as important through collaborative discussions between the College of Education and UAPR: (1) Departmental Mission; (2) Program Overview; (3) Departmental Strategic Foci; (4) Program Objective with Related Outcomes; (5) Assessment Procedures ; (6) Timeline; (7) Supporting documents.

Coaching in Data Management In November 2002, program coordinators and a faculty representative were provided with training on the TracDAT System (http://www.nuventive.com/). The training included an overview of the system, instructions on its use, as well as examples on how the system can be used to support the assessment reporting practices. During the course of the workshop, program coordinators also had the opportunity to enter part or all of their assessment plans into the system. The TracDAT software provides assistance in organizing, storing, and reporting assessment data. In the College of Education, program coordinators enter assessment plans, objectives, outcomes, assessment methods, related evidence for the assessment practice, timelines, and decisions made based on findings. The plans are stored within the system, and the program coordinator has the opportunity to update or modify the plan as needed (Bresciani, 2003). Additionally, the TracDAT system will generate reports that include such information as objectives and outcomes with related institutional objectives. In the College of Education, reports may include specific objectives with related outcomes and the relation to specific NCATE standards (Bresciani, 2003).

Results and Implications

Within the College of Education this process began with nine undergraduate programs. In the fall of 2003, the graduate programs in the College will initiate a more deliberate process of assessment. Assessment plan templates have already been distributed, and program coordinators will have the opportunity to learn how to use the TracDAT system. By highlighting the successes of the undergraduate programs, it may ease use of the system as graduate programs begin to design assessment plans. The design and implementation of a more deliberate assessment process and the use of technology does create an environment where assessment becomes a more routine part of the program and fosters continuous improvement. As of March 2003, all nine undergraduate academic programs in teacher education have assessment plans stored in the TracDAT system. Program coordinators have developed a report on one objective with related outcomes which include evidence and program change in response to the data. These reports were submitted in May, and a final report was generated based on the conclusions from all nine programs.

Principles for Deliberate Assessment How can a more deliberate process of program assessment design and implementation be initiated? Ongoing assessment practices and procedures are necessary to sustain continuous program improvement. By developing a more deliberate process, we can assist program coordinators and faculty who are already overburdened by demands outside of the realm of instruction. Technologies such as the TracDAT system can streamline the process of documentation and generation of reports. This process is based upon a series of five principles: (1) University and College support for the assessment initiative; (2) The development of a clear assessment mission; (3) Building a common language; (4) The development of outcomes-based assessment plans; (5) Coaching and continuous support in capacity building the development of assessment plans and the use of data management technology. Each principle is integral to the success of the entire process. There must not only be support in principle from the University and College but also monetary support. Faculty must be involved in the development of the assessment mission. It is impossible to conduct effective assessment without involving those who develop and teach specific courses that are related to the assessment initiative. Building a common language aids in the development of a systematic procedure and helps to ensure efficient data collection. The development of outcomes-based assessment plans provides a framework by which to assess programs and thus provides evidence of improvement or the need for changes to all stakeholders.

Using Technology for Data Management How can the use of specific technology impact the manner in which assessment is conducted in a college of education? Program assessment can be enhanced by the use of specific technologies such as the TracDAT system. There are three primary advantages to data management software of this nature. The ease of use is the most important feature, since many individuals can learn to use the system in one or two training sessions. Once assessment plans are entered into the system, they are easily modified or updated. Web-based software such as the TracDAT system also offers the advantage of being accessed from a variety of locations or by different individuals who have been granted access. A third advantage of the software is the generation of a variety of standardized reports based on information entered into the system. There are two disadvantages to this system, the first is the inability to statistically analyze data. The second is the data entry component of the TracDAT system is labor intensive and is the primary drawback for program coordinators. This can be alleviated by having one individual who was familiar with the system enter the data for each program.

Conclusion

In order to sustain continuous program improvement, colleges need to develop a deliberate program with a set of principles in place that define, document, and report findings based on evidence collected throughout the year. The College of Education at NCSU has implemented such a system. Systematic assessment in the College of Education is successful because of the principles supporting the system: University and College support for the process; collaboration of the administration and faculty on the development of the mission; use of a common language; development of outcomes-based assessment plans; and continual coaching and support for assessment plan development and data management.

References

Bresciani, M. (2003). Undergraduate Academic Program Review Frequently Asked Questions: What is TracDAT? Retrieved 6/10, 2003, from http://www.ncsu.edu/provost/academic_programs/uapr /FAQ/UAPRFAQwhatisTracDAT.html

Gray, P. J. (1997). Viewing assessment as an innovation: leadership and the change process. New Directions for Higher Education, 100, 5-15.

National Council for the Accreditation of Teacher Education. (2002). Professional Standards for the Accreditation of Schools, Colleges, and Departments of Education. Washington, DC: NCATE

North Carolina State University. (n.d.). About NC State University. Retrieved 6/12, 2003, from http://www.ncsu.edu/aboutncstate.html

North Carolina State University College of Education. (June 10, 2003). College of Education: Connecting To The Future. Retrieved 6/11, 2003, from http://www2.ncsu.edu/ncsu/cep/

North Carolina State University Provost. (2003). Undergraduate Academic Program Review: Common Language for Assessment. Retrieved 6/10, 2003, from http://www.ncsu.edu/provost/academic_programs/uapr/process/language.html

O'Neil, E. (1997, June). Using accreditation for your purposes. AAHEBulletin.com. Retrieved April 9, 2003 from http://www.aahebulletin.com/public/archive/usingacred.asp

Palomba, C. A. & Banta, T. W. (1999). Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. San Francisco: Jossey-Bass.

Reiff, J.C. (1982). A model for assessing teacher preparation programs. Psychology, A Quarterly Journal of Human Behavior, 19(1), 31-35.

TracDAT. (n.d.). Assessment management software [computer software and manual]. Pittsburgh, PA: Nuventive. Retrieved from http://www.ncsu.edu/undergrad_affairs/assessment /files/resources/TracDAT/TracDAT.htm

United States Department of Education. (n.d.). No Child Left Behind. Retrieved 3/21, 2003, from http://www.nochildleftbehind.gov/next/states/

Wilde, J. & Sockey, S. (1995). Evaluation handbook. Albuquerque, NM: Evaluation Assistance Center-Western Region, NM Highlands University. Retrieved 6/10, 2003, from http://www.ncela.gwu.edu/miscpubs/eacwest/evalhbk.htm

Yorke, M. (1998). The management of assessment in higher education. Assessment & Evaluation in Higher Education, 23(2), 1.

Michael J. Maher, North Carolina State University Alan J. Reiman, North Carolina State University

Maher is a doctoral candidate in the Department of Curriculum and Instruction. Dr. Reiman is an associate professor in the Department of Curriculum and Instruction and coordinator of assessment for the College of Education.