Collaborative Action Research to Assess Student Learning and Effect Change.
Introduction and Background Information
In the next decade the nation will need over 2.2 million teachers (Riley, 1999). In general, colleges and universities around the nation have been called upon, now more so than in the past, to demonstrate their commitment and ability to meet the changing needs of American society (Goodlad, 1999; Schmidt, 2000). With regard to the need for teachers, higher-education institutions have been asked not only to meet this need, but also to solve the problem of preparing "quality" teachers for America in the 21st century (Riley, 1999). Thus, teacher preparation has become a priority item on the agendas of many colleges and universities, and it is an important presidential campaign issue.
At present, the U.S. Department of Education, through the 1998 Congressional Higher Education Act, Title II, Section 207, is requiring states and higher-education institutions to assess and publicize the effectiveness of their teacher education programs to prepare quality teachers. Section 207 of the law includes new accountability measures that require states and their colleges and universities to announce annually the percentage of students who have passed state teacher-certification exams, and to report on other quality indicators and licensure requirements as well. In essence, for higher education there will be a national and state report card (see U.S. Department of Education, 2000a, 2000b). Subsequently, if a college or university does not achieve a designated passage-rate set by the state in which the college or university resides, then the state's education department or its board of regents may close that institution's teacher education program or subject it to deregistration.
For example, the New York State Board of Regents has set an 80% student passage-rate for its higher-education institutions. That is, if less than 80 percent of the institution's teacher education students pass one or more teacher certification examinations, the institution's teacher education program is subject to deregistration. And if a program, found deficient, does not demonstrate significant annual improvement toward the 80% standard, it, too, is subject to deregistration (see New York State Regents Task Force On Teaching, 1998, pp. 24-25). For small colleges and universities this could be financially devastating; and for larger institutions it may mean that they will no longer be able to use their teacher education programs as the institution's "cash cow."
Further, the U.S. Department of Education and state education departments around the nation are requiring colleges and universities to demonstrate their "institutional" commitment to prepare quality teachers and their capacity to do so by mandating that the institution engage in a comprehensive assessment and/or audit conducted by a designated accrediting association or agency. What we are currently witnessing is not only an assessment of programs in teacher education and the unit of the college that sponsors the program (i.e., schools and departments of education), but also an assessment of the entire institution in terms of its capacity to prepare "quality" teachers. (See the National Council for Accreditation of Teacher Education [NCATE], 2000, and Teacher Education Accreditation Council [TEAC], 1999, for the standards, rubrics and criteria used to assess institutions.)
As a result of these events, college and university presidents are becoming more aware of the issues, problems, and concerns related to accreditation (i.e., involving the liberal arts and science faculties in the preparation of teachers). Also, they are becoming aware of the financial consequences of not securing accreditation for their institution. It is interesting to note that in New York State the education community has already predicted that 20% of the State's teacher education programs will probably close or be deregistered by year 2004 because of the stringent mandates. For some small colleges in the State, this could be financially devastating. It is clear to many educators in New York that in the next few years, in order to survive well, each institution will need to provide evidence of its ability to prepare competent teacher candidates as measured by a number of macro (institutional) and micro (classroom) criteria. Student learning outcomes and student performances will serve as the more important criterion measures. The scenario for colleges and universities in New York State is the same for higher education in other states as well.
It is in the effort to meet these challenges that collaborative action research can play a critical role. There are three major concepts we would like to promote and put forth: one, that school-college partnerships are critical to the process of preparing "quality" teachers in the 21st century; two, that collaborative action research engaged in by school and college personnel is an integral part of the process for instituting change and improvement in education and for demonstrating programmatic-institutional effectiveness in the preparation of teachers; and three, that data/findings of action research studies that focus on analyses and assessment of student performances can be used (1) to establish baseline data for the program, (2) as indicators of student learning, and (3) as evidence of the students' professional competence and/or achievement of teaching-learning standards.
For purposes of this article, we will focus on action research video studies that were conducted by members of a school-college partnership project and the use of the data to analyze and assess the teaching performances of teacher education students. Our intent is not to describe the studies in great depth, but rather to highlight one or two of the findings and explain their function in assessing student performances and effecting programmatic-institutional change and improvement. In the following sections we will present first, a synopsis of the school-college partnership project; second, a summary of the action research video studies engaged in by members of the partnership; and third, an explanation of selected data and findings, and their use in the assessment and change process.
The Partnership and Its Partners
In June of 1998 Dowling College, a private institution on Long Island in New York State, and the North Babylon School District entered into a partnership entitled Project SCOPE II, (School-College Operation in Partnership Education). Project SCOPE II targets the three separate education domains of the school's curriculum and staff development program, and the college's preservice and inservice teacher education programs, and then integrates them to effect change and improvement on both the school and college levels. The overall goal of the project is to change and improve K-12 schooling and teacher education (13-18) in a coordinated and integrative fashion. Originating as Project SCOPE in 1980 at the City University of New York/Queens College, the project at the time was one of a few in the nation to have taken a comprehensive and holistic approach to changing and improving school curriculum and professional education. Although small in terms of the number of participating members, it was and is unique in that members of the project act on the belief that each educational domain needs to be intertwined with the others in a meaningful way in order for substantial change to occur in education. (See Catelli, 1992, 1995 and 1997 for an in-depth explanation of the partnership and its accomplishments.)
In June of 1998, when Project SCOPE was expanded and adapted to other academic settings at Dowling College, under the title of Project SCOPE II, it retained its integrative, holistic model-approach to educational change. Project SCOPE II is a partnership program between faculty, teacher education students, administrators, and school teachers representing four elementary schools of a school district on Long Island in New York. As was the case in the original project, collaborative action research/inquiry is an integral part and a critical component of Project SCOPE II's partnership program at Dowling College. It is a powerful tool for instituting change and improvement in an integrative fashion on both education levels (see Catelli, 1995 and Sirotnik, 1988 for a further explanation of the role action research-inquiry plays within a partnership program).
Summary of the Action Research Video Studies
The action research video studies that were conducted during the 1998-2000 academic years were part of Phase One of the partnership's overall strategic plan to effect change and improvement in K-18 student learning (Catelli, Padovano, Costello, & Diver-Stamnes, 1999). The studies were participated in and conducted by the partnership's graduate education students (Dowling College), school teachers who served as cooperating teachers in the partnership project (the North Babylon School District), student teachers (Dowling's teacher candidates for initial teacher certification), and the professor-director of the project. All parties volunteered to become members of Project SCOPE II and subsequently played a role in either the design of the action studies and/or the implementation of Phase One of the partnership. The primary focus of the action research video studies is on the student teachers' classroom teaching performances and the behaviors of their pupils. The overall purpose of the studies is to establish baseline data for changing and improving the preservice elementary teacher education program as well as provide a descriptive analysis and an assessment of the teacher candidates' teaching performances.
Action research was the method employed in conducting the action studies. Traditionally known for its systematic and cyclical method of operation, (i.e., identification of a problem, data collection, data analysis, implementation and evaluation), "action research" is fundamentally employed as a methodological strategy for improving educational practice, and for the production of relevant and practical knowledge in education. Within the context of a school-college partnership, action research may be more appropriately labeled as collaborative inquiry, denoting a process of self-study involving the members of the partnership in a type of analysis and evaluation that is critical, reflective, and focused directly on the work of the partnering members in relation to the partnership's goals. The core, then, of the research that is conducted in a partnership program, and in particular in SCOPE's holistic partnerships, resides in the fact that it is collaborative, participatory, and critical; and that it is directed at changing and improving education on both levels in an integrative way.
Nine action video studies were conducted by thirty-five of SCOPE II's action researchers arranged in eight research teams. The eight teams identified and analyzed videotaped classroom instructional behaviors (i.e., asking questions) of twenty-nine elementary-education student teachers (teacher candidates). Each research team was responsible for analyzing four to five student teacher videotaped lessons that were taught to classes of no more than thirty-five elementary school pupils. All twenty-nine student teachers were members of Project SCOPE II's partnership program during the 1998-1999 academic year. Each student teacher was videotaped during the second month of his/her student teaching experience, after one formal observation by a college supervisor had taken place.
Using multiple observational systems and adaptations of systems, the eight teams of action researchers during the 1999-2000 academic year coded the videotaped classroom behaviors for five second intervals and analyzed teacher-pupil performances. Frequencies and percentages of time the student teachers and their pupils spent exhibiting classroom instructional behaviors were computed. The resulting percent occurrences of the behaviors indicate the amount of time the student teachers and their pupils engaged in specific instructional actions that facilitate pupil learning and that reflect teaching and learning standards. Assessment criteria and rubrics that are related to state and national standards were then applied to the classroom video performances in order to discern either levels of the teacher candidates' performance or the presence-absence of an instructional behavior. Thus, the analyses of the teacher candidates' teaching performances were conducted in an effort to:
1. identify emerging classroom teaching behaviors, instructional actions, and patterns;
2. identify classroom pupil behavior;
3. establish baseline data for changing and improving the preservice elementary teacher education program; and
4. assess the teacher candidates' instructional performances in regard to criteria extracted and adapted from four sources:
a) New York State's standards for teachers and student (pupil) learning (New York State Regents Task Force on Teaching, July 1998; New York State's brochure of 27 Student Learning Standards, 1996);
b) standards set forth by the National Board For Professional Teaching Standards (1998);
c) criterion measures identified by Educational Testing Services' PATHWISE program (1995);
d) the pedagogical preferences of a representative of the partnering school district.
The ultimate goal of the work is to institute the type of change that brings about student learning on both the school (K-12) and college (teacher education) levels, and demonstrate programmatic and institutional effectiveness in preparing competent teacher candidates. The series of studies exemplify SCOPE II's integrative-holistic approach to changing and improving preservice and inservice teacher education and K-12 student learning in a coordinated fashion.
To achieve the overall purpose of the action video studies, a number of research goals were formulated. Two of the research goals are listed below:
Goal 1: To determine the percentage of class time devoted to "substantive instructional teacher-pupil behaviors," (i.e., asking questions, giving corrective feedback), and the percentage of time devoted to "non-substantive behaviors," (i.e., managerial, organizational, and disciplinary behaviors).
Goal 2: To determine the number of pupils cognitively and/or physically "on task" during a class period. Figure 1 identifies the substantive and non-substantive teacher-pupil behaviors that were coded and analyzed (see Figure 1).
Figure 1. Substantive and Non-Substantive Grouping of Instructional Teacher-Pupil Behaviors and Categories
Substantive Instructional Non-Substantive Behaviors/Actions- Instructional Behaviors/Actions (Managerial/ Organizational/ Disciplinary)- Teacher-Pupil Behavior/ Teacher-Pupil Categories: Behavior/Categories: Teacher -- 1. Accepts Feelings/Sets Climate -- 6a. Gives Directions-- Managerial/ 2a. Praises or Encourages Organizational Functions 3. Accepts, Uses or Extends Ideas of Pupils(s) -- 5. Lectures or Presents Information to 7. Teacher Establishes 5a. The Whole Group -- or Enforces Codes 5b. A Small Group within the of Behavior Large Group -- 5c. An Individual 10a. Silence and 5d. Articulates the goal -- Confusion objective rubric-standard or expectation for the lesson -- 6. Gives Directions--Subject Matter (Content/Skill)-- 11. Teacher Ilustrates/Demonstrates and Talks -- 12. Teacher Talks and Pupil Demonstrates/Illustrates -- 10b. Silence-Teacher Observes Pupils(s) -- 2b. Gives Corrective Feedback -- 4. Teacher Questions -- 4a. Asks Pupil(s) Lower-Order Questions -- 4b. Asks Pupil(s) Higher-Order Questions -- Pupil Behavior- 8. Pupil Talk-Response -- 9. Pupil Talk-Initiation -- 13. Other: Specify --
Selected Findings and Their Use in the Assessment and Change Process
Basically, for Goals 1 and 2, the preliminary data and findings reveal that twenty-six (or 90%) of the twenty-nine student teachers engaged in substantive instructional behaviors for at least three-fourths (75%) or more of their class time or lesson. Seventy-five percent (75%) was set as the minimum "time" criterion. Lessons ranged from twenty minutes to one and a half hours. It should be mentioned that those lessons that exceeded one hour were coded only up to the first hour of the lesson. With regard to Goal 2, in all twenty-nine videotaped lessons, twenty-one or 72% of the student teachers had at least 85% of their pupils "on-task" for the lesson. Thus, a high percentage (90%) of the student teachers conducted lessons in which 75% or more of their class time was spent in substantive instructional behaviors--while a relatively low percentage of student teachers (10%) spent 25% (or higher) of their class time in management and discipline actions.
Upon closer examination of the data we found that the three student teachers who had below 75% of the time devoted to substantive instructional behaviors either had very young pupils (i.e., kindergarten age), or they were conducting an initial lesson using the method of "cooperative learning" which required more direction and pupil management. There were, however, cases where better management techniques could have been employed. The overall intent of Goal 1 is to have the partnership members, i.e., the professors, supervisors, and cooperating teachers, prepare preservice teachers to conduct lessons where management time is kept to a minimum, and time devoted to substantive instructional matters and behaviors is maximized--thereby allowing for the possibility of increased pupil learning.
Both Goals 1 and 2 relate to or were adapted from the following sources: (a) item 3 of the New York State Teacher Standards which states that "The teacher effectively manages classrooms that are structured in a variety of ways, using a variety of instruction methods.... "(New York State Board of Regents Task Force On Teaching, 1998, p.14); (b) criterion C5 of Educational Testing Services' PATHWISE program which assesses the student-teachers' classroom performances on "Using instructional time effectively." (Educational Testing Service [ETS], 1995, p.40); (c) a review and an analysis of the 27 New York State Student Learning Standards; and (d) a review of the NCATE's Standard 1, the Candidate's Performance ... the Candidate's Knowledge, Competence, etc. (NCATE, May, 1999, pp. 3-9).
The data and videotapes were analyzed further to reveal specific teaching behaviors that facilitate pupil learning and those instructional actions identified as pedagogical preferences by a cooperating teacher of the partnering school district. The instructional behaviors that were cited by the teacher include the following behavioral categories seen in Figure 1:  Accepts pupil(s) feelings and/or Sets the climate and tone of class; [4a-4b] Asks lower and/or higher-order questions;  Accepts, uses or extends ideas of pupils;  Gives corrective feedback to pupils; [5d] Articulates the goal-objective-rubric or expectation for the lesson; and  Pupil(s) responds to questions. Our findings reveal that for these instructional classroom behaviors, the student teachers, as a group, had low frequencies or percent occurrences of time engaged in each of the behaviors. These classroom behaviors are pedagogical areas or instructional actions that facilitate pupil learning. They are the areas of competence that should be targeted for improvement in the teacher education program.
In addition to conducting descriptive analyses, all twenty-nine student-teacher videotaped performances were scored according to selected assessment criteria and rubrics adapted from the PATHWISE Program (ETS, 1995) and the National Board For Professional Teaching Standards (1998). The analyses of all that data have yet to be completed. However, with regard to learning outcomes or areas of teaching competence, the criteria set for assessing the videotaped performances, at an initial level, revolved around the following competency statements for student teachers:
* To conduct lessons for which at least 75% of the class time is devoted to substantive instructional teacher-pupil behaviors/actions (i.e., asking higher-order questions), and no more than 25% of class time is devoted to non-substantive behaviors/actions (i.e., managerial, organizational, and disciplinary behaviors).
* To conduct lessons that have at least 85% of the pupils cognitively and/or physically "on task" during the lesson.
* To demonstrate selected instructional behaviors/actions that facilitate pupil learning and achievement (i.e., asking higher-order questions, giving corrective feedback).
* To demonstrate acceptable levels of classroom performances with regard to selected teaching-learning rubrics and/or behavioral criteria that are linked to and aligned with national and state teaching-learning standards.
We envision that the resulting analyses and continuous assessment of videotaped performances of future student teachers of the program will be used as indicators and evidence of student-teacher learning.
To summarize, the preliminary findings cited previously as well as additional data and findings of the nine action research studies will be used
* to establish baseline data for future studies that focus on effecting changes and improvements in the preservice teacher education program;
* as indicators of student-teacher learning;
* as evidence of professional competence and/or achievement of national, state, and local teaching-learning standards;
* to institute changes in courses and field modules of the program that target those pedagogical areas where we find that the student teachers (teacher candidates) need to develop higher levels of skill or at least have more experience prior to student teaching (i.e., asking higher order questions; giving corrective feedback to pupils, etc.); and
* to design content for inservice training sessions to assist the partnership's cooperating-supervising teachers in coaching their student teachers in the instructional behaviors that relate to national, state, and local standards of teaching and learning.
Finally, it is our hope that the descriptive data and findings of the nine action research video studies conducted during the 1998-2000 academic years of the partnership will serve to initiate change and improvement in the teacher education program, and serve as baseline data for subsequent institutional assessment studies for purposes of accreditation. Further, and on a larger scale, we envision that the work presented will serve as a template for higher education in its effort to prepare quality teachers in the 21st century and provide evidence of its capacity to do so. Also, it is our hope that collaborative action research as portrayed in this article will contribute to the knowledge-base on assessment of student learning, and on research associated with effecting educational change and improvement on both the college and school levels.
Catelli, L. A. (1992). Against all odds: A holistic urban school/college partnership--Project SCOPE. Action In Teacher Education, 14 (14), 42-51.
Catelli, L. A. (1995). Action research and collaborative inquiry in a school-university partnership. Action In Teacher Education, 16 (4), 25-38.
Catelli, L. A. (1997). An holistic perspective on school-university partnerships in the twenty-first-century-theory into practice. In D. Lambert and A. Hudson (Editors), Exploring futures in teacher education: Changing key for changing times (pp. 228-246). London, UK: Institute of Education, University of London.
Catelli, L. A., & Nix, W. (1992). Project SCOPE: An urban partnership venture-descriptions and predictions. In G. Graham and M. Jones (Editors), Collaboration between researchers and practitioners: An international dialogue - Proceedings from the 1991 AIESEP-NAPEHE World Congress (pp. 105-110). Atlanta, GA: AIESEP/NAPEHE.
Catelli, L. A., Padovano, K., Costello, J., & Diver-Stamnes, A. Strategic models of school-college partnerships to effect change and improvement in student learning and teacher education: A comparison of two approaches. Conducted a symposium and presented a paper at the 1999 International Congress for School Effectiveness and Improvement (ICSEI). San Antonio, Texas, January 4, 1999.
Educational Testing Service. (1995). PATHWISE: Orientation guide. Princeton, NJ: Author.
Goodlad, J. (2000). Rediscovering teacher education: School renewal and educating educators. Change, 31(5), 29-33.
National Board For Professional Teaching Standards. (1998). Middle childhood/Generalist standards. Southfield, MI: Author.
National Council for Accreditation of Teacher Education. (May, 1999). Institutional orientation: State of New York (Tab II, NCATE 2000 standards revision, March 22, 1999). Washington, DC: Author.
New York State Regents Task Force On Teaching. (July, 1998). Teaching to higher standards: New York's commitment. Albany, NY: The University of the State of New York - The State Education Department.
New York State Department. (1995). New York State curriculum, instruction and assessment frameworks. Albany, NY: Author.
Riley, R. (February, 1999). Riley calls for fundamental changes to improve American Education; Announces steps to address accountability and teacher quality: The sixth annual state of American education speech. Government Speeches. Webcast of the Secretary's Speech. [http://www.ed.gov/Speeches/9990216-a.html].
Schmidt, P. (June 30, 2000). States set a course for higher-education systems--master plans aim to ensure cohesive response to technological and economic changes. The Chronicle of Higher Education, Section: Government and Politics, vol. XLVI, no. 43, pp. A27-28.
Sirotnik, K. (1988). The meaning and conduct of inquiry in school-university partnerships. In Kenneth A. Sirotnik and John Goodlad (Eds.), School-university partnerships in action: concepts, cases, and concerns (pp. 169-90). NY: Teachers College Press.
Teacher Education Accreditation Council. (1999). TEAC: Prospectus. Washington, DC: Author - One Dupont Circle Suite 320, Washington, DC: Author - Department of Education, Office of Post Secondary Education, 1990 K Street, N.W., Washington, DC 20006.
U.S. Department of Education. (2000a). Guide for preparing state and institutional reports on the quality of teacher preparation. Washington, DC: Author - Department of Education, Office of Post Secondary Education, 1990 K Street, N.W., Washington, DC 20006.
U.S. Department of Education. (2000b). Initial report of the secretary on the quality of teacher preparation. Washington, DC: Author - Department of Education, Office of Post Secondary Education, 1990 K Street, N.W., Washington, DC 20006.
Linda A. Catelli, Dowling College Joan Carlino, Belmont Ave. Elementary School, North Babylon, New York
Catelli is a Professor of Education and Director of Project SCOPE II in the School of Education where she teaches graduate courses <Carlin@aol.com>. Carlino teaches second grade and has recently received a master's degree in education from Dowling College.