Teaching performance assessment: a comparative study of implementation and impact amongst California State University campuses.
Concerns about teacher competency have existed since the beginnings of public education in the nineteenth century (Arends, 2006). While research clearly indicates that teacher quality is a critical factor impacting student achievement in schools, how best to define and measure teacher quality is still widely debated. Amid the increasing calls for teacher accountability have come challenges to teacher preparation programs to demonstrate their effectiveness, resulting in a variety of reforms to create more rigorous preparation, certification, and licensing. (Darling-Hammond, Pacheco, Michelli, LePage, Hammerness, & Youngs, 2005; Goldhaber, 2002; Pecheone & Chung, 2006).
In California, a substantial transformation of teacher preparation was launched by the 1998 passage of Senate Bill 2042, which created new standards for subject matter preparation, professional preparation and induction, and which included the requirement for teacher preparation programs to institute "systematic and psychometrically sound teaching assessments ... to confirm the quality of the teaching workforce and increase public confidence in teachers, teaching and teacher preparation" (Sandy, 2006, p. 17).
Purpose of the Study
This article is based on the perceptions of California State University administrators and provides a comparative study of the challenges and benefits resulting from the implementation of the teaching performance assessment requirement of SB 2042 standards 19-21 on the California State University (CSU) campuses. With 23 campuses and almost 450,000 students, the CSU system is the largest and most diverse university system in the country, and it produces 55% of California's teacher education graduates each year.
Assessing Pre-Service Teachers
Although examinations for pre-service teachers are not new, traditional teacher licensure tests, which typically focus on basic skills, subject matter knowledge and pedagogical knowledge, have been criticized for oversimplification of teaching; failure to measure candidates' ability to synthesize subject matter, students, and context to make decisions; weakness regarding predictive validity; and adverse impact on minorities (Porter, Youngs & Odden, 2001; Goldhaber, 2002; Pecheone & Chung, 2006). A further concern as program administrators begin to connect teacher quality at the pre-service level to student achievement is reflected in standard 1 of the National Council for Accreditation of Teacher Education (NCATE), which requires that candidates demonstrate the ability to assess student learning and use the results to inform instruction (Mitchell, Allen & Ehrenburg, 2006).
The concerns about forced response tests and the need to address the increasing emphasis on external accountability have led to the development of standards-based performance assessments in teacher education (Arends, 2006). Darling-Hammond and Snyder (2000) suggest that there are positive side effects to the use of authentic performance assessments that incorporate the teaching context. Among these effects is their potential for influencing teachers' and their students' learning. In examining pre-service teachers' experiences with the Performance Assessment for California Teachers, for example, Chung (2008) indicated that in addition to expanding their teaching knowledge and skills, the candidates' teaching was impacted by "(1) a shift from concern with teacher activities, activity structures, or student engagement to a greater concern with student learning; and (2) increased awareness of the need for strategies to reach English learners" (p. 17). Other positive side effects of authentic performance assessments suggested by Darling-Hammond and Snyder (2000) include their use for program improvement and their potential for increasing collaboration and communication among those involved in the preparation of teachers, both in the higher education institution and in the P-12 schools. California's requirement that each candidate pass a Teaching Performance Assessment is consistent with recent trends suggesting that performance assessments that are contextualized in the field more adequately judge beginning teacher pedagogical competence (Pecheone & Chung, 2006).
A major consideration in the implementation of performance assessments, however, is their cost. Porter, Youngs, and Odden (2001) reviewed a variety of performance assessments developed and adopted during the late 1990s and concluded that the implementation of such assessments is an expensive proposition. Main factors in the cost of implementation include the development or purchase of an assessment system; training assessors; administering the assessment; and scoring the assessments. As examples, they stated that the high cost of implementing Praxis III resulted in the development of a modified version, Pathwise. In addition, they cited the 1997 implementation of BEST in Connecticut, which cost the state $3.6 million. Indeed, the implementation of the California teaching performance assessment was initially put on hold for three years pending the provision of adequate funding resources. Although state-funding support has still not materialized, the decision was made to go forward with mandated implementation in 2008-09. While the performance assessment is generally seen as a rigorous measure demonstrating a high level of teacher preparation for licensure, as Sandy (2006) warns, "the absence of a firm commitment from the state to support and sustain this work will undermine its effectiveness and limit its impact over time" (p. 17).
Overview of the California Teaching Performance Assessment
Senate Bill 2042 required all teacher candidates to pass an assessment of teaching performance in order to earn a preliminary Multiple Subject (elementary) or Single Subject (secondary) Teaching Credential. This assessment of teaching performance is designed to measure the candidate's knowledge and skills in relation to California's Teaching Performance Expectations (TPEs). The Teaching Performance Assessment (TPA) model of the California Commission on Teacher Credentialing (CCTC) is the CalTPA, which was developed in collaboration with professional educators and the Educational Testing Service (ETS). While state law requires that teacher preparation programs implement an approved teaching performance assessment, program sponsors have been allowed to develop alternative TPA models for approval by the Commission. Two additional models have received approval: the Performance Assessment for California Teachers (PACT), which was developed by Stanford University and a consortium of public and private universities, and Fresno Assessment of Student Teachers (FAST), which was developed by California State University, Fresno, and is approved solely for its use (CTC, 2008).
While differences exist between these models, each is a comprehensive assessment comprised of multiple tasks that require teacher candidates to demonstrate the proficiencies delineated in the TPEs in a classroom setting. These proficiencies include understanding students and the school context; effective planning of instruction and assessment; adaptation of instruction and assessment for students with special needs, including English learners; use of assessments to monitor student learning and inform instruction; and reflection on professional practice. CalTPA and PACT also require a video of the candidate working with students in the classroom. All three of these models are in use by campuses in the CSU system. Of the 22 CSU campuses with teacher preparation programs, twelve use CalTPA, nine use PACT, and one uses FAST.
Standards 19-21 of the Multiple and Single Subject Program Standards govern the administration of the teaching performance assessment. These standards require that each TPA be administered for validity, accuracy, and fairness. This includes ensuring "strong consistency of scoring among assessors" and monitoring of scoring and results to "ensure equitable treatment of candidates." In addition, each TPA must be evaluated by qualified educators who have been selected according to established criteria and undergone structured training, calibration, and re-calibration processes. Lastly, the standards require programs to annually commit "sufficient resources, expertise, and effort to its planning, coordination and implementation." Each campus is responsible for documenting the administration, scoring, and reporting of the assessment in accordance with state accreditation procedures.
Following assessment, candidates receive performance information that is clear and detailed enough to (a) serve as a useful basis for their Individual Induction Plans developed within an approved Induction Program, or (b) guide them in study and practice as they prepare for reassessment as needed. Individual results of the assessment are used as one basis for recommending candidates for Preliminary Teaching Credentials, and aggregated assessment results are used in appropriate ways to improve programs.
Methodology of the Study
The deans and associate deans of education from 22 CSU campuses across the state meet quarterly to conduct system-wide business, collaborate on important initiatives, and forge a strategic direction for colleges and schools of education throughout the system. A main topic of the summer 2008 meeting was the implementation of the California Teaching Performance Assessment (TPA). It became clear during discussions of this topic that each of the 22 campuses had different understandings of the protocol for implementation; plans for application of the standards; and challenges in funding, support, collaboration, and achievement of goals. Both among and within the three similar models there were variations in implementation plans. The concerns resulting from this disparity in responses to the implementation mandate resulted in the convening of a task force to develop a comparative study of campus practices related to the TPA. Representatives from four southern California and four northern California campuses agreed to meet together to carry out this study. Task force members represented expertise in each of the three TPA models.
In fall 2008, the task force met via conference calls to develop and refine items for a twenty-two-question survey. The survey was sent to deans of education at the 22 campuses with teacher preparation programs. CalStateTEACH, a CSU statewide, fully online preparation program for elementary sachool teachers was not included in this survey. The campuses had two weeks to complete the survey and return it for comparison. The task force requested that individuals most knowledgeable about the TPA on each campus should write the responses. Associate deans completed most surveys, with others completed by either the dean, the TPA coordinator, or the assessment coordinator. Of the 22 campuses, 19 returned the survey. The surveys were completed electronically and the responses aggregated for each item. The task force was then provided a Web link to view the results, and a sub-committee of the task force analyzed the data and summarized the findings. A report on the findings was presented at the fall 2008 meeting of the CSU deans and associate deans.
The survey questions fell within nine clusters: (1) number of students and tasks by TPA type; (2) staffing; (3) training and scoring; (4) technology support; (5) remediation/resubmission; (6) costs and funding; (7) collaboration; (8) lessons learned; and (9) next steps. The list of survey questions is presented in Table 1. The campuses had two weeks to complete the survey and return it for comparison.
It is important to keep in mind that the questions were of a speculative nature. Although many campuses had piloted or conducted portions of the assessment prior to July 2008, full implementation was not required until academic year 2008-09. Therefore, the data and narrative responses requested in the survey were in many instances projections and estimations. The fact that these numbers were estimates and that some policy decisions were still undetermined may have discouraged responses to some questions and may account for the failure of three campuses to return the survey.
Limitations of the Study
As previously mentioned, data gathered were of a somewhat speculative nature. The implementation of the TPA was mandated for candidates beginning their credential programs after July 1, 2008, so in many cases, campuses had not yet fully implemented. In addition, although there was a high response rate (19 of 22 campuses, or 86%), three campuses were not included in the data. Their responses would have provided a more complete picture of projected implementation.
For purposes of this discussion, the clusters of responses are organized into two categories: challenges and benefits. Because of the concerns about the lack of earmarked state funding support for implementation of the teaching performance assessment, challenges refer to any aspects of implementation that are associated with costs. Benefits refer to those results of the assessment implementation that relate to the positive effects of authentic, context-based teaching performance assessments.
Each of the campuses was asked to indicate the number of candidates enrolled in their elementary and secondary programs in the 2008-09 implementation year. Of the 22 campuses surveyed, 19 responded. The results are presented by total number of candidates in ascending order in Table 2.
The size of the CSU teacher preparation programs varies widely, from fewer than 100 candidates to over 2000. This variation creates significant differences in how implementation of the teaching performance assessment will impact campuses in terms of demands on fiscal and human resources. It is important to note that the numbers in Table 2 reflect total numbers of candidates enrolled in programs, not all of whom will complete the assessment in 2008-09. In addition, as this is a headcount, it does not reflect the number of teaching performance assessment tasks to be scored. The California TPA and FAST models are composed of four separate tasks, while PACT has one comprehensive teaching event completed by all candidates with three additional teaching tasks for Multiple Subject (elementary) candidates. For all of the models, there is an additional requirement for random double scoring of a percentage of the tasks to insure inter-rater reliability.
A major concern in the implementation of the teaching performance assessment is how to provide adequate oversight and management of the many related responsibilities. All 19 of the campuses that responded to the survey indicated that they had or planned to have a person assigned responsibility for TPA coordination, although only 18 campuses provided information about that role. A summary of their responses is provided in Table 3.
Of the campuses responding, the largest number (9) indicated that this role was assigned to a tenure-track faculty member. Five of the campuses indicated that an administrator, typically an associate dean, fulfilled this role, while the remaining four campuses indicated that this role had been assigned to a part-time, non-tenure-track faculty member. For tenure-track faculty in this role, assigned time for TPA coordination varied from a one-course reduction to a full-time assignment. In one case, an additional summer stipend was included. Administrators fulfilling this role typically estimated that they devoted 18 to 20 hours per week to TPA coordination, while part-time faculty typically received a one-course assignment for this work. While smaller programs were more likely to assign this role to an administrator, and mid-size to large programs were more likely to use a tenure-track faculty member, part-time faculty fulfilled this role for mid-size programs as well as for the larger programs, and an associate dean fulfilled the role in the largest program in the CSU system. The impact of the size of the program in determining how to assign this role is, therefore, unclear. Certainly concerns for cost effectiveness may play a role, as part-time faculty time is less costly. If an average salary for an associate professor is $75,000, for example, there would be an average cost of .58 FTE per campus equal to $43,500 per campus and $957,000 for the 22 campus CSU System. Nevertheless, competing concerns for consistency and longevity may explain the greater use of tenure-track faculty and administrators.
Among the responsibilities identified for this position in the survey responses were the following:
* work with programs to integrate preparation for and completion of teaching performance assessment tasks into curriculum as appropriate;
* ensure accessibility to information regarding the teaching performance assessment for students, faculty, and scorers;
* create, manage, and monitor TPA budget;
* manage the consent process, especially for videotaping;
* manage equipment, including purchase, maintenance, and checkout;
* work with programs to ensure an adequate supply of qualified trainers and scorers;
* coordinate local training, calibration, and re-calibration activities;
* report scores to candidates;
* coordinate remediation process;
* maintain ongoing communication with TPA external coordinators;
* coordinate use of the electronic platform; and
* assist with analysis of data and preparation of reports for internal and external review.
In the survey, the campuses were also asked if they had or planned to have one or more individuals assigned to assist the TPA Coordinator. Of the 18 campuses that responded, 16 (88.9%) responded affirmatively. Seven campuses (43.7%) indicated that they would assign a tenure-track faculty member for this role, two campuses (12.5%) would assign a part-time faculty member, five campuses (31.3%) would assign a staff member, and two campuses (12.5%) would assign student assistants. Calculations were made using an average professor's salary of $75,000, an average staff salary of $35,000, and an average student salary of $23,400. The average cost per responding campus would be $11,316 per campus or a total of $248,952 for the CSU system.
Another cost added to the accountability model for educator preparation is the need for academic unit assessment coordinators. Although this position is not related exclusively to teacher performance assessment, it clearly adds to the overall cost of operation and makes use of TPA data for performance measures. Fourteen campuses reported having an assessment coordinator. Assigned time ranged from .25 to 1.0 FTE. The cost associated with the 14 campuses, with the same values as the above paragraph, is an average of $48,750 per campus and $1,072,500 for the CSU system.
Scoring the Assessments
Several of the survey questions focused on plans for training scorers and scoring the assessments. The question regarding stipends for training to become an assessor or scorer was not clear, and only five campuses responded to this question directly. One campus did not offer compensation for training, one campus paid $91 per day, and three other campuses paid $100, $150, or $200 per day. There was an average of 24.5 assessors/scorers needed per responding campus. Given these campus responses, an average cost per campus for scorer training at the same average pay rates listed above would cause responding campuses to realize an average cost of $13,440. That means that the CSU system might pay $282,240 to train TPA assessors/scorers.
The campuses were also asked who the scorers would be and whether and how they would be compensated. Among the groups identified as potential scorers were tenure-track faculty, lecturers, university supervisors, retired teachers and administrators, teachers, and district and county office personnel. The plans for compensating scorers are summarized in Table 4.
For both the CalTPA and the PACT campuses, paying a stipend was the most common approach to compensating scorers, and there was some variation in the amounts of the stipends. CalTPA campuses paid smaller amounts for the first three tasks of the assessment and typically twice as much for the more comprehensive fourth task. Considering the scoring as part of a faculty member's workload was less common for PACT campuses than for CalTPA campuses or the FAST campus. This is likely due to the structure of the latter two assessments as four separate tasks that are more easily embedded in course work, as opposed to the single comprehensive teaching event in the PACT assessment, which can take as much as three hours or more to score. Using a unit of assigned time as compensation for scoring was less common, yet used by a few campuses. At standard replacement rate, even at seven teaching events per unit, this approach is over twice as costly as the highest stipend amount considered.
To understand the magnitude of the scoring costs, it must be understood that for the CalTPA campuses responding, the number of tasks to be scored, at four tasks per candidate, ranged from 410 to over 3,000 tasks. For the PACT campuses, the number of teaching events to be scored ranged from 165 to 367. An average cost to CalTPA campuses is $100 per candidate. The average cost per for PACT campuses is $106 per candidate. If there were an average of 501 candidates per campus (see Table 2), the cost per campus would be $103 per candidate, equal to a total of $51,603. Including 10% double scoring brings that total to $56,763. Multiplying that amount by 22 campuses comes to an overall CSU System cost of $1,248,786.
Ongoing training, calibration, and professional development are essential components to the reliability of the assessment scoring. Seven campuses did not report a specific plan for addressing these components, and the FAST campus provided in-house professional development related to the TPA. Of the nine campuses reporting that they were sending or planning to send faculty for ongoing professional development, cost amounts ranged from $3000 to $12,000. An average cost of $7,013 per campus should be projected. The CSU System would then realize a cost of $154,286 per year to provide professional development for Teacher Performance Assessment scorers.
For candidates who do not pass the TPA, credential programs are required to provide procedures for double scoring, appeal, remediation, and resubmission. Survey respondents estimated that 4-10% of candidates would be required to resubmit all or part of the assessment. Some campuses allow for one resubmission, while others reported that two resubmissions were allowed in order for candidates to improve their scores. Plans for addressing this requirement varied among campuses, but, in general, double scoring and resubmission were monitored and implemented by the TPA coordinator with faculty involvement in the remediation.
Some campuses reported that remediation would be provided through student teaching seminars or courses dedicated to assessment preparation or through referral to a qualified faculty member. Four campuses indicated that candidates not passing all or part of the assessment would be required to enroll in an additional course through extension or continuing education. The fee for the course would cover both remediation and the cost of scoring the resubmission. One campus reported that in cases where the entire PACT Teaching Event needed to be repeated, candidates would be required to re-enroll in student teaching in the subsequent semester.
Another major cost of TPA implementation at CSU campuses is the technology, software, and related support needed. The management of TPA data from required tasks, teaching events, or other assessments requires a mechanism to easily manage and track submissions, assessment scoring, and evaluator comments. A variety of software packages and portfolio systems are available for this purpose, and campuses were asked to indicate what they were utilizing.
TaskStream was the electronic portfolio system selected by 12 of the 19 CSU campuses. Two other comprehensive assessment and reporting tools, TK-20 and LiveText, are each used by one of the CSU campuses. Of the five remaining campuses surveyed, two were using variations of TaskStream integrated with other tracking tools, two were using locally developed portfolio and data management systems, and one was still evaluating available products. The commercial systems generally involve a fixed cost assessed either to the institution or to the candidates. Additional costs include the time needed for faculty and student training in the use of the systems. The costs for locally developed systems are less clear, as they are primarily human resource costs associated with development and management.
Another substantial cost associated with the teaching performance assessment results from the requirement of both CalTPA and PACT for candidates to submit a video of their work with students in the classroom. There was wide variation among CSU campuses in plans for funding video equipment costs, in the amount of equipment already available for use and in the acquisitions projected. Twelve campuses reported purchasing video cameras, tripods, and/or scanners. Three campuses noted that there is equipment available for students to check out of the media center. The one campus using the FAST model does not require video recording of lessons. There have been 410 video cameras intentionally purchased for this purpose by 12 campuses. At an average cost of $125 per camera, the total expenditure per campus is approximately $4,270. Each campus would need to add tripods, tapes, batteries, CDs, DVDs, replacement, and other equipment. At a minimum of $25 per camera for these supplies, the cost to each campus would be at least $854 annually. Each campus, based on these projections, would spend approximately $5,124 per year on video equipment. The CSU system would realize an annual cost of $112,728.
Several programs indicated that candidates were encouraged to use their own video equipment or to request the use of equipment available at their school sites. Others mentioned that resources were available elsewhere on campus for candidate checkout as needed. Concerns were expressed about allocating sufficient ongoing funding for equipment replacement costs and repairs. The need to store, maintain, and make the video equipment available for candidates presents additional demands on facility and staff resources.
One of the primary motivations for the creation of the CSU Deans of Education Task Force was the issue of cost. Twelve questions in the survey had cost and/or funding implications. While the costs associated with the implementation of the teaching performance assessment are difficult to ascertain, based on data from the survey, the individual CSU campuses will pay on average approximately $171,575 during AY 2008/09 to fully administer the assessment, although costs may vary based on campus size and individual application procedures. In addition, it is estimated that the CSU system will realize a total expense of $3,761,210. Table 5 presents a summary of associated expenses.
The lack of state funds specifically earmarked for implementation of the teaching performance assessment, combined with a new state budget that includes significant cuts to funding for higher education, creates a financial "perfect storm" in which implementation of this costly assessment must occur. Not surprisingly, many CSU deans of education are challenged to secure funding for the initial TPA implementation in AY 2008/09 and to sustain adequate funding support in subsequent years. Responses to the survey question regarding sources of funding included a range of answers that underscore the precariousness of funding support. While three campuses indicated that their funding would come from general funds, eight of the campuses planned to use one-time monies, including carry-forward funds and special allocations from Provosts. Four campuses indicated that they would have to use funds from internship and alternative certification grants. Other strategies included adding courses and units, using salaries from unfilled tenure-track lines, and reducing part-time faculty. While some private universities have added additional student fees to cover the cost of the TPA, this strategy was not allowed on CSU campuses because it would add yet another cost to already existing testing fees, credential fees, and other such costs that could become a deterrent to attracting future teacher candidates.
The problems with many of these funding strategies are that they don't provide adequate funding, they may negatively impact program quality and enrollments, or that they are not sustainable over time. With an average cost per campus of almost $200,000 annually, these data provide a strong case for the need for earmarked state funding for the TPA.
Considering the many challenges and considerable costs associated with the teaching performance assessment, it is important to consider what benefits might accrue from its implementation. The findings from the survey responses indicate three areas of benefit that align with the research by Darling-Hammond & Snyder (2006): increased communication and collaboration; influence on teachers' and students' learning; and use of results for program improvement. Because full implementation of the assessment has not yet occurred, this study was not able to determine if the benefits resulting from the TPA are cost effective. Benefits are determined over time, and are not evident at this time. Further study will be necessary to reveal if the required cost of implementation is cost effective by the benefit provided to P-12 student learning.
Increased Communication and Collaboration
Due to the regional nature of the California State University mission, collaboration with key constituents is essential to the success of programs, especially in Education. Most communities have multiple competitors seeking the same credential students, increasing the need for strong collaboration between CSU teacher preparation programs and P-12 districts and local businesses. Of the survey respondents, 81.3% reported that piloting and implementing the teaching performance assessment has increased collaboration with local districts and other university campuses. Consistent across many campuses that responded is collaboration with local district and county directors of the Beginning Teacher Support and Assessment (BTSA) induction program. This collaboration is particularly important due to the articulation implications of the TPA between pre-service teacher preparation and in-service teacher development. There are also multiple campuses where district and county office personnel are participating in the scoring of assessments. Respondents report a more focused dialogue with superintendents, principals, and district administrators. One campus described their Field Partners Convocation as a first annual academic event that brings the community into the life and culture of the university.
Collaboration between campuses has also benefited from the TPA implementation. Several regions reported bringing multiple universities together, both public and private, for bimonthly or regular meetings to discuss collaboration, share scoring opportunities, compare and assist with job descriptions, and provide support to each other. Learning from one another's successes and failures, seeking additional collaborations, and cross training for faculty are other benefits cited in the survey. One campus cited opportunities to collaborate with state agencies as a result of the TPA. Collaboration at this level has also led to networking meetings across the system and with CCTC that have assisted many campuses in the development of the TPA process, helped with legal issues and concerns, afforded opportunities to brainstorm and create best practices related to TPA implementation, and provided strategies to deal with potentially controversial issues such as videotaping and supporting special needs students, including English learners.
Influence on Learning
Completion of the teaching performance assessment is an arduous and often stressful task for student teachers and interns who are also coping with the demands of course work and teaching. One campus surveyed candidates following administration of the assessment and found that "candidates indicate that they find it to be a generally useful exercise, though they struggle to manage the timelines and some of the technical demands, given all the other requirements of the credential program." Nevertheless, a number of campuses reported positive responses from students regarding the benefits of the experience for their professional preparation. Among the observations on the positive influence of the assessment on candidates' learning shared by campus respondents were the following:
* Teacher candidates benefit by the project in that they are required to synthesize and apply information and skills learned in various courses. They become very aware of the complexity of teaching and demands of the profession.
* Candidates feel it crystallizes everything they have learned from our program and helps hone their skills.
* The feedback received from candidates is that they appreciate the preparation from their PACT experience.
* Candidates see the value as they reach the culminating Task 4, and can adequately demonstrate their skills and knowledge.
Although some faculty expressed concerns about the impact of the TPA on faculty and student workloads and about the potential for narrowing the teacher preparation curriculum, most felt that the TPA was beneficial in developing teacher candidates who are better prepared to enter the profession and to influence the learning of P-12 students. One respondent, for example, indicated that, "faculty felt the reflective questions in tasks enabled students to more deeply reflect on their teaching and observation." Another indicated that candidates were benefiting from an additional focus in the program on academic language and assignments associated with the assessment tasks.
While it is still too early to assess the full impact of the teaching performance assessment, following the full implementation of the TPA in spring 2009, important information on the benefits related to candidate learning will come from two surveys administered by the CSU System's Center for Teacher Quality. The CSU System-wide Evaluation of Teacher Preparation provides the results of surveys administered to both graduates of initial credential programs completing their first year as in-service teachers and their employers on their perceptions of the quality of preparation they received. The results for each campus are provided as composite scores on groups of questions substantively related to each other. Currently a task force is aligning the items on these surveys to the Teaching Performance Expectations. Since each of the TPA models is also aligned to the TPEs, it will be possible in future to conduct pre- and post-implementation studies to determine value added.
Responses to the survey also reveal that responding campuses are making use of assessment results to inform curricular change and improvement. One respondent described faculty as "eager to analyze results" following a spring 2008 pilot. Another indicated that the TPA had stimulated "excellent conversations" and "healthy dialogue" related to candidates' performance, even though many faculty still harbored strong reservations about the process. Most respondents indicated that the assessment results were useful in identifying program strengths and areas for improvement. The discussions sparked by the assessment results have led to changes in program curricula, course content, and course assignments. Areas identified by campuses as needing more curricular support included adapting instruction to meet special needs, supporting English learners, developing academic language, using assessments to monitor student learning, and linking research and theory to teaching practice and reflection. One respondent indicated that the changes prompted by the teaching performance assessment had made their programs "more consistent in terms of content delivery, tools used (e.g., lesson design, unit design) and pedagogical language and vocabulary." Two respondents indicated that their changes had led to a stronger focus on outcomes-based program design.
Conclusions and Next Steps
Asked what next steps they would recommend for the task force, many respondents suggested expanding the cross-campus discussions to include procedures, policies, and templates related to the teaching performance assessment as a way to share best practices and solve common problems. The venues suggested for facilitating these discussions included both the CSU Deans of Education meetings and the creation of ongoing job-alike or TPA Coordinator forums for solving common problems, finding solutions, and assisting others with implementation. These meetings could also help with facilitating inter-rater reliability studies, to share training responsibilities, and to develop a common pool of scorers. Respondents also proposed conducting joint research on the effectiveness of videotaping and the impact of the assessment on preparing quality teachers. An additional byproduct of this study is the clarity of purpose that must be shared with subject matter faculty in collaboration with Teacher Education faculty. By far the most common suggestion, however, was that a plan be developed to secure funding that is both sufficient and sustainable over time.
Challenges described by the survey respondents make clear why solving the funding issue is critical to the effective implementation of the TPA. Eliciting "a strong, consistent voice and action from the Chancellor's Office regarding resources and support for implementation" was desired by one respondent, but another cautioned against having the CSU system mandate a single assessment protocol for all campuses in light of the time, resources, and funding that have already been expended by campuses to pilot and implement a TPA model. One respondent hoped the state legislature might still provide funding specifically for TPA implementation. There was even a suggestion that legislators might be persuaded to reshape the TPA to decrease the number of tasks, thus reducing some of the potential costs.
The task force recommends revisiting this study after two academic years in order to see how full implementation was engaged and to provide direction for on-going administration of the assessment. Some of the questions that might be addressed in a future study include: (1) How do the costs and benefits of implementation and ongoing administration compare across the three TPA models? (2) Are there any notable differences between Northern and Southern California campuses in terms of costs and benefits? (3) How successful are each of these assessment models in weeding out poorly performing candidates, improving teacher quality, and increasing longevity in the profession? The inclusion of CalStateTEACH, the CSU statewide, fully online preparation program for elementary teachers should also be included in this future study in order to examine what similarities and differences exist between traditional and non-traditional programs in providing on-going administration of the assessment. Examining the effectiveness of all three state approved models and their impact on both candidates and programs will yield very useful information for California teacher education as well as for other states considering the development of field-based teaching performance assessments.
Arends, R. J. (2006). Performance assessment in perspective: History, opportunities, and challenges. In S. Castle & B. Shaklee (Eds.), Assessing teacher performance: Performance-based assessment in teacher education (pp. 3-22). Lanham, MD: Rowman & Littlefield Education.
California Commission on Teacher Credentialing. (2008). Standards of quality and effectiveness for teacher preparation programs for preliminary multiple and single subject teaching credentials. Sacramento, CA.
Chung, R. (Winter, 2008). Beyond assessment: Performance assessments in teacher education. Teacher Education Quarterly, 35(1), 7-29.
Darling-Hammond, L., Pacheco, A., Michelli, N., LePage, P. , Hammerness, K., & Youngs, P. (2005). Implementing curriculum renewal in teacher education: Managing organizational and policy change. In L. Darling-Hammond & J. Bransford (Eds.).Preparing teachers for a changing world; What teachers should learn and be able to do (pp. 442-479). San Francisco: Jossey-Bass.
Darling-Hammond, L., & Snyder, J. (2000). Authentic assessment of teaching in context. Teaching and Teacher Education, 16(5-6), 523-545.
Goldhaber, D. (Spring 2002). The mystery of good teaching: Surveying the evidence on student achievement and teachers' characteristics. Education Next, 2(1). Retrieved October 10, 2008 from http://www.hoover.org/publications/ ednext/3368021.html
Mitchell, A., Allen, S., & Ehrenburg, P. (2006). Spotlight on schools of education: Institutional responses to standards 1 and 2. Washington, DC: National Council for Accreditation of Teacher Education.
Pecheone, R., & Chung, R. (January/February 2006). Evidence in teacher education: The Performance Assessment for California Teachers (PACT). Journal of Teacher Education, 57(1), 22-36.
Porter, A., Youngs, P., & Odden, A. (2001). Advances in teacher assessments and their uses. In V. Richardson (Ed.), Handbook of research on teaching (4th ed.) (pp. 259-297). Washington, DC: American Education Research Association.
Sandy, M. (Spring 2006). Timing is everything: Building state policy on teacher credentialing in an era of multiple, competing, and rapid education reforms. Issues in Teacher Education, 15(1), 7-19.
Curtis L. Guaglianone
California State University, Bakersfield
California State University, Chico
Gary W. Kinsey
California State Polytechnic University, Pomona
& Robin Chiero
CalStateTEACH, California State University, Fresno
Curtis L. Guaglianone is the assistant to the Provost for Program Development at California State University, Bakersfield, Bakersfield, California; Maggie Payne is the associate dean of the College of Communication and Education at California State University, Chico, Chico, California; Gary W. Kinsey is the interim associate dean of the College of Education at California State Polytechnic University, Pomona, Pomona, California; and Robin Chiero is the regional director of CalStateTEACH at California State University, Fresno, Fresno, California.
Table 1 CSU Teaching Performance Assessment Survey Cluster Survey Item 1 1. Projected number of students in your programs for 2008/09. 2. Number of Tasks that you will score during AY 2008/09. 2 3. Do you have or plan to have someone assigned with responsibility for TPA/PACT coordination? 4. Do you have or plan to have one or more individuals) assigned to assist the TPA/PACT Coordinator including anticipated classification, time base, and primary responsibilities? 5. Do you have or plan to have someone who serves as overall Assessment Coordinator with responsibilities beyond TPA/PACT coordination including classification, time base, and primary responsibilities? 6. Do you have other coordination time that is assigned to TPA implantation, advising, assessing, etc.? 3 7. Do you, or do you intend to, pay a stipend for the participation of those who become trained TPA Assessors/PACT scorers? 8. What are your current thoughts (plans?) about compensation for assessors/scorers? Is scoring of TPA tasks/PACT Teaching events part of faculty workload? 9. How many trainers of assessors will you need during AY 2008/09? 10. How many scorers will you need during AY 2008/09? 11. How much is the stipend you pay for assessing TPAs? 4 12. Is there a particular software package or portfolio system you are using to manage TPA data? 13. How are you handling equipment costs (digital cameras, tripods, microphones)? 5 14. Have you made any projection regarding your costs and sources of funding related to remediation and resubmission? If so, please describe your current projections or considerations. 6 15. What are the additional assignments and costs associated with Technical Assistance Meetings, Consortia Meetings, Training or trainers Meetings, etc.? 16. What are your sources of funding related to TPA implementation? 17. Other costs, concerns, commitments resulting from TPAs? 7 18. Has the TPA implementation been an impetus for additional collaboration with P-12 or with other campuses? 8 19. What have you discovered or learned during the pilot regarding how performance assessment benefits your candidates, program, and faculty? 9 20. What "next steps" could the Education Deans' group take that would be most helpful to you as you prepare to begin the implementatin this year? 21. What other information, if any, would you like to have from other campuses that might help you with planning, structuring, staffing, and/or implementing? 22. Please explain any above item that you feel may need further explanation due to unique circumstances on your campus. Table 2 Number of Candidates per Respondent Campus, Unduplicated Count in Ascending Order Campus Total Multiple Subject Single Subject (Elementary) (Secondary) Humboldt 91 50 41 Channel Islands 105 70 35 San Luis Obispo 196 103 93 Chico 224 142 82 Sacramento 250 140 110 San Diego 251 159 92 Dominguez Hills 309 200 109 San Francisco 352 200 152 San Marcos 371 319 52 East Bay 390 204 186 Stanislaus 400 213 187 Porrona 410 210 200 Los Angeles 420 204 216 Northridge 570 322 248 San Bernardino 780 405 375 Fullerton 820 570 250 Bakersfield 890 485 405 Fresno 1,000 600 400 Long Beach 2,100 1,200 900 Total 9,519 5,381 4,138 Average 501 283 218 Table 3 TPA Coordinator Assignments TPA Coordinator Number of Campuses Assigned Time (N = 18) Tenure-track faculty 9 (50%) .25-1.0 of workload Lecturer 4 (22.2%) 1-1.5 course release Administrator 5 (27.8%) 18-20 hours per week Table 4 Compensation for Scoring by Assessment Model Compensation Plan Number of Amount of Compensation Campuses Ca1TPA Part of workload 3 (33.3%) N/A Stipend 6 (66.7%) Tasks 1-3 $25-$40 each, Task 4 $50-$80 Assigned time 0 (0%) N/A PACT Part of workload 1 (11.1%) N/A Stipend 4 (44.4%) $50-$100 per Teaching Event Assigned time 2 (22.2%) 1 unit per 5-7 Teaching Events To be determined 2 (22.2%) TBD FAST Part of workload 1 (100%) N/A Table 5 Summary of Teaching Performance Assessment Costs Expense Estimated Cost Estimated Cost by Campus for CSU System TPA Coordinator $43,500 $957,000 TPA Coordinator Assistant $11,316 $248,952 Assessment Coordinator $48,750 $1,072,500 Initial Training for Assessors $13,440 $295,680 Scoring of Assessments $56,763 $1,248,786 Equipment $5,124 $112,728 Professional Development for TPA $7,013 $154,286 Total $185,906 $4,089,932