Content Validity of Preservice Teacher Portfolios in a Standards-Based Program.

The introduction of a standards-based teacher education program in Kentucky was framed by eight New Teacher Standards adopted state wide. At Eastern Kentucky University, the Teacher Education Portfolio was implemented to (1) provide more meaningful, valid indicators of what preservice teachers know and can do, (2) enhance both teaching and learning, and (3) provide useful assessment information for reporting to people outside the classroom or local context. This article presents the eight New Teacher Standards and describes the connection between the content in an assessment course and the standards. A course assignment resulted in a portfolio entry designed to demonstrate the students' knowledge and skills for Standard four, "The teacher assesses learning and communicates results to students and others."

The introduction of a restructured standards-based teacher education program at Eastern Kentucky University mandated the inclusion of portfolios for preservice teachers. There were three major expectations for the preservice teacher portfolio. The portfolio was to: (1) provide more meaningful, valid indicators of what preservice teachers know and can do, (2) enhance both teaching and learning, and (3) provide useful assessment information for reporting to people outside the classroom or local context. Recommended practices related to portfolios (e.g., After & Spandell, 1992; Wiggins, 1989; and Wolf, 1991) were reviewed during the design stage of the Teacher Education Portfolios. If preservice teacher portfolios are to be used as assessment, issues of validity and reliability (Ghiselli Campbell & Zedeck, 1981; Mehrens, 1992) must be addressed. The first reliability consideration was the consistency of scoring of both the parts and the whole portfolio. This issue is still being resolved. Portfolio entries are all scored with a four-point rubric, however, the scoring is done by individual instructors/professors. Clearly, the most valid measures of whether or not a student can perform a certain task is to design a performance assessment that requires the student to "show what you can do." However, even readers with only tangential knowledge of assessment can recall the relationship between reliability and validity. Reliability is a necessary but not sufficient condition for validity. That is, a test cannot be valid unless it is reliable. Feldt (1997) recently examined this relationship by exploring the frequently quoted dictum of classical test theory which states that validity can never be greater than reliability (classical test theory offers an equation for calculating the relationship). Felt concluded, "If test authors allow themselves to be overly swayed by the desire to attain high reliability, they might easily favor those items that measure lower order skills" (p. 386). The Teacher Education Portfolio was planned to assess higher order thinking skills (Bloom, 1956). The preservice teacher is required to write a reflection for each portfolio entry, thus including "evaluation," Bloom's highest cognitive category. This article will address two validity issues: (1) the measures taken to assure the content validity of the overall preservice teacher education portfolio and (2) the alignment of the performance based assessment in a newly-created "Assessment in Education" course to assure content validity. Teacher education in Kentucky is Standards based. The Kentucky Standards Board was established with the comprehensive Kentucky Education Reform Act (KERA) of 1990. KERA also mandated an internship year as teacher certification requirement. The eight New Teacher Standards are used as evaluative criteria for the Kentucky Teacher Internship Program (KTIP). Because these standards define what teachers should know and be able to do, they were used as content guides for the Teacher Education Portfolio. The eight standards are:

1. The teacher designs/plans instruction and learning climates that develop student abilities to use communication skills, apply core concepts, become self-sufficient individuals, become responsible team members, think and solve problems, and integrate knowledge.

2. The teacher creates a learning climate that supports the development of student abilities to use communication skills, apply core concepts, become self-sufficient individuals, become responsible team members, think and solve problems-and integrate knowledge.

3. The teacher introduces/implements/ manages instruction that develops student abilities to use communication skills, apply core concepts, become self-sufficient individuals, become responsible team members, think and solve problems, and integrate knowledge.

4. The teacher assesses learning and communicates results to student and others with respect to student abilities to use communication skills, apply core concepts, become self-sufficient individuals, become responsible team members, think and solve problems, and integrate knowledge.

5. The teacher reflects on and evaluates specific teaching/learning situations and/or programs.

6. The teacher collaborates with colleagues, parents, and other agencies to design, implement, and support learning programs that develop student abilities to use communication skills, apply core concepts, become self-sufficient individuals, become responsible team members, think and solve problems, and integrate knowledge.

7. The teacher evaluates his/her overall performance with respect to modeling and teaching Kentucky's learner goals, refines the skills and processes necessary, and implements a professional development plan.

8. The teacher demonstrates a current and sufficient academic knowledge of certified content areas to develop student knowledge and performance in those areas.

Each course in the new teacher education program at EKU has a performance event or assessment that has been designed by the instructor/professor to address at least one of the New Teacher Standards. Designing the teacher education portfolio around these eight standards is very similar to the process a classroom teacher (or professor) uses to design course tests/assessments. The eight standards represent the "content" portion of the familiar test "blueprint" or "table of specifications."

Developing a table of specifications, or test blueprint, before designing any assessment assures that the relevant content of the domain will be sampled in a representative manner (Linn & Gronlund, 1995). The "blueprint" for the Teacher Education Portfolio was designed as a "Teacher Portfolio at a Glance," a matrix that the preservice teacher used to indicate where each entry fit the Standards. The eight New Teacher Standards are "tabs" in the portfolio. The Teacher Portfolio at a Glance page allows the reader to do a fast "content validity" check. Portfolio entries may be cross indexed, that is, they may be relevant for more than one of the eight standards.

The following is the performance assessment that was required in a newly designed "Assessment in Education." Students in the class are for the most part upper level undergraduates (some postgraduate students returning for teacher certification are also in the class). The students come from all majors in the College of Education. Thus, music, art, English, science, middle school, and elementary school preservice teachers add much variety to the course. The objectives for this course, incorporating national and state assessment standards are:

Upon completion of this course the student will be able to:

1. Choose assessment methods appropriate for classroom instructional decisions.

2. Develop assessment instruments appropriate for classroom instructional decisions.

3. Administer, score, and interpret the results of both externally-produced and teacher-produced assessment methods.

4. Use assessment results when making decisions about individual students, planning teaching, developing curriculum, and school improvement.

5. Develop valid pupil grading procedures which use pupil assessments.

6. Communicate assessment results to students, parents, other lay audiences, and other educators.

7. Recognize unethical, illegal, and otherwise inappropriate assessment methods and uses of assessment information.

8. Use multiple assessments and sources of data.

9. Make appropriate provisions for assessment processes that address social, cultural, and physical diversity.

10. Accurately assess student performance using the established criteria and scoring guides consistent with Kentucky's assessment program and the Commonwealth Accountability Testing System (CATS).

11. Promote student self-assessment using established criteria and focus student attention on what needs to be done to move to the next performance level.

12. Systematically collect and analyze assessment data and maintain up-to-date records of student progress.

The following assignment was used to assess Objective 2, "The student will be able to develop assessment instruments appropriate for classroom instructional decisions."

Test and Construction Project

Part I

Paper-and-pencil test

1. Select some course or unit of study within a course. Create a title for the project.

2. Identify the content to be taught, the grade level, and the time frame for the instruction.

3. List five to 15 instructional objectives.

A. Use the appropriate "Bloom verb" to indicate the cognitive level expected.

B. Use only one action verb per objective.

C. Have at least one objective at each of the three cognitive levels that can be measured with paper-and-pencil tests.

4. List the subject-matter topics to be covered in the instruction.

5. Create a Table of Specifications (Test Blueprint) for the test.

6. Create a 20-item test using a combination of supply-type, short-answer, and selection-type items. The test will include: (a) complete directions, (b) test items that are appropriate for the specific learning outcomes being measured, and (c) a scoring key. Each test item should be keyed to a specific learning objective (listed in the list of objectives).

Part II. Essay Questions

7. Using the same topic and unit of instruction, create two extended-response or restricted-response essay questions. In asking the question, use the "Bloom verb" to indicate the cognitive level expected. (These questions will be used to measure "higher-order" thinking.)

A. Define how the essay questions will be scored.

B. If a rubric is to be used, design the rubric.

Part III. Performance-based Assessment

Using the same topic and unit of instruction, create a performance-based assessment to measure learning. Include the knowledge and thinking skills that will be required for the student to complete the task. Design a rubric to be used in scoring the task.

Note: We will have an Internet assignment that will include a visit to the KDE Website. The Kentucky Department of Education has included examples of performance-based assessments and scoring rubrics. These may be adapted for your grade-level and content.

General Instructions

A Bibliography of books and other source materials used in completing the project should be included.

This Bibliography should be in American Psychological Association (APA) style (see 4th edition of the style manual).

The class was organized as one hour of lecture and one hour of lab twice a week. The lab time was spent working on the Test Construction Project in group and individually. Student evaluations of the course are always positive. A typical comment, "I didn't know anything about test construction when I took this class. It's really interesting!"

References

Arter, J. A., & Spandell, V. (1992). Using portfolios of student work in instruction and assessment. Educational Measurement: Issues and Practice, 11 (1), 36-44.

Feldt, L. S. (1997). Can validity rise when reliability declines? Applied Measurement in Education, 10(4), 377-387.

Ghiselli, E.E., Campbell, J. P., & Zedeck, S. (1981). Measurement theory for the behavioral sciences. New York: W. H. Freeman and Company.

Linn, R. L., & Gronlund, N. E. (1995). Measurement and assessment in teaching (7th ed.). Columbus, OH: Prentice Hall.

Mehrens, W. A. (1992). Using performance assessment for accountability purposes. Educational Measurement: Issues and Practice, 11 (1), 3-9, 20.

Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70, 703-713.

Wolf, D. P. (1989). Portfolio assessment: Sampling student work. Educational Leadership, 46(7), 35-39.

Beverly M. Klecker, Ph.D., Eastern Kentuky University.

Correspondence concerning this article should be addressed to Beverly M. Klecker, Ph.D. 406 Combs Building Eastern Kentucky University 521 Lancaster Avenue Richmond, KY 40475