Improving teachers of minority students' attitudes towards and knowledge of standardized tests. (The scholarship of teaching and learning).

Abstract

Standardized educational tests have important consequences for both minority and non-minority students. However, there are inconsistencies across these groups with respect to their understanding of the importance of these consequences, and with respect to their test preparation activities. In this study, we worked with teachers from two elementary schools that served predominantly minority populations. The teachers were surveyed regarding their attitudes towards and knowledge of standardized tests. After these data were gathered, a workshop was conducted to discuss the strengths and weakness of standardized tests. Before the workshop, teachers exhibited primarily negative attitudes towards standardized tests and demonstrated limited knowledge of important testing concepts such as reliability and validity. Those teachers who illustrated better understanding of these concepts tended to have more favorable attitudes towards standardized tests. The implications of improving teachers' attitudes and knowledge of standardized tests for improving the test performance of minority students is discussed.

**********

In this paper we report the results of the first stage of a study designed to improve minority students' performance on standardized tests. It is a common finding that schools and school districts comprising predominantly minority students tend to do poorer on standardized educational achievement tests than those comprising predominantly non-minority students. Many causes for this difference have been hypothesized, such as differences in educational resources and opportunity to learn, and factors related to test bias or inappropriate testing practices. However, cultural and psychological factors that could lead to test performance differences among groups have not been widely investigated (an exception is research conducted by Claude Steele and his colleagues, discussed in the next section). For example, many students in the United States are taught at an early age that tests are important and that they should try their best to do well on them. Other students are completely unfamiliar with the consequences tied to their performance on standardized tests. A hypothesis motivating the present study is that increasing minority students' motivation to do well on standardized tests will result in meaningful improvement in their performance on these tests. Thus, our goal is to create a culture of testing success.

One way to improve students' knowledge of and motivation to do well on standardized tests is to work with their teachers. Several studies revealed that elementary and secondary school teachers lack general knowledge of standardized testing and appropriate test preparation activities (Impara, Divine, Bruce, Liverman, & Gay 1991; Impara & Plake, 1995; Nolen, Haladyna, & Haas, 1992; O=Sullivan & Chalnick, 1991; Popham, 1991; Taylor & Walton, 1998). Similarly, many teachers have negative attitudes towards standardized tests (Green, 1992; Nolen, et al., 1992). Thus, our first goal in raising minority students' test scores was to work with their teachers to improve their understanding of standardized testing, including how to prepare students for taking these tests, and how to use standardized test results to improve instruction.

Our initiation of these activities in pursuit of a culture of testing success is consistent with the suggestions of Steele (1997) who argued that changes in minority students' expectations could lead to improvements in their test performance. Steele and his colleagues (e.g., Steele, 1997; Steele & Aronson, 1995) describe several psychological factors that could negatively affect minority students' academic performance. Paramount among these situational factors are "stereotype threat" and "disidentification" with school. The phenomenon of stereotype threat applies to high achieving students who are engaged in school and for whom academic success becomes part of their self-concept. When confronted with a situation in which members of their group are expected to perform poorly, for example when taking a standardized test, the "threat" of this "stereotype" causes undue anxiety that negatively affects their test performance. The disengagement phenomenon refers to students who disengage from academics in part due to the stereotype that their cultural group is not expected to do well in school. As described in the next sections, by creating a culture of testing success within schools that serve predominantly minority students, we strive to reduce the stereotype threat and disidentification with school that may be experienced by these students.

Contrary to common criticisms, standardized educational tests are designed to promote fairness in evaluating student achievement. Standardized tests attempt to level the playing field for all students by ensuring uniform test content and administration conditions. However, as stated above, the assessment playing field is not level in terms of students' and teachers' attitudes towards and knowledge of these tests. There are cultural differences in the importance attributed to standardized tests, and better attitudes towards these tests are associated with improved student achievement. Therefore, this study focused on improving teachers' attitudes towards and knowledge of standardized tests. We hypothesized that: 1) standardized tests would not be popular among teachers who primarily teach minority students (because these tests tend to consistently provide "bad news" regarding their students' achievement); 2) like most elementary school teachers, these teachers would have limited knowledge regarding the purposes and utility of standardized tests; and 3) by improving teachers' attitudes towards and knowledge of standardized tests, their test preparation activities would improve. This study reports our findings regarding the first two hypotheses.

Method

Instruments

To measure the teachers' attitudes towards and knowledge of standardized tests, two scales were developed. The first scale was an "attitudes towards tests" scale (ATT), the second scale was a "knowledge of tests" scale (KOT). The ATT comprised 14 statements regarding standardized tests (see Table 1). The teachers were asked to respond to each statement and indicate their agreement along a six point Likert-type scale where "1" indicated "strongly agree" and "6" indicated "strongly disagree." Five of these 14 statements were borrowed from the survey developed by Green (1992). The other nine items were developed by the authors to better fit the purposes of this study. The KOT comprised 16 items that measured fundamental measurement concepts such as reliability, validity, and interpreting test results. Twelve of these items were adapted from a scale developed by Impara and Plaice (1995) designed to assess teachers' understanding of the "standards for teacher competence in educational assessment" (American Federation of Teachers (AFT), National Council on Measurement in Education (NCME), National Education Association (NEA), 1990); the remaining items were developed by the authors. Fourteen of the KOT items were multiple choice, and two were true false.

Reliability data for the ATT were gathered from the teachers participating in this study (n=50) and graduate students in assessment and research methods classes in the School of Education at a large northeastern university (n=60). All graduate students were tested at the beginning of the course, before a lecture on principles of testing. The internal consistency reliability estimate (coefficient alpha) for the scale was .80. Reliability data for the KOT were gathered from the teachers participating in the study and from educational research methods and test construction classes at this same university (n=131). Again, all students were assessed before instruction on testing principles. This scale also exhibited good internal consistency reliability (alpha=.87). For some analyses, the teachers' responses were compared with those from the graduate students in the pilot sample.

Participants

Two elementary (K-5) schools in an urban school district participated in this study. Both schools serve predominantly minority students, most of whom are Latino/Latina or African American. A "rap session" was conducted at each school to determine the testing issues the teachers thought were important. Thirty-nine teachers from one school, and eleven teachers from the other, participated in these sessions and completed the ATT and KOT questionnaires. Subsequently, twenty-nine teachers from the first school participated in an eight-hour workshop on standardized testing. No data were collected on the race or sex of the participants.

Workshop

The curriculum for the workshop was based on issues brought up at the rap session and on suggestions from previous research regarding what teachers need to know about standardized tests (e.g., AFT, NCME, NEA, 1990; Schafer, 1991). The workshop covered: purposes of standardized tests, distinctions between norm- and criterion-referenced tests, fundamental concepts in reliability and validity, the pros and cons of different test formats (multiple-choice, performance, and portfolio assessments), understanding and interpreting standardized test scores, and how to prepare students to take standardized tests. In addition, recent research on testing minority children was discussed, and the "ABC's of Testing" video, produced by the Joint Committee on Fair Testing Practices (1993) was viewed and discussed. At the conclusion of the workshop the ATT and KOT questionnaires were distributed, but unfortunately, only seven questionnaires were returned, which prevented us from making reliable pretest/posttest comparisons.

Results

Attitudes Towards Standardized Tests

In general, the teachers' responses to the ATT reflected negative attitudes. In particular, the teachers indicated they did not believe standardized tests should be used to evaluate teachers, nor did they believe that standardized tests were valid indicators of student achievement. They also endorsed the belief that standardized tests force teachers to teach to the test. They tended to endorse the notion that standardized tests are biased against bilingual students, but they did not express the belief that these tests are biased against women or African Americans. Perhaps the best summary of the teachers' attitudes was their responses to the statement "I hate standardized tests." Forty of the forty-nine teachers who responded to this question selected a response option on the "agree" part of the scale. Twenty-six percent of the teachers selected "1" (strongly agree) on the six-point scale. The survey statements and a summary of the teachers' responses are presented in Table 1. The results summarized in Table 1 indicate that teachers had negative attitudes associated with some uses of standardized tests, but that they did see some utility, such as their use for measuring important instructional outcomes. The total ATT scores were compared across the elementary school teachers and the graduate students from the research methods and assessment courses. The mean scores on the ATT differed by about one point across the two groups (32.7 for the graduate students and 31.5 for the elementary school teachers); this difference was not statistically significantly different (t91= .55, p=.59). Thus, the attitudes towards standardized testing were similar across the teacher and graduate student groups. See issue's website <http://rapidintellect.com/AEQweb/spri02.htm>

Supplementary analyses were also conducted to determine if teachers' attitudes towards standardized tests were associated with their number of years of teaching experience and whether they had previous course work on testing. Neither of these variables indicated a significant correlation with ATT scores (both r statistics were near zero). In addition to the attitude statements on the ATT, the teachers were also asked if they were interested in learning more about interpreting standardized test scores, and to indicate their expectations regarding how well their students in general would perform on standardized tests this year. Further, the teachers were asked how well their bilingual and African American students would perform on these tests this year. Finally, the teachers were asked if they expected the performance of their students on standardized tests to improve within the next several years. The teachers' responses to each of these five additional questions exhibited positive correlations with the ATT total score (range of r statistic was .29 to .47, all p<.05). Those teachers who were more interested in learning more about interpreting standardized test scores, and those who had higher expectations regarding student performance, had more positive attitudes towards standardized tests.

Knowledge of Standardized Tests

In general, the teachers' responses to the KOT indicated that they had a reasonable understanding of content validity and classroom testing, but had very poor understanding of test score reliability and interpreting the results from standardized test scores. The most striking misconceptions were related to reliability. For example, when asked what action would increase the reliability of a multiple choice test, 62% of the teachers chose the incorrect response "add an essay component." The mean score of the teachers on this scale was 9.4, which represents a percent correct score of 59. The mean score of graduate students in the pilot test sample was 11.44, which represented a percent correct score of 72. The difference in mean KOT score between the teachers and graduate students was about half a standard deviation, and was statistically significant (t(179)= 3.26, p= .001).

The correlation between the KOT and ATT scores was near zero (r=. 10), suggesting that attitudes towards standardized tests are not related to knowledge of these tests. However, two of the fifteen KOT questions did exhibit statistically significant (at p <.05) correlations with ATT scores: one question measuring the distinction between norm- and criterion-referenced tests (answered correctly by 68% of the teachers), and the other measuring the ability to distinguish reliability from validity (answered correctly by only 29% of the teachers). Teachers who answered these two questions correctly tended to have higher ATT scores (i.e., better attitudes towards standardized tests).

Group Discussions

The specific issues discussed at the workshop focused on five topics: 1) purposes of standardized tests (including norm-referenced vs. criterion referenced distinctions), 2) reliability and validity concepts, 3) types of standardized tests, 4) teaching test taking strategies to students, and 5) understanding and interpreting test scores. These topics were introduced in a discussion format, and the teachers were actively engaged in the sharing of material. Standardized test classroom reports, disseminated by the school district the previous year, were used as examples to help understand these data. In addition, the teachers were given a series of open-ended questions in small groups asking them to identify strategies for improving the test performance of their students. Some of the strategies identified by the teachers were: testing should be a school-wide program involving all school personnel, testing should be discussed as a time to celebrate what was learned, teachers should talk to students regularly about assessments, parents should be involved through workshops, and any successes the children demonstrate on the tests should be acknowledged and rewarded (e.g., a special lunch). The teachers also came up with strategies for changing the physical environment during testing, strategies for incorporating test preparation into the curriculum, and strategies for involving parents in test preparation activities.

Discussion

The results of this study provided at least partial support for our hypotheses. First, in general, the teachers surveyed had negative attitudes towards standardized tests. However, they did acknowledge some of the benefits of standardized testing. Second, the teachers' knowledge of standardized tests was limited. Although their responses reflected knowledge of appropriate testing practices, the results suggest that these teachers could benefit from further instruction regarding (a) understanding test score reliability, (b) how to interpret standardized test scores and (c) how to use test results to improve their instruction. This finding is consistent with the Standards for Teacher Competence in Educational Assessment of Students (AFT, NCME, NEA, 1990). These topics were discussed at the teacher workshop and the teachers seemed to benefit from these discussions.

One of the most positive outcomes of the workshop was the number of ideas the teachers generated for creating a culture of testing success. As mentioned above, the teachers talked about making the improvement of test scores a team effort that involves teachers, students, parents, and administrators. The ideas to add test-taking strategies to their lesson plans, and to reward students for doing well on tests, are other examples of how such workshops can empower teachers to help improve the test scores of minority students. The fact that the teachers began to think about test preparation in a positive way is encouraging, since we hope such activities will lead to improved test scores for their students.

The results suggest that working with teachers regarding the understanding of standardized testing could be an important first step in promoting a culture of testing success within schools that often view such tests with frustration, despair, or apathy. Our strategy of holding a "rap session" for teachers to air their complaints about testing, in advance of the workshop, was helpful for building rapport with the teachers and for tailoring the workshop towards their needs.

An important anecdote to add to this study is that, for both schools, the students' standardized achievement test scores increased (in comparison to national norms) during the administration following the workshop. This increase was the first in the history for both schools. In no way can we conclude that our work with the teachers was responsible for this increase due to the lack of control of extraneous variables in this study. However, the motivation behind this study was to improve teachers' understanding of and attitudes towards standardized tests, with the hope that such changes would reflect positively on students' test performance. Although too few posttest data are available for analysis, our interactions with the teachers indicate that they benefited from the workshop and changed their test preparation strategies.

Working with teachers regarding their understanding of and attitudes towards standardized tests could be an important component of reducing the achievement gap between minority and non-minority students. Once teachers have acquired more positive attitudes towards standardized tests, and subsequent to better training in preparing students to take such tests, the parents could be brought into the process, This would further facilitate students' understanding of the importance of standardized tests, and increase their motivation to do well on them. It may also reduced disidentification with school as well as stereotype threat (Steele, 1997). We hope future research focuses on implementing such teacher and parent workshops and evaluating whether such workshops improve (a) teacher and parent attitudes towards standardized tests, (b) the degree to which minority students are prepared to take such tests, and (c) the motivation of minority students to do well on these tests.

References

American Federation of Teachers, National Council on Measurement in Education, & National Education Association (1990). Standards for teacher competence in educational assessment of students. Washington, DC: American Federation of Teachers.

Green, K. E. (1992). Differing opinions on testing between preservice and inservice teachers. Journal of Educational Research, 86, 37-42.

Impara, J. C., Divine, K. P., Bruce, F. A., Liverman, M. R., & Gay, A. (1991). Does interpretive test score information help teachers? Educational Measurement: Issues and Practice, 10(4), 16-18.

Impara, J. C., & Plake, B. S. (1995). Comparing counselors' school administrators' and teachers' knowledge in student assessment. Measurement and Evaluation in Counseling and Development, 28, 78-87.

Joint Committee on Fair Testing Practices (1993). The ABCs of school testing [Film]. (Available from the National Council on Measurement in Education, 1230 17th Street, NW, Washington, DC 20036)

Nolen, S. B., Haladyna, T. M., & Haas, N. S. (1992). Uses and abuses of achievement test scores. Educational Measurement: Issues and Practice, 11(2), 9-15.

O'Sullivan, R. G. & Chalnick, M. K. (1991). Measurement-related course work requirements for teacher certification and recertification. Educational Measurement: Issues and Practice, 10(1), 17-19.

Popham, W. J. (1991). Appropriateness of teachers' test-preparation practices. Educational Measurement: Issues and Practice, 10(4), 12-15.

Schafer, W. D. (1991). Essential assessment skills in professional education of teachers. Educational Measurement: Issues and Practice, 10(1), 3-6.

Steele, C.M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613-629.

Steele, C.M., & Aronson, J. (1995). Stereotype threat and the intellectual performance of African Americans. Journal of Personality and Social Psychology, 69, 797-811.

Taylor, K. & Walton, S. (1998). Children at the center: A workshop approach to standardized test preparation, K-8. Portsmouth, NH: Heinemann.

Stephen G. Sireci, University of Massachusetts Amherst Brunilda DeLeon, University of Massachusetts Amherst Ernest Washington, University of Massachusetts Amherst

Stephen, Associate Professor and co-Director of the Center for Educational Assessment in the School of Education, specializes in test development and test evaluation. Brunilda, Associate Professor in the School of Education, teaches in the Department of Student Development and Pupil Personnel Services and is affiliated with the Social Justice in Education doctoral program. Ernest, Professor in the School of Education, teaches in the Department of Teacher Education and Curriculum Studies.