Reliability for the Greek Version of the "Test of Everyday Reasoning (TER)".

The core critical thinking skills, identified in "The Delphi Report" as essential elements for workplace and educational success, are targeted in a standardized 35 item multiple-choice assessment tool entitled the "Test of Everyday Reasoning (TER)" which is designed to provide a representation of a person's overall critical thinking ability. In this paper the reliability of the Greek version of the "Test of Everyday Reasoning (TER)" is investigated. In the research participated 350 Greek students. Reliability analysis was conducted for the overall sample of the research and the subgroups: (a) young students (11-13 years old) and university students, (b) male and female. Data showed that the Greek TER was a reliable instrument measuring critical thinking for the above mentioned groups.

**********

In current educational research there is a widespread acceptance that critical thinking development is an important dimension of education (Coles & Robinson, 1989; Ennis, 1987; Garnett & Tobin, 1984; Krusse & Preseissen, 1987; McGuinness & Nisbet, 1991; Meadows, 1996; Paul et al., 1995; Perkins, 1993; European Union, 1995: 12; Gadzella et al., 2006). In Greece the "Cross Thematic Curriculum Framework for Compulsory Education" (1) and the new Curricula for elementary and secondary education state clearly that critical thinking is an important educational aim (Official Gazette, 2003).

In spite the fact that the necessity of developing student's critical thinking is proposed by academics, researchers and educators, there is great difficulty in defining it and, consequently, in assessing it. The concepts advanced by R. Ennis (1987), R. Paul et al. (1995), M. Lippman (1991), H. Siegal (1988) and R. Sternberg (1985a, 1985b, 1987) among others were prominent and influential. In the relevant literature critical thinking is conceptualized according to where emphasis is given each time: i.e. as logical fallacies (Dreyfus & Jungwirth, 1980; Jungwirth & Dreyfus, 1990; Jungwirth, 1987), as formal reasoning processes or skills (Blair & Johnson, 1980; Lawson, 1982, 1985; Obed, 1997), as scientific reasoning in general (Friedler et al., 1990) etc.

An historical benchmark in conceptualizing critical thinking is the consensus of a panel of 46 leading theoreticians, teachers and critical thinking assessment specialists from several disciplines as it is described in the conference proceedings of the American Philosophical Association (APA) widely known as "The Delphi Report" (Facione, 1990a: 12). Based on the APA Delphi consensus conceptualization of critical thinking a series of psychometrical instruments were created among which was the "Test of Everyday Reasoning (TER)" that was used in our research. The Delphi conceptualization was reaffirmed by independent research in the 1993/1994 national survey of employers, educators and policy makers, a replication study conducted by the National Center for Higher Education Teaching, Learning and Assessment (Jones et al., 1995).

Description and Scales of TER

The Test of Everyday Reasoning (Facione, 2001) is a 35 multiple-choice item test designed to measure reasoning and critical thinking skills. The items of TER are multiple choice questions designed to be scored dichotomously (2) with one correct answer and three or four distractors which represent frequently made errors or are designed to attract the attention of those who exhibit what are known as dispositional failures in reasoning (Engel, 1999). TER is suitable for persons in late childhood, adolescent and adult populations because the only background knowledge that is assumed is readily achievable through normal maturation and elementary schooling.

TER provides six scores for each individual: (a) an Overall Score which represents the number of the items answered correctly, indicating the overall ability of critical thinking, (b) three sub-scales: (i) "Analysis", (ii) "Evaluation", (iii) "Inference", and finally (c) two sub-scales which follow a more traditional conceptualization for critical thinking and reasoning skills: (i) "Deductive Reasoning", and (ii) "Inductive Reasoning" (Facione, 2001: 11-12, 25).

Content Validity, Construct Validity

The content validity refers in general to how the specific items comply with two standards: (a) if the items represent the entire set of possible test items within a specified domain and (b) if 'sensible" methods of test construction are employed.

TER complies with (a) standard of representation of all possible test items since every item of TER was carefully selected for its theoretical relationship to the Delphi Critical Thinking conceptualization.

In regard to standard (b) the appropriateness of a multiple-choice test format to measure critical thinking must be established. Authors of measurements texts agree that higher order cognitive skills can validly and reliably be measured by well-crafted multiple-choice items (Haldyna, 1994: 28). Most of the critical thinking assessment experts who participated on the Delphi panel (Facione, 1990a) agree to this point. Furthermore, the construct validity of TER is grounded on the results of relative research indicating that it is strongly correlated (0,766) with "California Critical Thinking Skills Test" (CCTST) (Facione et al., 2002; 1990b; 1990c; 2001).

Methodology

In order to translate TER into Greek we followed the California Academic Press translation policies assuring the quality, validity, reliability, cultural sensitivity and ethical use of TER in Greece. The result of this procedure was an authorized translation. This research aims to present the Kuder-Richardson 20 coefficient for the Greek version of TER for the overall sample and the subgroups of (a) students (11-13 years old) and university students, (b) male and female participants. Our main research interest was focused in young students since we wanted to determine whether TER was a reliable psychometrical instrument for the assessment of young students' critical thinking skills.

The study design was a questionnaire survey. In an effort to minimize the instrumentation effect the researchers themselves administered the questionnaire. For data storage and analysis SPSS 13.0 for Windows (SPSS Inc. Chicago, IL, USA) was used.

A field-test was designed, including primary school students, secondary school students and undergraduate students. We used a random cluster sampling technique, in which schools served as clusters, for primary and secondary school students. All the undergraduate students originated from the Faculty of Primary Education of the National and Kapodistrian University of Athens. The sample originated from urban, suburban and rural areas of Greece representing the regional diversity of Greece.

Results

In the research 350 students participated. There was special interest for small ages (11-13); therefore most of our sample (N=310) included students within this range of age. A smaller sample of university students (N=40) was included in order to acquire comparable results, since no relevant research existed for the age range that our study targeted. Participants' age ranged from 11 to 26 years.

The analysis aims to present the Kuder-Richardson 20 (KR-20) coefficients of the Greek TER for the overall sample as well as for specific subgroups.

In the test manual (Facione, 2001: 16) four separate KR-20 coefficients are presented as a statistical evaluation of the internal consistency of TER derived from four different samples which range from 0,72 to 0,89 (N=145, KR-20=0,78; N=201, KR-20=0,76; N=582, KR-20=0,72; N=113, KR-20=0,89).

For the overall Greek sample the KR-20 coefficient was found 0,74 (N=350) within the above mentioned range.

The KR-20 coefficient for the subgroup of the 11 to 13 year old students was found high (N=310, KR-20=0,84). The KR-20 coefficient for the subgroup of the university students was found lower (N=40, KR-20=0,652) which might be explained by the small sample size.

Finally, the KR-20 was found relatively higher for females (N=184, KR-20=0,703) than for males (N=166, KR-20=0,681).

Discussion

Currently in Greece there is an extended discussion concerning the development and the assessment of students' critical thinking skills which is also reflected in the new curricula for primary and secondary education, as already mentioned. Therefore the research targeted mainly the ages of 11-13 years old in an effort to meet the need for instruments assessing critical thinking in Greece. Consequently the results of our research conclude that TER is suitable for 11-13 years old students. On the other hand we included in our sample the undergraduate student group in order to acquire comparable research results with the already conducted research, since there has been no other published research on TER for the target age group (11-13 year old students). Although the reliability coefficient was relatively low for university students in comparison with the already conducted research it was high enough to assume appropriateness for the specific age group.

Probably the female students of the sample completed the questionnaire more carefully than the male students. This is a possible explanation for the higher KR-20 that the females achieve. It must be noted, though, that all the university students were female.

Concluding, the items in the Greek version of TER form a scale that has reasonable internal consistency/reliability for all subgroups of our sample.

References

Blair, J. & Johnson, R. (1980). Informal Logic. Interness, CA: Edgepress.

Coles, M. & Robinson, W. (1989). Teaching Thinking: A Survey of Programmes in Education. Bristol: The Bristol Press.

Dreyfus, A. & Jungwirth, E. (1980). Students' Perceptions of the Logical Structure of Curricular as Compared with Everyday Contexts - Study of Critical Thinking Skills. Science Education 64(3), 309-321.

Engel, S. M. (1999). With Good Reason: An Introduction to Informal Fallacies. (6th ed.). New York, NY: St. Martin's.

Ennis, R. (1987). A Taxonomy of Critical Thinking. In: Baron, J. and Sternberg, R. (Eds.), Teaching Thinking Skills. New York: Freeman.

European Union (1995). White Paper on Education and Training--Teaching and Learning--Towards the Learning Society.

Facione, P. A. (1990a). Critical Thinking: a Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. Research Findings and Recommendations ("The Delphi Report"). Washington, D.C.: ERIC [ED315423].

Facione, P.A. (1990b). The California Critical Thinking Skills Test-College Level: Technical Report #1--Experimental Validation and Content Validity. Washington, D.C.: ERIC [ED327549].

Facione, P. A. (1990c). The California Critical Thinking Skills Test--College Level: Technical Report #2--Factors Predictive of CT Skills. Washington, D.C.: ERIC [ED327550].

Facione, P.A. (2001). Test Manual: The Test Of Everyday Reasoning--A Measure of Thinking Skills. Millbrae, CA: Insight Assessment/The California Academic Press.

Facione, P. A., Facione, N. C., Blohm, S. W. & Giancarlo, C. A. (2002). Test Manual: The California Critical Thinking Skills Test (Form A, Form B, Form 2000). Millbrae, CA: Insight Assessment/The California Academic Press.

Friedler, Y., Nachmias, R. & Linn, M. (1990). Learning Scientific Reasoning Skills in Microcomputer Laboratories. Journal of Research in Science Teaching 27(2), 173-191.

Gadzella, B. N., Hogan, L., Masten, W., Stacks J., Stephens, R. & Zascavage, V. (2006). Reliability and Validity of Watson-Glaser Critical Thinking Appraisal-Forms for Different Academic Groups. Journal of Instructional Psychology 33(2), 141-143.

Garnett, J. P. & Tobin, K. G. (1984). Reasoning Patterns of Preservice Elementary and Middle School Science Teachers. Science and Education 68(5), 621-631.

Haldyna, T.M. (1994). Developing and Validating Multiple Choice Test Items. Mahwah, NJ: Lawerence Erlbaum Associates, Publishers.

Jones, E. A., Hoffman, S., Moore, L., Ratcliff, G., Tibbetts, S. & Click, B. (1995). National Assessment of College Student Learning: Identifying the College Graduate Essential Skills in Writing, Speech and Listening, and Critical Thinking. Washington, DC: National Center for Educational Statistics. US Department of Education, Office of Educational Research and Improvement. OERI publication NCES 93-001.

Jungwirth, E. & Dreyfus, A. (1990). Identification and acceptance of a posteriori casual assertions invalidated by faulty enquiry methodology: An international study of curricular expectations and reality. In: Herget, D. (Ed.), More history and philosophy of science in science teaching (pp. 202-11). Tallahassee, FL: Florida State University.

Jungwirth, E. (1987). Avoidance of logical fallacies: A neglected aspect of science education and science-teacher education. Research in Science and Technological Education 5, 43-58.

Krusse, J. & Preseissen, B.A. (1987). A Catalog of Programs for Teaching Thinking. Philadelphia: Research for Better Schools.

Lawson, A. (1982). The Nature of Advanced Reasoning and Science Instruction. Journal of Research in Science Teaching 19(9), 743-759.

Lawson, A. (1985). A Review of Research on Formal Reasoning and Science Teaching. Journal of Research in Science Teaching 22(7), 569-617.

Leech, N. L., Barrett, K. C. & Morgan, G. A. (2005). SPSS for Intermediate Statistics: Use and Interpretation. (2nd ed.). Mahwah, NJ: Lawerence Erlbaum Associates, Publishers.

Lippman, M. (1991). Thinking in Education. Cambridge, UK: Cambridge University Press.

McGuinness, C. & Nisbet, J. (1991). Teaching Thinking in Europe. British Journal of Educational Psychology 51(2), 46-58.

Meadows, S. (1996). The Child as Thinker. London: Routledge.

Morgan, G. A., Leech, N. L., Gloeckner, G. W. & Barrett, K. C. (2004). SPSS for Introductory Statistics: Use and Interpretation. (2nd ed.). Mahwah, NJ: Lawerence Erlbaum Associates, Publishers.

Obed, N. (1997). Investigating the nature of formal reasoning in Chemistry: Testing Lawson's Multiple Hypothesis Theory. Journal of Research in Science Teaching 34(10), 1067-1081.

Paul, R. W., Binker, A. & Weil, D. (1995). Critical Thinking Handbook: K-3rd Grades. Santa Rosa, CA: Foundation for Critical Thinking.

Perkins, D. (1993). Creating a Culture of Thinking. Educational Leadership 51 (3), 98-99.

Siegal, H. (1988). Educating Reason: Rationality, Critical Thinking, and Education. New York, NY: Routledge.

Sternberg, R. (1985a). Teaching Critical Thinking, Part 1: Are We Making Critical Mistakes? Phi Delta Kappan 67(3), 194-198.

Steinberg, R. (1985b). Teaching Critical Thinking, Part 2: Possible Solutions. Phi Delta Kappan 67(4), 277-280.

Steinberg, R. (1987). Teaching Critical Thinking: Eight Easy Ways to Fail before You Begin. Phi Delta Kappan 68(6), 456-459.

(Footnotes)

(1) Translated from the Official Gazette issue B, nr 303/13-03-03 and issue B, nr 304/13-03-03 by members of the Greek National Pedagogic Institution. Available online from: http://www.pi-schools.gr/programs/depps/ index_eng.php [Accessed: 10-5-2007].

(2) Missing items are considered as wrong answers.

Katerina Malamitsa, Ph.D., Faculty of Primary Education, University of Athens. Michael Kasoutas, PhD., Student, Faculty of Primary Education, University of Athens. Greece, email: [e-mail]; Panagiotis Kokkotas, Professor, Faculty of Primary Education, University of Athens.

Correspondence concerning this article should be addressed to Dr. Michael Kasoutas at [e-mail]