The common European framework of reference and the assessment of oral English in Spain. (Language Teaching & Learning).

Abstract

The Common European Framework of Reference is a publication of the Council of Europe in the European Year of Languages (2001) that puts forward assessment criteria to set up the basis for the international comparison of objectives and qualifications. This paper discusses how the "Framework" can be adapted for oral assessment in a specific EFL learning context: "Escuelas Oficiales de Idiomas" in Spain. The article develops four rating scales, it modifies its descriptors to grade the progression Spanish students actually make in English, and it includes examples of common mistakes that have been collected from students' exams.

Introduction

The Common European Framework of Reference (2001) is an ambitious publication of the Council of Europe in the European Year of Languages that attempts to define terminology and clarify practices of modern language learning, teaching and testing in order to lay down the basis for international comparison of objectives and qualifications (Framework: 2001: Chapter 2, p. 2).

The core chapters of the document (4-7) describe and classify all aspects and elements involved in linguistic performance, from tasks and strategies to language acquisition and language learning, from errors to curriculum design. These chapters could justify the publication as a sound introduction to the contemporary science of language learning and teaching, but the "Framework" is more than an extended glossary of terms and puts this linguistic analysis into a testing perspective: chapters 8-9 discuss rating scales, levels, descriptors, types of assessment, evaluation criteria ... And at the end, four appendixes grade communicative language competence with 41 scales, in the six broad levels that are described by the Council of Europe: C2 (Mastery), C1 (Effective-Proficiency), B2 (Vantage), B1 (Threshold), A2 (Waystage), and A1 (Breakthrough), which are "higher and lower interpretations of the classic division into basic, intermediate and advanced" (Framework: 2001: Chapter 8, p. 10).

The appendixes are an invaluable tool for teachers around the world who are working to pin down communicative competence and make assessment more objective and transparent, but they do not offer pre-cooked recipes. The "Framework" is meant to be a trigger for discussion, it takes a constuctivist perspective on education (Williams and Burden, 1997: 49-56), and constantly invites its users to reflect and apply such criteria to his/ her teaching situation. For example, the "Framework" describes over 24 categories that could have a bearing on oral assessment, therefore examiners will have to discard many to choose a more manageable number of categories:

   "Received wisdom is that more than four or five categories starts to cause
   cognitive overload and that seven categories is psychologically an upper
   limit" (Framework: 2001: Chapter 9, p. 11)

This article will show how the "Framework" can be adapted to improve assessment criteria in an EFL context such as "Escuelas Oficiales de Idiomas" in Spain. Four rating scales will be discussed whose descriptors have been modified to reflect the progress that is observed in Spanish learners of English.

Escuelas Oficiales de Idiomas in Spain

"Escuelas Oficiales de Idiomas" (E.O.I.) offer language training to adults in general: university students, professionals, upper high-school students, housewives etc. Each school offers several languages structured in five courses and issues two certificates a "Certificado de Ciclo Elemental" after three courses, and a "Certificado de Ciclo Superior" at the end of the five courses, which are also open to external students who wish to have validation of their foreign language competence.

At the E.O.I. n[degrees]l Zaragoza, speaking is the most difficult skill to acquire because of the lack of opportunities to practice outside the school. English oral exams at "Certificado Elemental" and "Certificado Superior" are similar in procedure: students are given two short texts and have some time to choose one and think about what they are going to say; then, they have to read the text aloud, summarize it briefly and talk about the text and the topic of the story; after that, the examiners ask the candidate a few questions related to what he/she has said. The exam proper lasts about 10 minutes, and the main difference between the two levels is that at "Certificado Elemental" the exam deals with topics of everyday life (work, travel, holidays ...) whereas at "Certificado Superior" the candidate is expected to talk about mass media topics of a broader cultural spectrum like "Environment", "Health", "Technology", "Crime"...., which demand a wider range of accurate vocabulary.

During the exam, the teachers interact with the student alternatively, and they take notes under four categories: "Fluency", "Pronunciation", "Grammar" and "Vocabulary". When the exam is over, they discuss their impressions and give a mark based on their judgment, common sense and experience, so the marking system could be described as "impression marking" (Underhill: 1987: 101) or "holistic scoring" (Hughes: 1989: 86-91). Inter-marker reliability is supported by changing the teams of examiners for each oral exam session, and transparency is guaranteed because the candidates have their exams taped. They may also have a subsequent interview with the examiners to find out how they scored on the exam. If they are still not satisfied with the explanations, they can ask the inspectors for a re-assessment of their oral test, and in that case, the examiners have to write a report explaining the grounds for their decision.

Adjusting the "Framework" to Oral Assessment at "Escuelas Oficiales de Idiomas"

The "Framework" offers two complementary advantages for the holistic marking system at the English department: first of all it provides examiners with a common, well-defined metalanguage to put into words subtle oral performance analyses which go beyond broad subjective impressions like "incomprehensible accent", "not fluent enough" or "didn't really say much". This common language can also help examiners to negotiate a mark more rationally, give detailed feedback to learners and, in short, assess oral performance more objectively (Framework: 2001: Chapter 9, p. 7).

Secondly, the large number of scales in the "Framework" appendixes can draw attention to discursive aspects of oral performance which are missing in the four categories that the examiners have traditionally focused upon. Therefore, in this article, a new category "Command" has been added to include the discursive aspects of oral speech performance as well as the confidence and autonomy that the candidate shows to deliver speech. This article has also grouped "Grammar" and "Vocabulary" together because the assessment criteria strike a healthy balance between "discursive categories", and "accuracy categories". Discursive categories are "Fluency" and "Command". These categories look at pragmatic or communicative aspects of oral performance that are bordering Rhetoric and their descriptors can be assessed in a continuum with "more" or "less". "Accuracy categories" like "Pronunciation" and "Grammar and Vocabulary" look at particular aspects of linguistic performance and their descriptors can be usually assessed as "right" or "wrong".

The descriptors in these four rating scales have not always been transcribed literally from the "Framework"; there are two major variations across all scales. The first one is that more sub-levels have been added to the six broad levels of the Council of Europe (Framework: 2001: Chapter 8, p. 10- 11) to meet the need to describe a "pass" and a "fail" level in the five courses. The contrast in students' performance between previously planned speech and spontaneous speech is instrumental in defining several of the "fail" sub-levels -which is consistent with the research carried out by Skehan (1998: 108-112) about the influence of planning conditions on task performance. Secondly, this adaptation for the EOI n[degrees] 1 Zaragoza, differs from the Framework because there is a mixture of positive and negative wording in the scales: although it is true that positively worded descriptors serve as proficiency objectives whose achievement is rewarding and motivating for the learner (Framework: 2001: chapter 8, p. 7), it is also true that mistakes and linguistic limitations should also be tactfully pointed out in assessment reports as they are often inevitable to back up a negative evaluation before a student who has a very high opinion of him/herself. Besides, correcting learner's mistakes is an everyday practice when foreign languages are principally learnt in the classroom, so students themselves expect that kind of feedback from examiners.

The descriptors in the scales of the "Framework" have been adapted to EOI context in various degrees (the full scales can be found on the web page of the journal). The Fluency scale transcribes the descriptors from its homonymous category in the Framework. The scope of this scale is the flow of speech, i.e., whether the candidate feels at ease when he/she is making the speech or struggles to put words or sentences together. Examiners will have to pay attention to the characteristics of speech delivery that in classical Rhetoric were dealt with in the "action" stage of constructing speech (Spang: 1979): speech tempo, hesitation, false starts, "cul-de sacs", repetition, periphrasis, silence ... The adaptation of the Fluency scale for the EOI n[degrees] 1, Zaragoza, adds a few lines to describe the evolution in the acquisition of rhythmic patterns in Spanish learners' speech, the assumption being that prosodic features like sentences stress and the use of weak forms, the rhythmic grouping of words or the delivery of a speech word by word, are more objective evidence of fluency than intuitive concepts like, "ease of expression", or "tempo" that are used in the "Framework". In the correspondence of scales, the pass mark in Certificado Elemental would be equivalent to B1, and would be achieved when the candidate can tell a story although some sentences are delivered word by word and there is pausing. The pass mark in the Certificado Superior would be equivalent to the sub-level described in the Framework between C1 and B2: the student can communicate fluently and spontaneously even in longer stretches of speech.

The Pronunciation scale has been rewritten to capture the peculiar progression that is perceived in most Spanish students of English, as well as the most common errors that are noticeable along that progress. The pass mark in the Certificado Superior has been lowered to B2, but this is a very tentative approximation because the Framework describes these levels with very little detail and may set levels far too high for a classroom learning context, for example, there is no indication of a foreign language realization above B 1 level, when it is well-known that Spanish bilingual speakers in the US, keep a noticeable accent even though their command and linguistic accuracy may be as good as a native speaker's. The scaling that is put forward here describes Ciclo Elemental as the level where students start to articulate consonants accurately even at the end of a word (dog, kid, visit, like) and begin to pronounce both consonants in easy clusters distinctly (hotels, perhaps, works), they group words in rhythmic patterns, and they start using schwa or some other central vowels (in words like "vegetables", "comfortable", in weak forms etc.). The reason for this choice is based on the assumption that Spanish is a language that articulates syllables around five, very clear, extreme vowels, whereas English is a language with central, obscure vowels in which articulation often means accuracy in the realization of consonant phonemes. This has a prosodic consequence in connected speech, the grouping of words in rhythmic patterns (especially in British English, Gimson: 1980: 258-260), which facilitates consonant articulation without vowels. In Ciclo Superior, the acquisition of all the different English phonemes is undertaken (/??/ /??/ /d ??/ /s/ /z/ /w/ /j/), as well as the most difficult consonant clusters (effect, lasts, quite good, great care ...), but errors or approximations such as the introduction of a prop /e/ to pronounce liquid /s/, the pronunciation of final /m/ as /n/, or the reduction of the diphthong in "old" to a short vowel- are still made by students who barely pass the Ciclo Superior level especially when their attention is partially engaged in other cognitive processes, like forward planning.

The Command scale is a condensed mixture of the descriptors in "Overall Spoken Production", "Sustained Monologue (describing experiences)", "Sustained Monologue (putting a case in a debate)", and "Interviewing and Being Interviewed". In this category, there is little modification of the descriptors; just a few references have been added to the ability of the student to respond confidently, extensively and spontaneously to the interlocutors. Here teachers will analyze the complexity of the candidate's speech from a discursive or rhetorical point of view (Spang: 1979): whether the student can produce simple sentences, can tell simple stories, can tell stories with detailed descriptions, or can defend views with arguments and examples in a debate. Examiners will also include their observations about confidence and autonomy: if the student shows initiative to expand points or, on the contrary, depends on the text or on the prompts from the examiners to speak. In the correspondence of scales, the pass mark in Certificado Elemental would be equivalent to B1, when the student can tell a straight forward narrative or a simple description on familiar subjects, whereas the pass mark in Certificado Superior would be equivalent to the sub-level described in the Framework between C1 and B2, that is, when the speaker is confident enough to give full descriptions, to tell complex stories and to defend points of view in a discussion.

The Grammar and Vocabulary scale transcribes quite faithfully the descriptors in "Grammatical Accuracy", "Vocabulary Range" and "Vocabulary Control", but it also adds some examples of common mistakes that have been collected from students' oral performance at each level. For example, in Ciclo Elemental learners make more errors with the forms of tenses and modals, word order, the use of Spanish words, infinitives of purpose, gerunds and infinitives, comparatives, "broken syntax" (omission of prepositions and particles), or they even confuse the verbs "to have" and "to be" in simple expressions; whereas in Ciclo Superior there are usually mistakes with connectors, modal verbs, wrong words, "other" and "another", correlation of past and present tenses, there are examples of unclear expression when the point is missing in a tangle of unfinished sentences and periphrases ... The pass level at Certificado Elemental corresponds to B1, whereas the level at Certificado Superior corresponds to the sub-level between C1 and B2 in "Grammatical Accuracy"; in "Vocabulary Range", "Grammar Accuracy" and "Vocabulary Control" the C1 level is equivalent to a 7-8 (B) mark in Certificado Superior.

Critics may say that the adaptations proposed here do not respect the boundaries established by the four categories, that, for example, in the Fluency scale there are references to rhythm which is also discussed in Pronunciation, but this is often unavoidable because of the "halo effect" that is described by Hughes (1989: 94) whenever several aspects or oral performance are judged independently of the others in analytic scoring. What really matters is that the scales have horizontal coherence, that is, that the descriptors at one level are consistent with the descriptors in other scales at the same level.

Conclusion.

The Common European Framework of Reference, and particularly its scales, have proved to be an invaluable document to provide the metalanguage that can make impressionistic assessment of oral exams more rational, objective and transparent.

In the case of an oral exam in Spain which stems from a text and includes tasks such as reading aloud, summarizing, making a short argumentative, narrative or descriptive speech and interacting in a conversation, the range of scales in the Framework has firstly drawn attention to a gap in the assessment criteria that needs to be fulfilled: the overwhelming weight of accuracy scales (Grammar, Vocabulary and Pronunciation) that examiners have used traditionally at the EOI n[degrees] 1, Zaragoza; this imbalance can be redressed by adopting the Fluency scale in the Framework and by blending four other scales into one called "Command", which describes the confidence the candidate displays to make different types of discourse. Other adaptations of the Framework have been made to fit that specific examining situation, for example, the Pronunciation scale has been rewritten completely to describe more closely the progression of Spanish EFL students, and the number of descriptors has increased to describe "fail" sub-levels, which, unlike those in the Framework, show what the candidates has not been able to achieve. In any case, the ideas in the Framework have always been the catalyst that has stirred reflection and debate, which can eventually lead to the improvement of oral exam criteria.

Other teachers and examining institutions around the world who seriously want to develop more sophisticated methods of evaluation in the 21st century, will surely find in the Framework a thorough and comprehensive reference point to shed light on their endeavor.

References

The Common European Framework of Reference. (2001). http://culture.coe.fr/lang/eng/eedu2.4.htm

Gimson, A.C. (1980). An Introduction to the Pronunciation of English. London: Edward Arnold.

Hughes, A. Testing for Language Teachers. (1989). Cambridge: Cambridge University Press.

Skehan, P. (1998). A Cognitive Approach to Language Learning. Oxford: Oxford University Press.

Spang, K. (1979). Fundamentos de retorica. Pamplona: Ediciones de la Universidad de Navarra.

Underhill, N. (1987). Testing Spoken Language. Cambridge: Cambridge University Press.

Williams, M. and R. Burden. (1997). Psychology for Language Teachers. A Social Constructivist Approach. Cambridge: Cambridge University Press.

J. Ignacio Bermejo is a teacher of English and the vice-principal in charge of the Self-Access Center. His research interests include language learning through the media, assessment, autonomous learning, psycholinguistics and cultural development through language learning.