A Validity Study of the Kentucky's Performance Based Assessment System with National Merit Scholars and National Merit Commended.This study explored the validity of Kentucky's Performance Based Assessment System with National Merit Scholars and National Merit Commended. A significant percentage of these scholars were judged by the Kentucky Kentucky, state, United States Kentucky (kəntŭk`ē, kĭn–), one of the so-called border states of the S central United States. It is bordered by West Virginia and Virginia (E); Tennessee (S); the Mississippi R. tests as nonmastery in reading, math, science and social studies as well as in their writing portfolio. The results indicated that this assessment system is highly suspect as a valid measure for high achieving scholars. "During the 1980's, testing became a cornerstone cornerstone Ceremonial building block, dated or otherwise inscribed, usually placed in an outer wall of a building to commemorate its dedication. Often the stone is hollowed out to contain newspapers, photographs, or other documents reflecting current customs, with a view to of the education reform movement. Testing burgeoned; the consequences for performance on tests were greatly increased; and assessment increasingly became a means of changing behavior rather than a mechanism for gathering information ... Now the nation stands poised on the brink of yet another wave of test-based reform, and again we appear prepared to undertake it without sufficient quality control" (Dunbar, Koretz and Hoover, 1991, p. 301). Performance assessment is the current favored testing technique for reform (Madaus, 1994). Performance assessment is not a new measurement technique. It was used before the multiple-choice technique, and continues to be used in many disciplines: art, music, technical areas, and athletics athletics or track and field also track-and-field games Variety of sport competitions held on a running track and on the adjacent field. It is the oldest form of organized sports, having been a part of the ancient Olympic Games from c. (Bergen, 1994). However, performance assessment is increasingly being used to measure the entire curriculum, as well as individual schools, for state accountability purposes. Many states are using performance assessment, on a piece-meal basis, or considering its use (O'Neil, 1992). At present, Kentucky is the only state to mandate reform and measure it solely with performance assessment. The Kentucky Education Reform Act of 1990 mandated that the Department of Education develop and administer a performance based test of student achievement. In 1991 the initial Kentucky performance based assessment system was called Kentucky Instructional Results Information System (KIRIS KIRIS Kentucky Instructional Results Information Systems ). KIRIS was created as the major part of a school accountability system which assessed a school's success in attaining Kentucky's educational goals (Academic Expectations). Initially this system included open response questions in reading, math, science, and social studies; written response to performance events; and writing and math portfolios. Based on the student's test performance they were classified into four categories; Novice and Apprentice A person who agrees to work for a specified time in order to learn a trade, craft, or profession in which the employer, traditionally called the master, assents to instruct him or her. (nonmastery) and Proficient pro·fi·cient adj. Having or marked by an advanced degree of competence, as in an art, vocation, profession, or branch of learning. n. An expert; an adept. and Distinguished (mastery). The number of students in each category forms the basis of the school's Accountability Index. The major purpose of the test was to hold schools accountable through a system of rewards and sanctions Sanctions is the plural of sanction. Depending on context, a sanction can be either a punishment or a permission. The word is a contronym. Sanctions involving countries: The General Assembly meets annually in the state capitol building in Frankfort, Kentucky, convening on the first Tuesday after the first Monday renamed the performance assessment system from KIRIS to CATS (Commonwealth Accountability Testing System). The new test will look much like the old one and will continue to use performance based assessment for high stakes High Stakes is a British sitcom starring Richard Wilson that aired in 2001. It was written by Tony Sarchet. The second series remains unaired after the first received a poor reception. decisions. The test was intended to evaluate schools not individual students. Therefore, Kentucky's performance based assessment is used both to drive instruction and for school accountability. "Such a system is not generally used, primarily because of the issues of reliability and validity connected with the development of new forms of testing" (Steffy, 1993, p. 43). "Researchers who study assessment reforms have pointed out a fundamental tension between assessment as an inducement Inducement Electra incited brother, Orestes, to kill their mother and her lover. [Gk. Myth.: Zimmerman, 92; Gk. Lit.: Electra, Orestes] Hezekiah exhorts Judah to stand fast against Assyrians. [O.T. to instructional reform and assessment as a measurement tool" (Koretz, et al., 1996). Purpose of the Study Although KIRIS was initially presented as a measure of school, not individual performance, individual scores were and have continued to be generated and reported to students and parents. School districts have been encouraged to use and have used individual scores for selection and evaluation of students. The Kentucky Department of Education (KDE (K Desktop Environment) A GUI-based user interface primarily for Unix and Linux machines, but also available for Windows and Mac. The source code is freely distributed and is maintained by developers around the world. ) asked schools to put KIRIS scores on high school transcripts. Kentucky universities were also asked to use KIRIS assessment results as part of admission criteria admission criteria the rules for the establishment of comparable groups in any comparison of differences in the performance or responses of the group. The criteria may be permissible age group, the previous productivity, the freedom from disease and so on. . Therefore, the validity Kentucky's performance based assessment is an appropriate and important consideration and should continue to be investigated. An independent evaluation of KIRIS was conducted by The Evaluation Center at Western Michigan University Western Michigan University, at Kalamazoo, Mich.; coeducational; founded in 1903 as Western State Normal School, became accredited in 1927 as a college, gained university status in 1957. which concluded that: "A key step toward improving validity will be to obtain external confirmation, as for example from the ACT, that the accountability index does manifest manifest 1) adj., adv. completely obvious or evident. 2) n. a written list of goods in a shipment. MANIFEST, com. law. A written instrument containing a true account of the cargo of a ship or commercial vessel. 2. an acceptable level of reliability" (KIER 1995 p.66). In three separate studies Strong and Sexton sex·ton n. An employee or officer of a church who is responsible for the care and upkeep of church property and sometimes for ringing bells and digging graves. (1995, 1996, 1997) reported a great deal of disagreement between KIRIS scores in Reading, Math and Science and the corresponding ACT tests. In their sample 52 percent of high school seniors judged as nonmastery on KIRIS Reading scored at the mean or above on the ACT Reading test. Forty-eight percent of those seniors judged as nonmastery on KIRIS Math scored at the mean or above on the ACT Math test. Eighty-nine percent of those seniors judged as nonmastery on KIRIS Science scored at the mean or above on the ACT Science test. Kentucky's performance assessment system has been the subject of review by noted researchers. One of the most significant evaluations of Kentucky's performance assessment was commission by the Kentucky Office of Educational Accountability in 1995. A panel of experts reviewed the measurement quality of the Kentucky system and concluded that further construct validation See validate. validation - The stage in the software life-cycle at the end of the development process where software is evaluated to ensure that it complies with the requirements. evidence was needed to support its various uses and interpretations. (Hambleton, et al., 1995). Rand Rand See Witwatersrand. rand 1 n. See Table at currency. [Afrikaans, after(Witwaters)rand. , a nonprofit A corporation or an association that conducts business for the benefit of the general public without shareholders and without a profit motive. Nonprofits are also called not-for-profit corporations. Nonprofit corporations are created according to state law. institution that helps improve public policy through research and analysis, conducted research on the effects of Kentucky's performance assessment in 1996. Their study concluded that "the evaluation of KIRIS is likely to be a complex and long-term Long-term Three or more years. In the context of accounting, more than 1 year. long-term 1. Of or relating to a gain or loss in the value of a security that has been held over a specific length of time. Compare short-term. process. Validation requires numerous types of information, particularly in the case of assessments such as KIRIS that use innovative performance-assessment formats, serve multiple functions, and are designed to change instruction" (Koretz, at al., 1996). The most recent comprehensive study of the Kentucky system was conducted for the Kentucky Legislative Research Commission (Catterall, et al., 1998) concluded that not enough validation research has yet taken place on KIRIS. They describe the April 1997 draft technical manual as "sketchy in its presentation of validity evidence, and perhaps could be described as a bit self-serving self-serving adj. referring to a question asked of a party to a lawsuit or a statement by that person that serves no purpose and provides no evidence, but only argues or reinforces the legal position of that party. " p. 35, and noted that many of the validity questions should have been answered before applying rewards and sanctions. All of the above researchers question the continued use of KIRIS in a high stakes environment and call for further studies of the validity of KIRIS as an instrument to drive curriculum and instruction as well as evaluate schools and individual students. In addition concerns about the reliability and validity of KIRIS were reported by all Kentucky stakeholders Stakeholders All parties that have an interest, financial or otherwise, in a firm-stockholders, creditors, bondholders, employees, customers, management, the community, and the government. in three statewide surveys conducted by the Kentucky Institute for Education Research (1997a, 1997b, 1997c). These respondents In the context of marketing research, a representative sample drawn from a larger population of people from whom information is collected and used to develop or confirm marketing strategy. included local school board members, school superintendents Noun 1. school superintendent - the superintendent of a school system overseer, superintendent - a person who directs and manages an organization , teachers, principals, parents, and the general public. Description of Study The Office of Education Accountability sent a survey to every public high school in the state. The survey collected data on KIRIS results of the 1995 and 1996 National Merit Scholars and National Merit Commended. One hundred nineteen schools responded. The schools represented all eight regional service centers and the sample was judged to be representative of the entire state. Complete data were collected on 248 subjects. This study was designed to determine how National Merit Scholars, and National Merit Commended students performed on KIRIS. It is widely accepted that those students who earn these distinctions are able to read and write well and apply good thinking skills. Nitko 1997, stated that these were the abilities needed to score highly on the KIRIS. Indeed these exceptional scholars are highly recruited by and often given full scholarships at colleges and universities. This study investigates what percentage of these scholars will score at least at the minimum mastery level on KIRIS. Results Table 1 displays the percentage of 1995 National Merit Scholars and National Merit Commended Students who scored at the mastery and nonmastery levels on the four KIRIS tests and the Writing portfolio. Table 1 Percentage of Scholars Who Scored at Mastery and Non Mastery Levels on 1995 KIRIS Tests
KIRIS Test
Reading Math Science
Measures of
Scholarship N M NM M NM M NM
National Merit
Scholar 94 53 47 56 44 36 64
National Merit
Commended 33 27 73 30 70 15 85
KIRIS Test
Social Writing
Studies Portfolio
Measures of
Scholarship M NM M NM
National Merit
Scholar 53 47 84 16
National Merit
Commended 21 79 72 28
N = Number M = % Mastery NM = % Non Mastery Table 2 displays the percentage of 1996 National Merit Scholars and National Merit Commended Students who scored at the mastery and nonmastery levels on the four KIRIS tests and the Writing portfolio. Table 2 Percentage of Scholars Who Scored at Mastery and Non Mastery Levels on 1996 KIRIS Tests
KIRIS Test
Reading Math Science
Measures of
Scholarship N M NM M NM M NM
National Merit
Scholar 86 56 44 97 3 72 28
National Merit
Commended 34 44 56 97 3 74 26
KIRIS Test
Social Writing
Studies Portfolio
Measures of
Scholarship M NM M NM
National Merit
Scholar 82 18 75 25
National Merit
Commended 74 26 81 19
N = Number M = % Mastery NM = % Non Mastery A significant proportion of these high achieving students are identified by KIRIS as nonmastery in basic academic areas. The 1995 and 1996 scores tend to be fairly consistent in Reading test and Writing Portfolios. However they vary significantly from 1995 to 1996 in the areas of Math, Science and Social Studies. On the 1995 KIRIS Reading tests 47 and 73 percent respectively of National Merit Scholars and Commended were judged nonnastery by KIRIS. On the 1996 KIRIS Reading tests 44 and 56 percent respectively of National Merit Scholars and Commended were judged nonnastery by KIRIS. The question is not: "Why can't Johnny read?" but why can't National Merit Scholars read? These results beg the question Beg the Question is a graphic novel by Bob Fingerman. It chronicles the trials and tribulations of protagonists Rob — a squeamish freelance cartoonist/pornographer — and Sylvia — a beauty salon manager with loftier aspirations — as well as a of what mastery or nonmastery means on the KIRIS Reading test? On the other hand the Math results are mixed. Forty-four and 70 percent, respectively, of the 1995 National Merit Scholars and Commended were judged nonnastery in math by KIRIS. Yet the 1996 sample is closer to what might be anticipated with only three percent of each group being nonmastery. The authors are hard pressed to explain this significant change in KIRIS scores from the 1995 to 1996 samples. Conclusions The results of this study fail to provide evidence of concurrent validity concurrent validity, n the degree to which results from one test agree with results from other, different tests. of the KIRIS assessment for National Merit Scholars and National Merit Commended. The results of this study question the validity of using performance based assessments in making high stake decisions before additional research is conducted. These findings are consistent with the 1995 report conducted by The Evaluation Center at Western Michigan University which concluded that KIRIS does not provide parents with reliable individual level student scores nor does it provide evidence that the Accountability Index is sufficiently reliable and valid to support high stakes decisions. The recommendation of this study was that without such evidence its use for high stakes decision should be discontinued dis·con·tin·ue v. dis·con·tin·ued, dis·con·tin·u·ing, dis·con·tin·ues v.tr. 1. To stop doing or providing (something); end or abandon: . Since the publication of this first independent evaluation of KIRIS, independent researchers have consistently questioned the reliability and validity of KIRIS. This begs the question if a test is not valid for individuals how can it be valid for schools? Kentucky renamed its performance based assessment and increased the variety of functions the test is to serve. Therefore, the validity demands will be even greater for CATS than for KIRIS. Reference Bergen, D. (1994). Authentic performance assessment. Childhood Education, 70(2), 99-102. Catterall, J., Mehrens, W., Ryan, J., Flores Flores, town, Guatemala Flores (flōrəs), town (1990 est. pop. 2,200), capital of Petén department, N Guatemala. Flores was built on an island in the southern part of Lake Petén Itzá and on the site of the , E. & Rubin, P. (1998) Kentucky instructional results Information system: A technical review. Frankfort, KY: Office of Education Accountability, Kentucky General Assembly Dunbar, S., Koretz, D., Hoover, H. (1991). Quality control in the development and use of performance assessment. Applied Measurement in Education, 4, 289-304. Hambleton, R., Jager, R., Koretz, D., Linn linn n. Scots 1. A waterfall. 2. A steep ravine. [Scottish Gaelic linne, pool, waterfall.] , R., Millman, J., & Phillips, S. (1995). Review of the measurement quality of the Kentucky instructional results information system, 1991-1994. Frankfort, KY: Office of Education Accountability, Kentucky General Assembly. Kentucky Institute for Educational Research (1995, January). An Independent Evaluation of the Kentucky Instructional Results Information System. Frankfort, KY Kentucky Institute for Educational Research (1997a, January). 1996 Statewide education reform survey of local school board members. Frankfort, KY Kentucky Institute for Educational Research (1997b, January). 1996 Statewide education reform survey of school superintendents. Frankfort, KY Kentucky Institute for Educational Research (1997c, January). 1996 Statewide education reform survey of teachers, principals, parents and general public. Frankfort, KY Koretz, D., Baron baron Title of nobility, ranking in modern times immediately below a viscount or a count (in countries without viscounts). The wife of a baron is a baroness. Originally, in the early Middle Ages, the term designated a tenant of whatever rank who held a tenure of barony , S., Mitchell Mitchell, city (1990 pop. 13,798), seat of Davison co., SE S.Dak.; inc. 1881. Mitchell is a trade, distribution, and shipping center for a dairy and livestock area. , K. & Stecher, B., (1996) Perceived effects of the Kentucky Instructional results Information System (KIRIS). Institute on Educational Reform. Madaus, G.F. (1994). A technological and historical consideration of equity issues associated with proposals to change the nation's testing policy. Harvard Educational Review The Harvard Educational Review is an interdisciplinary scholarly journal of opinion and research dealing with education, published by the Harvard Education Publishing Group. The journal was founded in 1930 with circulation to policymakers, researchers, administrators, and teachers. , 64(1), 76-91. Nitko, A. (1997, May). A guide to test in Kentucky. Kentucky Institute for Educational Research Frankfort, KY O'Neil, J. (1992). Putting performance assessment to the test. Educational Leadership, 49(8), 14-19. Steffy, B.E. (1993). Top-down-bottom-up: Systemic systemic /sys·tem·ic/ (sis-tem´ik) pertaining to or affecting the body as a whole. sys·tem·ic adj. 1. Of or relating to a system. 2. change in Kentucky. Educational Leadership, 51(1), 42-44. Strong, Sue, & Sexton, L., (1995) Kentucky performance of science: Is it scientific? Eastern Kentucky Educational Review, 18(2), 27-35. Strong, Sue, & Sexton, L., (1996) Kentucky performance of reading: Valid?, Journal of Contemporary Education, XLVII(2) 102-106. Strong, Sue, & Sexton, L., (1997) Kentucky performance of mathematics: Do the numbers add up? Journal of Instructional Psychology 24(3), 202-206. Dr. Sue Strong, Dr. Larry C. Sexton, Faculty, Eastern Kentucky University Student Life The Eastern Kentucky University Office of Student Life works closely with Registered Student Organizations (RSO's), Greek Life, and Thursday Alternative Getaway (TAG). , Richmond. Correspondence concerning this article should be addressed to Dr. Sue Strong, Associate Professor, Counselor Education, Eastern Kentucky University, 4406 Combs, Richmond, Kentucky Richmond is the 6th largest city in Kentucky and the county seat of Madison County. It was named after Richmond, Virginia. In 2004, the city population was 30,008. It is home to Eastern Kentucky University. 40474-3111. |
|
||||||||||||||||||
Printer friendly
Cite/link
Email
Feedback
Reader Opinion