The use of assessment in improving technology-based instruction programs.

A college-based teacher preparation program undertook a grant-supported evaluation of its' curriculum focused on preparing teacher candidates for integrating technology into instruction. The project employed a pre posttest design, including the use of the 54-item Educator's Knowledge and Implementation of Technology instrument (EKIT), which provides information regarding technology-related capabilities summarized in five area subscores and a total score. Results of the study substantiated the usefulness of such instrumentation in a pre-posttest design for evaluating program impact on students, and for prioritizing areas for continuous program improvement based on low achievement and unsustained growth. Conclusions clearly establish the power of the pre-posttest design for the evaluation and continuous improvement of teacher training programs.

**********

Educators struggle with two demands that cause them to lose sleep (Blasik, 2002a, 2002b; Lewis 2002; Shaha, 2002). First is the need to prove that their programs are effective by validating them based on results, and second is the need for continuous program improvement. Sources of educational funding come with the program validation requirements focused on tangible student outcomes. In the age of increasing accountability, program funding and continuity are accompanied by expectations of proof that offerings are beneficial for student learning and meet objectives and requirements.

The second demand centers on the need to continuously improve instruction and the impacts it achieves. Continuous improvement reflects a clear focus on identifying where things need to improve, followed by a systematic approach to implementing program changes designed to remediate prioritized weaknesses and thereafter measure impact (Arcaro, 1995; Brown, 2001; Boulmetis, 2000; Ross, 1993). Ongoing program improvement is integral to any project with an evaluation component or that seeks to achieve lasting success (Quinones & Kirshstein, 1998; Smith, 2002). The overall objective is to achieve ever-higher levels in tangible measures of educational success.

Assessment, well designed and executed, helps educators resolve the demands for validation and continuous improvement (c.f. Baldrige, 2002; Brown, 2001; Walton, 1986). Through appropriate assessment approaches, educators can make goals and objectives tangible and evaluate whether they have been achieved and to what degree (Daniels, 2002; Shaha, 1997; Smith, 2002). Educators can then identify areas of success and strength, and isolate and prioritize areas for improvement. Clearly validation and improvement are best achieved when outcomes and desired results are clearly identified translated into plans, and then converted into instruments designed to gather the requisite information regarding success (Arcaro, 1995; Baldrige, 2002; Quinones, 1998; Stevens, 2001). To show gains in performance and program improvement, organizations must measure impacts and outcomes, and critically examine the results to achieve excellence (Blasik, 2002a, 2002b; Daniels, 2002; Shaha, 1997).

Increases in accountability and the demand for continuous improvement have also affected programs focused on preparing teachers to better use technology and incorporate it into instruction. In 1999, Utah Valley State College (UVSC) received grant funding from the U.S. Department of Education through Preparing Tomorrow's Teachers to Use Technology (PT3). The application required a substantial evaluation component. UVSC's evaluation plan included a quantitative pre-posttest design utilizing a number of assessment tools to measure the impacts of technology instruction on teacher candidates. Findings from the resulting assessment tools were used to determine what was learned and retained, and then to identify and implement program improvements focused on enhancing learning outcomes. The results of the assessments were designed and leveraged to improve the teacher education program (Farnsworth, 2002).

The evaluation plan for the PT3 capacity building grant purposely embedded the fundamentals of continuous improvement within a well conceptualized measurement approach. The objective was to understand and improve the technology focused instruction provided to teacher candidates (Farnsworth, 2002). Information gained from the assessment tools used led to a better understanding of the effects of teaching technology to teacher candidates and to achieving and sustaining college based program improvements. Execution of the evaluation plan provided program designers at UVSC with data that led them to raise the capabilities of their program and student beneficiaries to the next level. The study reported illustrates the impact of an approach where "assessment was for learning not only of learning" (Stiggens, 2002).

Method

The PT3 capacity building grant was designed to build the process for using assessments and subsequent analysis for program improvement in two areas: 1) determining the technology knowledge and skills of teacher candidates; and 2) determining the performance level of these skills and their persistence over time as a result of college-based classroom instruction. Teacher candidates took the Educator's Knowledge and Implementation of Technology (EKIT) as a means for assessing their self perception of technology-based skills and knowledge of implementing technology in instructional design and curriculum (see Appendix A). The EKIT assessment tool was adapted from the survey tool designed by the Utah Educator's Network (UEN).

The EKIT was also critical for validating the efficacy of teacher preparation curriculum in enhancing the technology skill levels of teacher candidates. Scores on the EKIT provide insights into specific 54 item based areas of competency and achievement, as well as cumulative competency as subscores in particular areas of technology skills and utilization. The items within the assessment tool are grouped into subscores as follows:

A. Instructional Design: Items 1-30, with the cumulative subscore labeled SubDesgn.

B. Supporting Instruction: Items 31-35, with the cumulative subscore labeled SubSupt.

C. Internet and Instruction: Items 36-45, with the cumulative subscore labeled SubIntnt.

D. Classroom Management: Items 46-49, with the cumulative subscore labeled SubClsrm.

E. Basic Teacher Computer Skills: Items 50-54, with the cumulative subscore labeled SubSkill.

Two cohorts of undergraduate education students, teacher candidates, completed the EKIT at the start and end of the 2000-2001 academic year.

* The Junior cohort (n=62) completed the EKIT prior to their coursework on technology skills and incorporating technology in classroom instruction for elementary students, and again at the end of the academic year and the close of that instruction.

* The Senior cohort (n=51) received the same coursework on technology skills and incorporating technology during their corresponding junior year. They completed the EKIT at the start and end of their fourth year in the education preparation program, during which time no additional instruction on technology integration took place.

Results

Figures 1 and 2 provide the graphic depictions of the results from the pre and posttest occasions. There are three areas of importance within the graphs:

[FIGURES 1-2 OMITTED]

1. The Total or cumulative scores are shown as the final, or right-most dots on the graphs.

2. The subscores for each major portion or content area are shown as the last five dots prior to the Total.

3. Individual item scores for each group are shown within the remainder of the graph, following the sequence within the instrument.

Figure 1 illustrates pre-test (i.e. pre-instructional) results for juniors, and results at the start of the academic year for seniors that experienced the instruction during the previous year. The seniors, having already experienced the instruction, were significantly better prepared than their entering junior counterparts on the pre test (F=212.52, df=1, 100, p=.001). While this seems intuitively obvious, it is only because the same assessment instrument was applied at the start and end--a pre-post design--of each academic year that such conclusions can be drawn.

Figure 2 illustrates the posttest, or post-instruction scores for juniors and the end of year scores for seniors that did not experience any additional instruction during that academic year.

Program Impact. A first obvious conclusion from the graphs is that the program's impact on juniors was significant as evidenced by contrasting their pre (Figure 1) and post (Figure 2) total scores (F=141.82, df= 1, 120, p=.001). The benefits of pre and post assessment are further illustrated by examining the change in assessment results among seniors, wherein seniors suffered a statistically significant loss in total scores between the pre (Figure 1) and posttest (Figure 2) occasions (F=10.06, df=1,89,p=.002).

Program Weaknesses. The Total posttest scores for juniors and seniors are not significantly different (see Figure 2). However there are some interesting differences between the groups on subtest scores. For example, the post subtest scores for Classroom Management Software (SubClsrm) were significantly lower for juniors than for seniors (F=24.89, df=1, 108, p=.001). This can be further understood by glancing to the left on Figure 2 and noting the lower scores on each of the four items related to classroom management software. Upon further examination of the curriculum offered to the juniors, it was validated that instruction on classroom management software was diminished from what was offered to the seniors during the previous year.

Often, examination of cumulative subscores alone can mask important detail. For example, subtest scores for Basic Teacher Computer Skills were not significantly different between juniors and seniors on the posttest as shown in Figure 2. However, further examination revealed that there are significantly lower scores on two items related to instruction that juniors received in two important areas, including spreadsheet programs (F=13.38, df=1, 109, p=.001) and database software (F=6.42, df=1, 108, p=.013). Curriculum for seniors included these two factors in the previous year, but juniors did not benefit from that same instruction.

Sustaining Program Impact. Analysis of the subtest scores led to another set of conclusions. Juniors and seniors finished on equal ground for some subtest scores (see Figure 2). For example, the posttest scores for Supporting Instruction (SubSuprt) and for Internet and Instruction (SubIntnt) showed no significant difference between juniors or seniors. This indicates that instruction the juniors received put them on equal ground with seniors who received instruction the prior year, and that gains seniors experienced from the prior year were sustained.

In contrast subtest scores for Instructional Design (SubDesgn) indicated that seniors lost capabilities between years (F=19.26, df=1, 89, p=.001). Additionally, nearly the entire range of items within the Instructional Design subscore in Figure 2 showed juniors scoring higher than their senior counterparts instructed the year before (F=8.76, df=1, 109, p=.004). Therefore, not only did the juniors show better results for the Instructional Design, but also seniors declined from where they were at the beginning of that academic year in this particular area. Two questions in the Instructional Design, Items 7 (TCLesn: I write technology-based lesson plans and curricula) and 8 (TCPlan: I plan technology-based lessons to teach technology skills) are examples of specific areas in which seniors significantly declined (F=9.40, df=1, 89, p=.003; F=21.39, df=1, 89, p=.001; respectively).

Discussion and Conclusions

Several important conclusions can be drawn regarding the technology-focused college-based instruction provided to teacher candidates:

* The instruction benefited teacher candidates, as evidenced in pre versus posttest scores.

* Lack of reinforcing instruction during the senior year led to an erosion in some technology skills, while other skills were sustained without retrenchment.

* Analyses of subtest and item-level data yielded detailed guidance for specific areas of instructional improvement.

More important than conclusions about the program, however, are those that were gained regarding the assessment and validation approach employed. The depth and kind of program-level understanding illustrated in this study cannot be achieved without adhering to the principles and processes embodied in the assessment approach implemented: the repeated, disciplined use of a single assessment tool fully aligned with the outcome objectives of the instructional program, all within a pre and posttest design. This approach was fundamental to assessing candidate learning, validating program success and prioritizing areas for improvement. The evaluation component of the planning process for the PT3 grant established the foundation for leveraging the principles of quality and continuous improvement.

The UVSC education department was able to recognize how well they met the goals and objectives they set for their curriculum. Then the department was able to isolate and prioritize areas for improvement and quantify the impact of improvements implemented. The findings of this study establish the value of assessment and evaluation based on the consistent use of objectives-focused tools within a pre-posttest design, for validating program efficacy and achieving continuous improvement.

Appendix A

E-KIT: Educator's Knowledge and Implementation of Technology

Name --
Date --
Institution --
Grade taught --

Please circle the number that best represents your response to each
item.

    Instructional Design                          Never          Always

 1. My students use technology-based
    presentations for classroom projects.         0   1   2   3   4   5

 2. My students create technology-based
    presentations.                                0   1   2   3   4   5

 3. I use technology terminology when
    instructing students.                         0   1   2   3   4   5

 4. I instruct students from the front of the
    class without the aid of technology.          0   1   2   3   4   5

 5. I use technology to supplement
    instruction.                                  0   1   2   3   4   5

 6. I use multiple delivery methods, including
    multimedia, simulation, and computer video
    presentation.                                 0   1   2   3   4   5

 7. I write technology-based lesson plans and
    curricula.                                    0   1   2   3   4   5

 8. I plan technology-based lesson to teach
    technology skills.                            0   1   2   3   4   5

 9. I plan opportunities for technology to
    support my regular classroom curriculum.      0   1   2   3   4   5

10. I plan student experiences to include
    technology integrated projects.               0   1   2   3   4   5

11. I search for and review software for
    potential student use.                        0   1   2   3   4   5

12. I make assignments where students need
    to use various types of applications such
    as word processing, database, spreadsheet,
    drill and practice, etc. where appropriate.   0   1   2   3   4   5

13. My students use software for creating
    their own products as part of my
    curriculum, such as HyperStudio,
    LinkWay, and mPower.                          0   1   2   3   4   5

14. As part of assignments, my students use
    communications software such as e-mail and
    Internet browsers.                            0   1   2   3   4   5

15. I have a formal strategy for assessing
    student technology products.                  0   1   2   3   4   5

16. I measure and record data on students'
    attitudes and perceptions about
    technology-based activities.                  0   1   2   3   4   5

17. I measure and record student progress on
    technology skills.                            0   1   2   3   4   5

18. I use records of student progress to
    direct my instruction.                        0   1   2   3   4   5

19. My students have access to my evaluation
    rubrics.                                      0   1   2   3   4   5

20. I know when a student is having
    difficulty with technology.                   0   1   2   3   4   5

21. I have the skills to remediate student's
    technology skill deficiencies.                0   1   2   3   4   5

22. I monitor students use of technology for
    producing work.                               0   1   2   3   4   5

23. Each student is involved in setting
    personal goals technology growth.             0   1   2   3   4   5

24. I take students to the lab or allow
    students to use in-room equipment where
    technology is NOT tied directly to what's
    being taught in the classroom curriculum.     0   1   2   3   4   5

25. I modify lesson plans to include
    technology.                                   0   1   2   3   4   5

26. I enrich classroom instruction with
    technology activities.                        0   1   2   3   4   5

27. Student-used software in my curriculum is
    drill and practice.                           0   1   2   3   4   5

28. I actively look for ideas where technology
    compliments classroom instruction.            0   1   2   3   4   5

29. I use drill and practice as a way to
    effectively involve students in technology.   0   1   2   3   4   5

30. I have broad technology-based activity
    experiences to draw from.                     0   1   2   3   4   5

Supporting Instruction

31. I have a system for filing and retrieving
    technology-based resources for curriculum
    development and enhancement.                  0   1   2   3   4   5

32. I collect and use online materials for
    curriculum development.                       0   1   2   3   4   5

33. I collaborate with others to gather ideas
    and resources for integrating technology      0   1   2   3   4   5
    into my curriculum.

34. I discuss educational technology and
    curriculum with my peers.                     0   1   2   3   4   5

35. I participate in technology professional
    development                                   0   1   2   3   4   5

Internet and Instruction

36. I preview Internet resources before
    including them in classroom instruction.      0   1   2   3   4   5

37. I have an organized process in place to
    assist students in their search for
    Internet-based information.                   0   1   2   3   4   5

38. I have collected and organized a set of
    educationally appropriate bookmarks for
    curriculum development.                       0   1   2   3   4   5

39. I teach students how to evaluate sources of
    online Information for use in assignments.    0   1   2   3   4   5

40. I rate Internet-based information for its
    age-appropriateness.                          0   1   2   3   4   5

41. I rate Internet-based information for
    up-to-date content.                               0   1   2   3   4

42. I rate Internet-based information for basic
    potential bias by the authors.                0   1   2   3   4   5

43. I use the Internet as a source of
    information for curriculum and instruction.   0   1   2   3   4   5

44. I query the Internet for educational          0   1   2   3   4   5
    information.

45. I teach students how to conduct searches on
    the Internet for completing assignments.      0   1   2   3   4   5

Classroom Management

46. I use classroom management software for
    grading software.                             0   1   2   3   4   5

47. I use classroom management software for
    automated attendance.                         0   1   2   3   4   5

48. I use classroom management software for
    progress reporting.                           0   1   2   3   4   5

49. I use classroom management software for
    student information.                          0   1   2   3   4   5

Basic Teacher Computer Skills

Rate your skill level in the following categories:

                                                  None           Expert

50. Word processor                                0   1   2   3   4   5

51. Spreadsheet programs                          0   1   2   3   4   5

52. Multimedia/presentation software              0   1   2   3   4   5

53. Database software                             0   1   2   3   4   5

54. Internet                                      0   1   2   3   4   5

References

Arcaro, J. S. (1995). The Baldrige Award for Education: How to measure and document quality improvement. Delray Beach, FL: St. Lucie Press.

Baldrige (2002). Malcolm Baldrige National Quality Award 2002: Education Criteria for Performance Excellence. National Institutes for Standards and Technology. Gaithersburg, MD.

Blasik, K., Sutton, C.O., Lewis, V.K., Shaha, S.H., et al. (2002). Year-round calendar schools in the Hallendale Innovation Zone: Evaluation Report. Prepared for the School Board of Broward County, FL.

Blasik, K., Sutton, C.O., Lewis, V.K., Shaha, S.H., Green, M. (2002b). Alternative to External Suspension Evaluation. Prepared for the School Board of Broward County, FL.

Boulmetis, J., & Dutwin, P. (2000). The ABCs of evaluation: Timeless techniques for program and project managers. San Francisco: Jossey-Bass.

Brown, M. G. (2001). Baldrige Award Winning Quality: How to interpret the Baldrige Award criteria for excellence, 11th edition. Milwaukee, WI: ASQ.

Daniels, S.E. (2002). First to the top in becoming the first to get Baldrige education awards, three institutions demonstrate strategies that can work anywhere. Quality Progress, 38 (5), 41-53.

Farnsworth, B.J., Shaha, S.H., Bahr, D.L., Lewis, V.K., & Benson, L.F. (2002). Preparing Tomorrow's Teachers to use Technology: Learning and Attitudinal Impacts on Elementary Students. Instructional Technologies. (In press)

Lewis, V.K. and Shaha, S.H. (2002) Year Round Calendar: Evaluation Project Report, Broward County Public Schools. Prepared for the School Board of Broward County, FL.

Quinones S. & Kirshstein R. (1998). An educator's guide to evaluating the use of technology in schools and classrooms. Prepared for: U. S. Department of Education, Office of Educational Research and Improvement.

Ross, J.E. (1993). Total quality management: Text, cases and readings. Delray Beach, FL: S. Lucie.

Shaha, S.H. and Sudweeks, R. (1997). Assessing Learning Outcomes: Provo, UT: Brigham Young University.

Shaha, S.H. and Lewis, V.K. (2002) Alternatives to External Suspension Program: Evaluation Project Report. Prepared for the School Board of Broward County, FL.

Smith, M.F. (2001). Evaluation: Preview of the Future #2. American Journal of Evaluation, 22 (3), 281-300.

Stevens, D.W. (2001). 21st Century, Accountability: Perkins III and WIA Information Paper 1002. Information Synthesis Project. National Dissemination Center for Career and Technical Education. College of Education, Ohio State University. Columbus, OH.

Stiggens, R. (2002, March 13). Assessment for Learning. Education Week.

Walton, M. (1986). The Deming Management Method. Putnam Publishing Group. New York, NY.

Wittrock, M. C. (1977). Learning and Instruction. McCutchan Publishing Corporation: Berkeley, CA.

Valerie K. Lewis, Grant Researcher; Steven H. Shaha, Statistics Consultant; Briant J. Farnsworth, Dean, School of Education; Linda F. Benson, Chair of Elementary Education; Damon L. Bahr Assistant Professor, Utah Valley State College

Correspondence concerning this article should be addressed to Valerie K. Lewis, Director, Performance Assessment and Evaluation, Performance Learning Systems, Inc. 2174 New Horizon Drive, Sandy, Utah 84093; email: [e-mail]