Investigating undergraduate students' attitudes on the use of the networked technology.

Abstract

In most evaluation studies, the question in focus concerns the comparative effectiveness of various types of web based learning systems to traditional ones rather than the innovation of the delivery model itself and the factors, which may contribute to its effectiveness. With this paper we are presenting an evaluation study which follows a specific summative evaluation method attempting to define clearly why the evaluation is being performed, what is being evaluated, when it will be performed, how it will be performed, and what was learned. This stepwise evaluation method is supported by specially developed pre-test and post-test questionnaires, which provide data for both quantitative and qualitative analysis. The focus of this evaluation is on the learning effectiveness of the course and its instructional model as well as the identification of extensions and revisions needed to be made.

1. Introduction

Networked technology is having and will continue to have a profound impact on education around the globe. However, the educational community has a lot to learn regarding how, and in what ways technology can enhance the instructional process. While there is a great amount of writing devoted to research on the impact of technology in education, there are certain gaps in these research efforts which require further investigation [3]. Among these gaps are the following:

   * There is a lack of a theoretical or conceptual framework

   * The potentially different learning styles of students are not taken into
     consideration, relative to the use of particular technologies

   * The feelings and attitudes of students have not been investigated.

This paper presents the findings of an evaluation study about an undergraduate course on "Compilers" offered by the Software Engineering Laboratory of the Electrical and Computer Engineering Department at the National Technical University of Athens, Greece, over a period of two years (i.e. the academic years 1999 and 2000). The main focus of the evaluation study was to identify the learning effectiveness of the use of Internet and World Wide Web to enhance/enrich the traditional instructional delivery mode as well as the identification of extensions and revisions needed to be made to them according to the students' opinions.

The structure of the paper is the following: In section 2, we describe the overview of the instructional design of the Compilers course. Section 3 presents the conceptual framework of the evaluation research. Section 4 deals with the results of the hands-on findings of the evaluation study. Section 5 contains few concluding remarks about the instructional process as well as about the use of WWW.

2. The conceptual framework of the research

The evaluation study followed a specific methodology, called CADMOS-E [7] which is a pre-test and post-test method incorporating some aspects of the illuminative evaluation approach [1, 4]. It is a stepwise method supported by specially developed pre-test and post-test questionnaires, which provide data for both quantitative and qualitative analysis of the impact and learning effectiveness of the technology and the enriched instructional delivery mode. The learning effectiveness was conceptualized as being influenced by a number of variables, such as:

   * Usage of the learning environment

   * Effect of the instructional model to students' learning styles

   * Contribution of the learning resources to the acquisition of knowledge
and skills

   * Effect of the instructional model to the acquisition of knowledge and
skills

   * Quality of the learning resources

   * A comparison of the enriched classroom delivery mode with the traditional
   model of teaching.

These variables are composite and are measured by multiple items, each measuring a slightly different aspect of the main variable.

3. The course on Compilers

The course on Compilers is a one semester undergraduate course in the curriculum of the Department of Electrical and Computer Engineering at National Technical University of Athens, Greece. This course belongs to the set of elective courses that a student must complete successfully during his/her studies. After the successful completion of this course, the students are better able:

   * To present and elaborate on the differences of a compiler, meta-compiler,
   parser and interpreter for modern programming languages

   * To state the techniques and tools that can be applied through the various
   phases of the development of a compiler

   * To follow systematic methodologies for the development of a compiler

   * To construct a good quality compiler using modern tools

An enriched classroom model based on the WWW has been developed for the instructional delivery [http://webct.softlab.ntua.gr]. The term "enriched" signifies that the open and distance learning methods have been used complementary to the traditional ones. Apart from traditional lectures, peer tutorials as well as on-line resources such as a web-based electronic book and on-line tools that facilitated the fulfilment of learning activities had been developed and integrated into the WebCT [9] learning environment, as supplements. So the students are provided with a variety of teaching aids for acquiring knowledge and skills without many constraints in time and place of instructional delivery.

The electronic book (e-book) was specially developed to be user friendly, pedagogically effective and aesthetically attractive. The e-book contains theory, and examples as well as self-assessment (multiple choice) exercises. The structure of the e-book follows the UK's Open University standards for structuring the learning material into blocks and units, while an indexed guided tour was adopted for navigation purposes. Each block consists of learning objectives in order that the student knows well ahead what to expect from the learning material. The student can also practice his/her skills about the development of the intermediate products of the compiler construction life cycle. Specific exercises about an exemplar compiler are given which must be solved with the aid of a customized programming environment that includes an editor with a built-in preview mechanism and a presentation environment of the solutions. The instructional design of the course was based on the belief that new technologies can facilitate the integration of drill & practice pedagogical method along with collaborative and explorative learning methods [6].

4. The Evaluation Study

4.1 Subjects

The total number of postgraduate students who responded to the evaluation study was 26 (3 women and 23 men) out of 47 who successfully completed the course. The drop out rate was 21,6%, that is, 11 students out of the 51 registered did not finish the course. This course is highly demanding and this rate was not surprising. Almost all students (96%) commented that the work required during the course was a lot. All the students (92%) had previous experience in using computers, Internet and the Web, and only five of them had attended a course with similar instructional delivery mode in the past. In terms of the time spent accessing the web-based learning environment, the majority of the students spent approximately 1-2 hours per week. Finally, none of them had any problem in reading material suggested in the bibliography that was written in English.

4.2 Instruments for Data Collection

The study was based upon two kinds of questionnaires that were given to the students. The first kind (pre-test) aimed at identifying the expectations of the learners from the learning effectiveness of the course and its delivery mode. This questionnaire was administered during the first days of the course delivery and not later than the first week, and consisted of 29 questions. The second kind (post-test) aimed to deduce overall judgements and criticism of learning effectiveness. The questions that appeared on the pre-test were also replicated in the post-test in a rephrased form. However, the latter instrument consisted of a wide number of closed-end questions that were used to evaluate in detail specific issues on the quality of the on-line learning material, the enriched classroom delivery mode and the instructional support provided during the course. It also included a section with a number of open-ended questions to supplement the quantitative data. The open-ended section concerned issues related to students' likes and dislikes towards the learning environment, the deficiencies concerning the enriched classroom model, and suggestions for improving the instructional support and the online material.

4.3 Data Analysis

Since the size of the sample was not statistically appropriate for quantitative analysis we made a comparative statistical analysis of the data collected from the "pre-test" and "post-test" questionnaires. The basic statistical analysis depicted the trends of the learners' preferences and dislikes. Moreover, a qualitative content analysis of the second part of the "post-test" questionnaires was performed which helped to elucidate a number of conclusions concerning the preferences and the dislikes of the learners about the quality of the course delivery, the design of the web-based learning material, and the mode of instructional delivery.

4.3.1 Usage of the learning environment

The usage of the learning environment according to the answers of the students is shown in Table 1. See issue's website <http://rapidintellect.com/AEQweb/win01.htm>.

4.3.2 Effect of instructional delivery mode to students' learning styles

Learning styles were measured by a number of variables which reflected the four learning categories identified by [2], "Concrete Sequential", "Abstract Random", "Abstract Sequential", and "Concrete Random". By comparing the answers of students in the pre and post-tests, we can identify whether the instructional delivery mode has affected their learning styles and in what direction. Such comparison can offer us valuable insight on whether the learning strategies incorporated into the instructional delivery mode match with the learners' profiles. For example, if the learning strategies put emphasis on computer-mediated communication and collaboration, it will be success if some students who preferred to work at their own pace and in isolation, change their learning style. The effect of instructional delivery mode to students' learning styles is presented in Table 2. See <http://rapidintellect.com/AEQweb/win01.htm>.

4.3.3 Contribution of the learning resources to the acquisition of knowledge and skills

Another important factor measured during the evaluation study was the contribution of the learning resources to the acquisition of knowledge and skills, both as expected before and as criticised after the completion of the course delivery. The question was: "Did the following resources of the learning environment help the acquisition of knowledge and skills?" Table 3. Contribution of the learning resources to the acquisition of knowledge and skills. See <http://rapidintellect.com/AEQweb/win01.htm>.

4.3.4 Effect of the instructional model to the acquisition of knowledge and skills

Table 4, exhibits the evaluation of effect of the instructional model to the acquisition of knowledge and skills, both before and after the completion of the course. For this specific variable, the post-test questionnaire was more extensive. Table 4a. Evaluation of the effect of the instructional model to the acquisition of knowledge and skills including pre- and post-test questions. Table 4b. Evaluation of the effect of the instructional model to the acquisition of knowledge and skills including only post-test questions. See <http://rapidintellect.com/AEQweb/win01.htm>.

4.3.5 Quality of the learning material

Table 5 shows how the students were evaluated the quality of the learning material. The quality criteria were adapted from the study of [8]. Table 5. Evaluation of the quality of the learning material. See <http://rapidintellect.com/AEQweb/win01.htm>.

4.3.6 Comparison of the new mode of teaching with the conventional one

There were some critical questions where the students were asked to compare the conventional way of teaching against the enriched teaching mode through the WebCT learning environment. Also, the students were asked an overall evaluation

of the teaching mode. Table 6 presents the mean of each measurement. Table 6. Evaluation of the teaching modes. See <http://rapidintellect.com/AEQweb/win01.htm>.

Question: Would you prefer to attend again a course, which combines the conventional way of lecturing with the new mode of teaching in the future?

Answer: Yes: 21 No: 5

Question: Which mode of teaching was more useful?

Answer: Through the WebCT environment: 12 Conventional way of teaching: 14

Conclusions

The students' experience from the usage of the new mode of teaching through the WebCT learning environment was positive but they found the combination of the new mode of teaching with the conventional one as the most ideal. As the qualitative analysis of the answers and comments of the students from the open-ended questions of the post-test questionnaire showed, there were two main reasons explaining the above preference of the students:

   1. The tutors neither raised enough questions for discussion nor answered
   frequently to student queries.

   2. The students preferred more face to face communication.

The advantages and disadvantages that made the students evaluate the system as useful are mentioned below:

Advantages

   * Easy acquisition of information related to the course.

   * Lack of restrictions in time and place in terms of communication with the
   tutors and studying of the learning material.

   * The existence of case studies.

   * More learning material for selective collection.

   * The existence of passed exam papers and their solutions.

Disadvantages

   * There were not many subjects for discussions.

   * The interactivity with the learning resources was limited (they suggested
   interactive tests, FAQs, manuals of lex/bison, etc.).

   * There were not so many links and other sources of learning material.

   * Most of the WebCT's facilities (such as e-mail, forums, etc.) were not
used.

It is evident that the web has become an integration platform for educational applications. The design, constructing and maintenance of such applications are not easy tasks [5]. The evaluation study of the course which was based on an enriched web-based instructional model showed that it is possible to offer a high quality course, which is appropriate for the student body for which it is intended. The upgrade of the learning material and its delivery mode will be focused on the development of more interactive material relevant as well as the development of on-line resources on specialised topics. Great care will be paid to the design of learning activities and the training of tutors on triggering and encouraging students' participation to computer mediated communication and collaboration.

We believe that the development of a web-based instructional system is an evolutionary process, that is, the end product is not released in one "big bang" at the end of the project. So, the ultimate purpose of the summative evaluation process is to identify the quality of the versions of a web-based instructional system, including their impact/effectiveness on learning, as well as to help the developers in the system's expansion and maintenance. The iterative nature of evaluation (i.e. as new versions come along) should assist in making the learning experience more effective since the feedback is used to continuously improve matters.

Our future work will be focused on performing co-relational research that will involves collecting data from a large period (e.g. 4 consecutive academic periods) in order to determine whether, and to what degree, a relationship exists between two or more quantifiable variables of the evaluation framework. One example of a co-relational study might be determining the relationship (correlation) between student satisfaction with the quality of interaction between the instructor and the learner as well as the type of technology used. Moreover, we have already started doing experimental research, which is the only type of research that can truly test hypotheses concerning cause- and-effect relationships. In such an experimental study, we manipulate and test hypotheses in order to find the most significant predicting variables which contribute to the prediction of the learning effectiveness of web-based instructional systems.

Bibliography

[1.] J. Calder (1995). Programme Evaluation and Quality: A Comprehensive Guide to Setting up an Evaluation System, Institute of Educational Technology, Open University, Kogan Page.

[2.] Gregorc, F. (1979). Learning/teaching styles: Their nature and effects. Student learning styles: Diagnosing & prescribing programs, 19-26.

[3.] Institute for Higher Education Policy (1999). What's the Difference: A Review of Contemporary Research on Effectiveness of Distance Learning in Higher Education, [http://www.Ihep.com/PUB.htm]

[4.] Learning Technology Dissemination Initiative (1998), Evaluation Cookbook, ISBN 0952873168, [http://www.icbl.hw.ac.uk/ltdi/cookbook/]

[5.] Lowe, D., and Hall, W. (1999), Hypermedia & the Web: An Engineering Approach, John Wiley & Sons Ltd.

[6.] McCormack, C., Jones, D. (1997). Building a Web-Based Education System. Wiley Computer Publishing.

[7.] S. Retalis, V. Makrakis, and E. Skordalakis (1998). A methodology for evaluating the effectiveness of the use of new technologies in ODL, Proceedings of EDMEDIAEDTELECOM'98, Freiburg, Germany.

[8.] M. Tessmer (1995). Formative Multimedia Evaluation, Training Research Journal, Vol. 1.

[9.] WebCT. (2001). WebCT. [Online] Available: http://www.webct.com/