An evaluation of short-term distributed online learning events.

The purpose of this study was to evaluate the effectiveness of short-term distributed online training events using an adapted version of the compressed evaluation form developed by Wisher and Curnow (1998). Evaluating online distributed training events provides insight into course effectiveness, the contribution of prior knowledge to learning, and participants' reaction to the technology. The adapted compressed survey form was found to be a valid and reliable instrument. Participants in the training events increased their knowledge. Furthermore, no relation between prior knowledge and self-reported learning was found. Lastly, participants reported a favorable rating of the technology. Immediate feedback, course relevance and overall course effectiveness were found to be determinants in the learning variable. In conclusion, short-term distributed online training events are effective methods of training Department of Defense employees and military personnel.

**********

Introduction

U.S. Secretary of Defense, William Cohen, said, "Innovative uses of information, computing, and network technologies will provide the opportunities to make the learning process better, faster, and more efficient, while reducing personnel tempo and improving readiness" (2001).

Past research in computer-based learning suggests a learning effect size as much as 0.50 standard deviations when multimedia is employed (Kulik & Kulik, 1991; Melmed, 1995; Fletcher, 2001). In addition, instructional time is reportedly reduced by an average of 30% compared to classroom instruction (Fletcher, 2001).

The Department of Defense (DoD) training doctrine has accepted technology enhanced distance learning as playing a principle role in the future of military training. Wisher (1999) supports this notion with the Army plan to convert 500 courses in the Total Army Distance Learning Plan in the 21st Century. Most available studies involve learning events that occur over an extended period of time (Wisher & Curnow, 1998; Wisher, Champagne, Pawluk, Eaton, Thornton & Curnow, 1999). Military training contains many short-term training events that occur in one day or less.

To evaluate short-term, distance training events, Wisher and Curnow (1998) developed a compressed evaluation form. The form was created to evaluate training events delivered over one-way video and two-way audio connections via satellite links.

While Wisher and Curnow (1998) provided evidence for the effectiveness of this delivery method, minimal research has been directed towards short-term distributed training events delivered via the Internet. Distributed learning can be defined as a delivery method of distance instruction where the student is separated from an instructor by both time and distance (Hanson, Maushak, Schlosser, Anderson, Sorensen & Simonson, 1996; Hedberg, Brown, & Arrighi, 1997; Belanger & Jordan, 2000). A key component of distributed learning is the feasibility to train anytime at anyplace (Wisher, Champagne et al., 1999; Bonk & Wisher, 2000; ADL, 2001). By adapting the compressed evaluation form for short-term online training events, program administrators, instructional designers, and course developers hope to be able to identify effective practices in short-term distributed online training events.

The purpose of this study is to evaluate the effectiveness of short-term distributed online training events using an adapted version of the compressed evaluation form developed by Wisher and Curnow (1998). Of interest are the following questions pertaining to short-term distributed online training events:

1. How did students assess their knowledge acquisition?

2. How does prior course knowledge relate to self-assessed learning outcomes for students?

3. What were students' reactions to the technology?

Background

Distance education is an educational process of formal or informal instruction using print or electronic communications media where instructors and students are separated. (McIsaac & Gunawardena, 1996; Belanger & Jordan, 2000; Yates & Tilson, 2000).

Separation, composed of two elements time and place, is a fundamental characteristic of distance education (Simonson, Smaldino, Albright, & Zvacek, 2000). Four combinations of time and place exist for an educational program. These combinations also known as the Coldeway quadrants include same-time, same-place (ST-SP), different-time, same-place (DT-SP), same-time, different-place (ST-DP), and different-time, different-place(DT-DP) (Simonson, Schlosser, & Hanson, 1999). Traditional classroom learning takes place in the ST-SP quadrant while distance education exists in the remaining three quadrants.

Distance education is not a static field. The evolution of technology has created new theories of distance education. For example, equivalency theory in distance education purports that distance education should be equivalent to the classroom experience (Simonson, Schlosser et al., 1999). The distributed learning theory advocates the use of individualized case-based instruction where students learn through doing (Dede, 1996). Distributed learning technology provides innovative instructional strategies like gaming and simulation activities.

Effectiveness of Distance Learning Using Computer-Based Training 

A general conclusion from meta-analysis research of computerized based instruction is that it has an effect size of approximately 0.30 to 0.35 standard deviations on assessment scores (Kulik & Kulik, 1991; Melmed, 1995; Bayraktar, 2002; Lowe, 2002). With the addition of multimedia, the effect size can be raised to 0.50 standard deviations.

Typically, the effect size is smaller for programs lasting over four weeks (Kahalili, 1994; Liao, 1992). Cavanaugh (2001) reports an effect size of 0.147 for K-12 distance education courses. Similarly, Christmann and Badgett (2000) cite an effect size of 0.187 to 0.209 for secondary students and an effect size of 0.127 in college students.

Studies also indicate that the time for instruction is compressed when technology is used (Kulik & Kulik, 1991). Computer-Based Instruction (CBI) took about two-thirds as much time as traditional courses in 32 post-secondary classrooms (Kulik & Kulik, 1991). More evidence indicates that the pace of instruction to reach objectives is increased by computer-based instruction. Orlansky and String (1977) found that the time on task to achieve objectives was decreased by 54%. Schmeekle (2000) found that the amount of instructional time for adult training in law enforcement was reduced by 50%. In a more generalized view, Fletcher (2001) found that the average instructional time is reduced by 30 percent utilizing computer-based training compared to traditional classroom training. Studies with adults had an average effect size across all areas of 0.25 standard deviations (Kulik & Kulik, 1991).

According to Clark (1994b), meta-analytic research in computer-based training may have fundamental flaws. Meta-analytic studies may not have accounted for the confounding of the media variables with instructional method (Clark, 1994b). Clark (1994a) indicates that it was the instructional method used and not the media that produced the positive outcomes in meta-analytic studies. Additionally, Clark states that the results of these effect sizes were not significant (Carter, 1996).

Distance Learning in the Military

The DoD is very enthusiastic concerning the use of distributed training courses online to increase training opportunities as evidenced by Secretary of Defense William Cohen's 2001 Annual Report. Delivery methods used by the military include print, terrestrial and satellite Video Teletraining (VTT), computer-mediated communication, and full-motion compressed digital video (Barry & Runyan, 1995).

Recently, the DoD has emphasized the use of the Internet as a distance learning delivery method (Bonk & Wisher, 2000; ADL, 2001). The DoD established the Advanced Distributed Learning (ADL) laboratories in 1997 to develop standards in distance learning delivery systems (ADL, 2001). The Internet and distributed learning are central to the DoD strategy of providing training anywhere and anytime (Bonk & Wisher, 2000). According to the DoD mandate, ADL has begun work on the Sharable Content Object Reference Model (SCORM) to enable content reusability, accessibility, durability and interoperability (ADL, 2001). ADL employs the use of Learning Management Systems (LMS) to allow for the reuse of content objects that have been tagged according to the SCORM data model.

Military distributed training research has indicated no significant difference in learning outcomes, but reduced costs and decreased instructional time (Hahn, Ashworth, Phelps, Wells, Richards & Daveline, 1991; Wisher et al., 1999; Fletcher, 2001). An evaluation of the System for Managing Asynchronous Remote Training (SMART), an Army distributed learning course, indicated that there was no significant difference in scores between the online group and the classroom group (Hahn et al., 1991).

Studies in the area of military distance education events over an extended time indicate no significant difference in student outcomes when compared to conventional classrooms as well as favorable returns on investments (Wisher & Curnow, 1998; Wisher & Priest, 1998; Wisher et al., 1999; Fletcher, 2001). Minimal research exists concerning the effectiveness of short-term, distributed, online training events.

Short-term Training Events

Training events that are less than one day in duration have special consideration when being evaluated (Wisher & Curnow, 1998). Short-term events do not lend themselves to traditional experimental designs like a timed series pretest/posttest design because of their brevity. Resources for such training events are most likely limited and therefore cannot support the costs of creating and administering lengthy evaluations. Short-term training events may not have a measurable business impact. Therefore, Wisher and Curnow (1998) developed the compressed evaluation form to evaluate short-term distance training events.

Measuring learning is made difficult in a short-term learning event because of both the novelty effect and the costs involved with creating a valid and reliable assessment. The novelty effect (increased effort because of the intervention) confounds the relationship between learning outcomes and the treatment (Clark & Sugrue, 1990). Clark and Sugrue (1990) reported that increased attention leads to an increased effort that in turn inflates achievement gains. In secondary education studies, an intervention lasting less than four weeks had an effect size around 0.50 standard deviations; after four weeks the effect size was reduced to 0.30 standard deviations and after eight weeks the effect size was reduced to 0.20 standard deviations (Clark & Sugrue, 1990; Kulik & Kulik, 1991). The novelty effect as a confounding variable may be diminished in adults as the use of computers is not a novel experience (Clark & Sugrue, 1990). Also, a short-term training event usually does not provide for additional assessments because of the costs to develop and implement (Wisher & Curnow, 1998). Therefore, a self-assessment learning measure is implemented in the survey.

The self-assessment learning measure is highly subjective. Participants usually overestimate their learning as compared to performance (Mabe & West, 1982; Campbell & Lee, 1988; Wisher & Curnow, 1998). Self-assessment in military training is a fairly accurate indicator and does not tend to overestimate knowledge or performance (Wisher & Curnow, 1998). This may be because participants have been involved in military training and self-assessment reporting for many years. Lane and Herriot (1990) indicated that validity of self-assessments increases when time of service is increased. Moreover, when students know their ratings will be compared to later performance, the validity of self-assessed scores is increased (Mabe & West, 1982).

Wisher and Curnow (1998) administered the compressed survey form to 1,044 participants in 8 different short-term training events. Instruction was delivered in one-way video and two-way audio format over satellite links to participants at remote sites. Wisher and Curnow (1998) indicated the form possessed face validity when prior knowledge was compared to learning outcomes. As hypothesized, students with little prior knowledge reported learning more than students with prior knowledge (F (1,476) = 28.16, p < 0.001), suggesting the form had face-validity. The authors reported that the form possessed external and internal reliability. The measure of previous knowledge and learning outcomes were consistent between samples indicating external reliability. A 2X3 analysis of variance (ANOVA) with the two levels of previous course experience ("yes," "no") and the three levels of learning ("little," "some," "lot") revealed no significant mean differences in the amount of perceived learning (F (2, 476) = 0.38, p = 0.67).

METHOD

The target audience for the Equal Opportunity Advisor for the Reserve Component (EOARC) course included Reservists, National Guard, and DoD civilians who advise commanders on equal opportunity issues. Students in the online courses were selected based on having completed the non-resident portion and resident portion of the EOARC course. Fifty-seven students (N = 57) completed the pre-resident and resident phases of the course. Thirty-six of the students were in the Army and twenty were from the Air Force. Nine students were active duty, sixteen were reserve component, and thirty students were members of National Guard Units. Thirty-five of the participants reported taking the course at their home, while thirteen participants were online at their work sites and eight participated at their military unit. One student took the course at a public library, and one student participated at an unknown location.

Course Description

The Nebraska National Guard has created the Advanced Learning Environments Research and Technology Project, otherwise known as Project ALERT, to generate distributed multimedia courses to be delivered via the Internet. The short-term distributed online training modules provided the non-resident phase one of the Equal Opportunity Advisor (EOA) courses and the EOA phase for the Reserve Component course. The EOA course is designed to provide an understanding of human relations in organizations. Graduates of the program serve as advisors to commanders to reduce or eliminate discriminatory practices. The non-resident portion of the course has the following eight sections and a final exam: Group Development Theory, Ethical Decision Making, Cross-Cultural Communications, Conflict Management, Effective Feedback, Ethical Decision Making, Perceptions, and Socializations.

The non-resident phase of the EOA course was completed online prior to the 3-week resident portion of the course. The Defense Equal Opportunity Management Institute (DEOMI) subject matter experts (SME) created the non-resident content units. The non-resident phase consists of eight distributed web-based modules. Each module contained a list of objectives and a learning check at the end of the course. After the completion of the eight units, students were required to pass an online 50-item exam with a score of at least 70%. Students enrolled in the non-resident portion of the course at least three months prior to attending the resident phase of EOARC.

Course Design and Delivery

The training events were designed with material provided by the SME, used in the conventional classroom version of the courses. Collaborative design meetings were held with SMEs, interactive media producers from Nebraska Educational Telecommunications, and Project Managers from Project ALERT to design the courses for delivery over the Internet. The training events employ journal activities, interactive activities with automated computer feedback, graphics, animations, audio, and video segments. The modules were designed with reusability in mind as indicated in the advanced distributed learning (ADL) guidelines (ADL, 2002). The instructional design also employed the ADL guidelines for content relevance, accuracy, and interactive instructional strategies. Course content was packaged in Avilar's WebAuthor content packaging program. The courses were then deployed to Avilar's WebMentor LMS running at the National Guard Professional Education Center (PEC) in Little Rock, Arkansas. According to the DOD and the ADL initiative, a courseware product that can deliver, track, and administer learning content, and also report on student progress and student interactions is considered a learning management system (LMS) (ADL, 2001).

The Avilar's WebMentor LMS enables Web clients (Internet Explorer and Netscape Navigator) to connect using the Hypertext Transfer Protocol (HTTP) and receive Hypertext Markup Language (HTML) encoded documents over Transmission Control Protocol/Internet Protocol (TCP/IP) (Brooks, Nolan, & Gallagher, 2001). Interactions in the course were created using Netscape's JavaScript and Macromedia's Flash program. Video and audio clips used in the modules were compressed using Apple's QuickTime format. The learner is viewed as an explorer in the modules; he/she has control over the technology and learning activities, and can explore the knowledge space (Jones & Paolucci, 1999).

The length of each module varied from two to four hours seat-time equivalency. The seat-time equivalency is the amount of time the module took in a traditional classroom. The total seat-time for the modules was 22 equivalent hours. Students reported spending less than the seat-time equivalent per hour (M = 2.8, SD = 0.93) for a total of 19.8 hours total for all modules. The compressed evaluation form was used on these modules because the total duration on the training event was less than 24 hours.

Survey Instrument

The survey consisted of 27 questions. Twenty-one of the questions were Likert-type scale with a 5-point scale (1 = poor; 5 = excellent). The survey contained five open-ended qualitative questions. Eight questions on the survey explored participants' motivation for taking the course, previous training in EOA topics, perceived learning, and online learning experiences. Thirteen questions dealt with the technology used, course design, accessibility speeds, and the learning environment. Three open-ended questions dealt with student reaction to the technology, one question was left entirely open (other comments), and the final question asked what other courses would be desirable to have online. Wisher and Curnow (1998) indicated the survey contained face-validity by comparing learning outcomes with prior knowledge.

Administration Procedure

The survey instruments were delivered by a Project ALERT representative and distributed by the course instructor. The forms were completed at the end of the resident course in one day. The completed forms were received the same day as distributed by the Project ALERT representative. Survey data was coded and complied into the software program Statistics Package for Social Sciences (SPSS) version 11.

RESULTS

The results of this study did not find a significant difference in prior knowledge compared to learning outcomes (F = 2.215, p = 0.14). However, when the scores of course relevance (M = 4.28, SD = 0.78) were compared with Wisher and Curnow (1998) reported scores (M = 4.1, SD = 0.96) the differences were not significant t (54) = 1.36, p = 0.17. In addition, the overall effectiveness score (M = 3.92, SD = 0.81) and the Wisher and Curnow (1998) reported scores (M = 3.89, SD = 0.98) were not significantly different t (54) = 0.22, p = 0.82. Thus, the reliability of the survey instrument is similar to the compressed form created by Wisher and Curnow (1998). In addition, a Cronbach's alpha was used to determine an estimate of the reliability of composite scores across the 22 items. The alpha coefficient for the 22 items was found to be 0.80 indicating the composite scores were reliable.

A maximum likelihood factor analysis was used to examine the correlation among survey items. The extraction method used was a principal component factoring rotation method using a Varimax rotation procedure. The rotated solution yielded five interpretable factors above [+ or -] 0.5, accounting for 63 percent of the total variance. Factor one measured the usability of the training product including quality of audio and video, navigation, and the overall effectiveness of the program. The second factor dealt with prior knowledge of the content. Factor three measured previous online course experience. Factor four measured the technical aspects of online training including connection speed and hours spent on the courses. The last factor dealt with the amount learned from the course. The rotated solution is shown in Table 1.

Self-assessment of Amount Learned

Total responses for the self-assessment of amount learned are shown in Table 2. Sixty-three percent of respondents felt they had learned "a lot" from the DEOMI modules. While 23% learned some and 14% reported learning little. The scale was interpreted with ratings of 1 and 2 as "little" a rating of 3 as "some" and ratings of 4 and 5 as "a lot." Overall respondents indicated they had learned some to a lot (M = 3.7, SD = 1.02).

The total responses for motivation are shown in Table 3. The results of the survey indicated that the students considered the training important and were motivated to take part in the training (M = 4.27, SD = 0.90). Eighty-six percent of students indicated that participation in the course was very important for them. Nine percent of respondents felt the participation was somewhat important and six percent of the participants felt participation was not important.

Prior Knowledge and Learning Outcomes

The study population was evenly split between those with previous course experience 49.1% and those without 50.9% (N = 57). Students who had participated in EOA courses (n = 28) indicated a mean increase of 3.68 in their learning outcomes. Students with no previous course experiences (n = 29) indicated a slightly higher mean of 3.72 for perceived learning (Figure 1). Perceived learning as a function of previous coursework is slightly higher for students who did not receive prior instruction.

[FIGURE 1 OMITTED]

The scores of 1 and 2 were paired in the little column, scored of 3 were included in the some column, and scores of 4 and 5 were paired into the lot column.

Student Reactions to the Technology

Students had a favorable rating of the technology as a whole (M = 3.77, SD = 0.76). Table 4 provides a summary of means and standard deviations for the technology questions.

When mean scores were compared with Army National Guard (ARNG) benchmark scores (Table 5, Figure 2) there were significant differences in audio t (54) = 2.51, p = 0.02 and video quality t (48) = -3.02, p = 0.003. Course relevance t (54) = 1.31, p = 0.191 and overall effectiveness of instruction t (54) = 0.95, p = 0.34 were not significantly different from the ARNG benchmark.

The audio quality had the lowest mean score with the highest standard deviation indicating that there was wide variance in the scores (M = 3.38, SD = 1.19). The frequency of the scores is reported as reported in Table 5 indicate 32% percent of students rated the quality of audio as medium in quality. A higher percentage of respondents reported the audio being higher in quality (46%) reporting a 4 or 5, than those rating it lower in quality (20%). Table 6 displays the frequencies if responses to audio quality.

[FIGURE 2 OMITTED]

Students indicated that they were connecting to the modules with a moderately fast connection (M = 3.5, SD = 1.38). A majority 56% of the students responded with a 4 or 5 indicating that students had broadband connections nearing a T1 while 25% of students reported slower connection.

Students reported spending more than 2 hours per unit (M = 2.84, SD = 0.94). When asked if the short-term training modules had prepared them for the in-resident phase of the course, 48% felt they were very prepared for the in-resident phase of the course, 40% were somewhat prepared and 13% did not feel that the online modules prepared them for the in-resident portion of the training (M = 3.42, SD = 0.86).

When asked what respondents liked most about the course 25% indicated they liked the navigational system of the modules. Twenty-three percent indicated the convenience of the courses and working from home. Nineteen percent indicated the self-pacing of the course. Feedback was mentioned by 10% of respondents and 10% also indicated they liked the audio, video, animation, and graphics.

Twenty-three percent of respondents indicated that they did not like the time that it took to progress through the modules, and they felt they were not given enough time to complete it. Twenty-one percent of respondents wanted to be able to print the entire course or have a hardcopy of the course. In addition, 19% of respondents felt the test questions were inappropriate or too difficult.

When asked what they would change in the courses, 31% of respondents indicated they wanted to have the ability to print. Fourteen percent indicated that the test questions should be modified and the test should be made shorter.

Twenty-six percent of respondents to the other comments question indicated a positive response to the modules including: "Excellent online course," "Good job," "Good starting point." An additional 15% mentioned that they would like to be able to print the entire course or have hardcopies available.

DISCUSSION

Summary of Major Results

Using a self-assessment of learning, respondents felt they had increased their knowledge of the subject (M = 3.7, SD = 1.02). Sixty-three percent of the students felt they had learned a lot, while only fourteen percent reported learning little. In addition, 88% of participants felt that the training modules had prepared them for the in-resident phase of the course (M = 3.42, SD = 0.84).

An unexpected result of the study indicated that prior knowledge was not related to self-assessed of amount learned. Results indicated no significant difference in learning based on experience (F = 2.21, p = 0.14). In a similar study, Wisher and Curnow (1998) found a significant difference in means based on previous courses F (1,476) = 28.16, p < 0.001. The results indicate a mediating variable in learning outcomes. One variable to consider is motivation to take the course; participants indicated it was important to take the course (M = 4.27, SD = 0.90). However, a Pearson Chi-Square test of independence indicated that motivation and learning are independent [chi square] (12, N = 56) = 15.02, p = 0.24). Another possible explanation is that when the content is simple, any medium can convey the learning objectives (Chanlin, 1999). Additionally, experts process content differently than novices. An expert will search content for very specific information within the content and modify prior knowledge as they attend to new information (Cennamo & Chung, 1995). Novices, on the other hand, are required to build new mental models as they progress through the content. Therefore, when simple content is combined with prior knowledge, students can quickly become experts in the content area. It is unlikely that experts would report learning new information in a short-term training event with simple content. Simple content would explain the difference in learning outcomes due to prior experience found in the Wisher and Curnow (1998) study.

The results indicated that the training event was relevant to the resident phase of the course and their careers (M = 4.27, SD = 0.78. In addition, course effectiveness had a mean of 3.93, indicating that students felt the training event was effective.

A correlation for the data indicated that learning was significantly related to computerized feedback, course relevance, and effectiveness of instruction (Table 7). The results of this study suggest that, for the DEOMI non-resident course, student self-assessment of learning was related to the relevance of the instruction and the overall effectiveness of the training event and the use of immediate feedback.

Reaction to the Technology

The results of the survey indicated that students reacted favorably to the technology used to deliver the training event (M = 3.77, SD = 0.76). Respondents indicated audio t (54) = 2.51, p = 0.02 and video t (48) = -3.02, p = 0.003 quality was lower than reported benchmarks set by the ARNG. These responses were expected because of the fundamental problem of transferring large amounts of data over bandwidth-challenged networks.

CONCLUSION

The results of this study were consistent with findings of Wisher and Curnow (1998) suggesting that a compressed form for short-term training events can be used to measure level 1 and level 2 of Kirkpatrick's model of evaluation. A factor analysis indicated that five factors (usability, prior knowledge, previous online experience, technical considerations, and assessment of amount learned) accounted for 63 percent of the total variance. Double loading of the relevance to job questions was found on the usability and prior knowledge factors, indicating the questions should be revised.

The use of self-assessments in learning provided information concerning the amount participants learned and what variables affect learning. Contrary to the Army National Guard distance learning events reported by Wisher and Curnow (1998), there was no significant relationship found between prior course experience and self-assessed learning scores.

Overall Significance of the Study

Participants in the short-term distributed online training event reported an increased in their knowledge. Distributed online training is an effective method of training DoD employees and military personnel. Previous courses in the subject area did not affect the learning. Immediate feedback, course relevance and the overall course effectiveness were found to be determinants in the learning variable. Finally, students indicated the technology used in the courses was overall satisfactory with video, and audio was reportedly lower in quality than benchmarks set in the Army National Guard courses reported by Wisher and Curnow (1998).

Implications

The use of a compressed survey form for short-term distributed online training events is useful in examining technical and learning issues of interest to stakeholders. The compressed evaluation form is designed to provide prompt feedback to program managers, subject matter experts, instructional designers, and courseware developers. Analysis of the survey variables may uncover unforeseen problems or trends. Modification based on survey results may lead to better-designed and more effective courseware. Results may also be used as a determinate in the choice of media for the training event.

Appendix 1

Survey Form

Project ALERT Distance Learning Evaluation Sheet

Today's Date: ___/___/___

Day Month Year

In an effort to examine the quality, delivery method, courseware and instruction for the Online Instruction, we would like your candid feedback and input. Please help us assess and improve our current and future Internet Courses, delivery system and instruction by completing this evaluation. Your responses will be kept confidential and used only in summary reports. Thank you for your cooperation.

Not important at all  (1)  (2)  (3)  (4)  (5)  Very important

1. Have you had previous training related to the training topic?

Yes  (1)  No  (2)

2. Compared to what you already knew about topics related to the training, how much more did you learn in the online course? Respond using the five-point scale below.

None  (1)  (2)  3)  (4)  (5)  A Lot More

3. How many other online courses have you previously participated in?

                               None  1 or 2  3 or 4  5 to 10  10 or more

For your military training?    (1)   (2)     (3)     (4)      (5)
For your civilian job?         (1)   (2)     (3)     (4)      (5)
For education or occupational  (1)   (2)     (3)     (4)      (5)
  training?
For other interests (e.g.      (1)   (2)     (3)     (4)      (5)
  hobbies etc.)?

4. Please respond to the following questions using a five-point scale with 1 being "Poor" and 5 being "Excellent". From your vantage point, how would you rate:

                                                              Not
                              Poor                 Excellent  Applicable

Design of the navigational    (1)   (2)  (3)  (4)  (5)        (NA)
  system:
Quality of audio:             (1)   (2)  (3)  (4)  (5)        (NA)
Quality of video:             (1)   (2)  (3)  (4)  (5)        (NA)
Quality of animation:         (1)   (2)  (3)  (4)  (5)        (NA)
Quality of pictures and       (1)   (2)  (3)  (4)  (5)        (NA)
  graphics:
Effectiveness of learning     (1)   (2)  (3)  (4)  (5)        (NA)
  activities:
Effectiveness of              (1)   (2)  (3)  (4)  (5)        (NA)
  computerized feedback:
Relevance to course or job:   (1)   (2)  (3)  (4)  (5)        (NA)
Overall effectiveness of      (1)   (2)  (3)  (4)  (5)        (NA)
  instruction:

5. How fast was your Internet connection to the course?

Slow (28.8 kps)  (1)  2)  (3)  4)  (5)  Fast (T1)

6. For DEOMI courses, how many hours AVERAGE did you spend on each module?

(1)  (2)  (3)  (4)  (5)

7. How well did the online modules prepare you for the in-residence portion of the DEOMI RCCEOA course?

Did not prepare me  (1)  (2)  (3)  (4)  (5)  Well prepared

8. What was your PRIMARY learning environment? (Please Circle)

Home  Military  Civilian  Public   Distance Learning Center  Other______
      Unit      Office    Library

9. What did you like most about the online training courses?

10. What did you like least about the online training courses?

11. What would you change to make the online training easier to use, more understandable or more interesting?

12. Other Comments:

13. What other courses would you like delivered in the online format?

Branch of Service: Circle One

Army  Air Force  Navy  Marine  Coast Guard  Other

Status: Circle One

Active Duty  Reserve  National Guard  Civilian

Table 1 Rotated Factor Matrix

                                                  Prior      Factor
Item                                   Usability  Knowledge  Experience

How important was it for you to                   0.685
  participate in this training?
Have you had previous training
  related to the training topic
Compared to what you already knew                 0.650
  about topics related to the
  training, how much more did you
  learn in the online course
How many other online courses have
  you previously participated in
For your military training                                   0.763
For your civilian training                                   0.572
For education or occupational                                0.804
  training
For other interests (e.g. hobbies                            0.709
  etc.)
Design of the navigational system      0.661
Quality of audio:                      0.648
Quality of video:                      0.731
Quality of animation:                  0.797
Quality of pictures and graphics:      0.738
Effectiveness of learning activities:  0.755
Effectiveness of computerized          0.611
  feedback:
Relevance to course or job:            0.496      0.466
Overall effectiveness of instruction:  0.744
How fast was your Internet connection
  to the course
For DEOMI courses, how many hours
  AVERAGE did you spend on each
  module
How well did the online modules                   0.656
  prepare you for the in-residence
  portion of the DEOMI RCCEOA course
What was your PRIMARY learning
  environment

                                       Technical  Amount
Item                                   Aspects    Learned

How important was it for you to
  participate in this training?
Have you had previous training                    0.790
  related to the training topic
Compared to what you already knew
  about topics related to the
  training, how much more did you
  learn in the online course
How many other online courses have
  you previously participated in
For your military training
For your civilian training
For education or occupational
  training
For other interests (e.g. hobbies
  etc.)
Design of the navigational system
Quality of audio:
Quality of video:
Quality of animation:
Quality of pictures and graphics:
Effectiveness of learning activities:
Effectiveness of computerized
  feedback:
Relevance to course or job:
Overall effectiveness of instruction:
How fast was your Internet connection   0.756
  to the course
For DEOMI courses, how many hours      -0.588
  AVERAGE did you spend on each
  module
How well did the online modules
  prepare you for the in-residence
  portion of the DEOMI RCCEOA course
What was your PRIMARY learning          0.588
  environment

Note: Extraction Method: Principal Component Analysis. Rotation Method:
Varimax with Kaiser Normalization a Rotation converged in 15 iterations.

Table 2 Respondents' Self-Assessment Learning Measure

Amount learned  Number of Students  Percentage of Students

Little                8                       14
Some                 13                       23
A Lot                36                       63

Table 3 Importance of Training Participating in Training Event

Important   Number of Students  Percentage of Students

Not              3                   6
Moderately       5                   9
Very            48                  86

Table 4 DEOMI Technology Means and Standard Deviations

Question                           N   M     SD

Navigational System                55  3.82  0.93
Audio Quality                      55  3.38  1.19
Video Quality                      49  3.78  0.96
Animation Quality                  56  3.80  0.98
Graphics Quality                   56  4.00  0.76
Learning Activities Effectiveness  56  3.79  0.87
Feedback Effectiveness             55  3.60  1.10
Course Relevance                   55  4.27  0.78
Overall Effectiveness              55  3.93  0.81

Table 5 DEOMI Technology Means and Standard Deviations

                                              DEOMI Events
Question                      ARNG Benchmark  M     SD    N   t-test

Quality of Audio                 3.8          3.38  1.20  55  2.51*
Quality of Video                 4.2          3.78  0.96  49  3.02*
Relevance                        4.1          4.27  0.78  55  1.31 (NS)
Effectiveness of instruction     3.8          3.90  0.81  55   .95 (NS)

* significant at p <.05 (NS) Not significant

Table 6 Frequency of Responses to Audio Quality

Quality    Number of Students  Percentage of Students

Poor           11                   20
Average        18                   32
Excellent      26                   46

Table 7 Correlation Matrix for Learning, Feedback, Relevance, and Course
Effectiveness

                       Computerized  Course       Effectiveness of
                       Feedback      Relevance    Instruction

Learning               + 0.435*      + 0.426*      +0.540*
Computerized Feedback                + 0.293**    + 0.549*
Effectiveness of                                  + 0.528*
  Instruction

*Correlation is significant at the .01 level (2-tailed) **Correlation
is significant at the .05 level (2-tailed)

References

ADL (2001). Sharable Content Object Reference Model (SCORM) Version 1.1. In P. Dodds (Ed.).

ADL (2002). ADL Guidelines 2002. Retrieved from http://www.adlnet.org/

Barry, M., & Runyan, G. (1995). A review of distance-learning studies in the U.S. military. The American Journal of Distance Education, 9(3), 37-47.

Bayraktar, S. (2002). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education 34(2), 173-188.

Belanger, F., & Jordan, D. (2000). Evaluation and implementation of distance learning: Technologies, tools and techniques. Hershey, USA: IDEA Group Publishing.

Bonk, C., & Wisher, R. A. (2000). Applying collaborative and e-learning tools to military distance learning: A research framework. Alexandria, VA: United States Army Research Institute for the Behavioral Sciences.

Brooks, D. W., Nolan, D. E., & Gallagher, S.M. (2001). Web-teaching : A guide for designing interactive teaching for the world wide web. New York, NY: Kluwer Academic/Plenum Publishers.

Campbell, D. J., & Lee, C. (1988). Self-appraisal in performance evaluation: Development versus evaluation. Academy of Management Review 13(2), 302-314.

Carter, V. (1996, February). Do media influence learning? Revisiting the debate in the context of distance education. Open Learning, 31-40.

Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7(1), 73-88.

Cennamo, K. S., & Chung, M.-L. (1995). An exploratory study of the cognitive processing of videotaped materials on a familiar and unfamiliar topic. International Journal of Instructional Media, 22(4), 15.

Chanlin, L.-J. (1999). Visual treatment for different prior knowledge. International Journal of Instructional Media, 26(2), 213.

Christmann, E., & Badgett, J. (2000). The comparative effectiveness of CAI on collegiate academic performance. Journal of Computing in Higher Education., 11(2), 91-103.

Clark, R. E. (1994a). Media and method. Educational Technology Research and Development, 42(3), 7-10.

Clark, R. E. (1994b). Media will never influence learning. Educational Technology, Research and Development, 42(2), 21-29.

Clark, R. E., & B. M. Sugrue (1990). International views of the media debate. International Journal of Educational Research, 14(6), 485-579.

Cohen, W. S. (2001). Annual report to the president and congress. Washington, D.C.: Department of Defense.

Dede, C. (1996). The evolution of distance education: Emerging technologies and distributed learning. The American Journal of Distance Education, 10(2), 4-36.

Fletcher, J. D. (2001). Technology, the Columbus effect, and the third revolution in learning. Submitted for publication.

Hahn, H. A., Ashworth, R. L., Phelps, R. H., Wells, R. A., Richards, R. E., & Daveline, K. A (1991). Distributed training for the reserve component: Remote delivery using asynchronous computer conferencing. Alexandria, VA: Army Research Institute for the Behavioral and Social Sciences.

Hanson, D., Maushak, N., Schlosser, C. A., Anderson, M. L., Sorensen, C., & Simonson, M. (1996). Distance education: Review of the literature (2nd edition). Iowa, USA: Iowa State University of Science and Technology, Research Institute for Studies in Education.

Hedberg, J. G., Brown, C., & Arrighi, M. (1997). Interactive multimedia and web based learning: similarities and differences. In B. H. Kahn (Ed.), Web based instruction. Englewood Cliffs, NJ: Educational Technology Publications.

Jones, T. H., & R. Paolucci (1999). Research framework and dimensions for evaluating the effectiveness of educational technology systems on learning outcomes. Journal of Research on Computing in Education, 32(1), 17-29.

Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler Publishers.

Kahalili, A. (1994). The effectiveness of computer applications: A meta-analysis. Journal of Research on Computing in Education, 27(1), 48.

Kulik, C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 2, 75-94.

Liao, Y. K. (1992). Effects of computer-assisted instruction on cognitive outcomes: A meta-analysis. Journal of Research on Computing in Education, 24(3), 367.

Lane, J., & Herriot, P. (1990). Self-ratings, supervisor ratings, positions and performance. Journal of Occupational & Organizational Psychology, 63(1), 12.

Lowe, J. (2002). Computer-based education: Is it a panacea? Journal of Research on Computing in Education, 34(2), 163-171.

Mabe, P. A., & West, S. G. (1982). Validity of self-evaluations of ability: A review and meta-analysis. Journal of Applied Psychology, 67, 280-293.

Mclsaac, M., & Gunawardena, C. (1996). Distance education. Handbook of Research for Educational Communications and Technology. New York, NY: Simon & Schuster Macmillan.

Melmed, A. (1995). The costs and effectiveness of educational technology: Proceedings of a workshop. DRU-1205-CTI, RAND, Santa Monica, CA, November 1995.

Orlansky, J., & String, J. (1977). Cost effectiveness of computer-based instruction in military training (IDA Paper P-1375). Arlington, VA: Institute for Defense Analysis.

Schmeeckle, J. M. (2000). Online training: An evaluation of the effectiveness and efficiency of training law enforcement personnel over the internet. Unpublished Dissertation, University of Nebraska-Lincoln.

Simonson, M., Schlosser, C. A., & Hanson, D. (1999). Theory and distance education: A new discussion. The American Journal of Distance Education, 13(1), 60-75.

Simonson, M., Smaldino, S., Albright, M., & Zvacek, S. (2000). Teaching and learning at a distance: Foundations of distance education. Columbus, Ohio: Merrill.

Wisher, R. A., Champagne, M. V., Pawluk, J. L., Eaton, A., Thornton, D. M., & Curnow, C. K. (1999). Training through distance learning: an assessment of research findings (Technical report 1095). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Wisher, R. A., & Curnow, C. K. (1998). An approach to evaluating distance learning events. Alexandria, VA, U.S. Army Research Institute for the Behavior and Social Sciences.

Wisher, R. A., & Priest, A. N. (1998). Cost-effectiveness of audio teletraining for the U.S. Army National Guard. The American Journal of Distance Education, 12(1), 8.

Yates, C., & Tilson, T. (2000). Basic education at a distance. New York, NY: Routledge/Falmer.

BRADLEY BARKER AND DAVID BROOKS 

University of Nebraska-Lincoln, USA

[e-mail]

[e-mail]