Printer Friendly

Automated feedback as a convergence tool.

1. INTRODUCTION

Increasing innovation and technological advancement with respect to training and learning technologies are creating new pressures for information systems instructors to effectively deliver quality technical material. As a wide variety of options for education delivery have emerged, including web-based training, and multimedia options including video-conferencing, web-based tutorials like Kahn Academy, and MOOCs, evaluating the efficacy of each option and its appropriateness relative to the purpose of the training is increasingly important. Media synchronicity theory (MST) suggests that certain media capabilities may be more or less appropriate given the primary communication task (Dennis, Fuller, and Valacich, 2008).

This research evaluates two content delivery options to determine whether an asynchronous experience can achieve the same efficacy as a traditional face-to-face (F2F) lecture format. With this study, we address the following research question:

   Can we design an asynchronous medium to provide
   media synchronicity and achieve similar learning
   outcomes as synchronous F2F instruction?


We examine a traditional F2F format, an asynchronous tutorial, and the combination of the two on the level of learning achieved by the recipients of the training. We derived the research design using MST (Dennis, Fuller, and Valacich, 2008). We collected data from three different student samples receiving training on PHP scripting. Students completed a pretest as well as a posttest after receiving one or more of the treatments. This method allowed us to examine the relative impact of each delivery mode separately, together, and in different orders.

2. PREPARATION OF MANUSCRIPTS

Media synchronicity theory (MST) identifies two fundamental communication processes. The first is conveyance, which is the transfer of information from one person to another. The second process is convergence, which is when two or more people agree on the same meaning for information.

In order to perform conveyance or convergence, an individual must engage in two individual processes: information transmission (preparing information for transmission, transmitting it through a medium, and receiving information from a medium) and information processing (understanding the meaning of information and integrating it into a mental model). (Dennis, Fuller, and Valacich, 2008, p. 576)

Conveyance and convergence have different requirements for both transmission and processing of information. One of the significant differences between the two is that conveyance requires information processing by an individual, but convergence requires information processing by more than one person to arrive at a shared meaning. Convergence assumes that the individuals involved have already transmitted some information. Conveyance requires that the individual receive the new information and process it. That is, the individual will need to analyze the new information and adjust their existing mental models to accommodate the information. This can be a time-consuming activity depending on the novelty of the conveyed information. "Convergence processes are the discussion of preprocessed information about each individual's interpretation of a situation, not the raw information itself" (Dennis, Fuller, and Valacich, 2008, p. 580). Two or more individuals are attempting to achieve a common understanding. Convergence, therefore, requires back-and-forth communication among the group of often small amounts of (individually pre-processed) information.

Synchronicity in MST refers to the communicators "exhibiting] a shared pattern of coordinated synchronous behavior with a common focus" (Dennis, Fuller, and Valacich, 2008, p. 581). Synchronicity is the result of the coordinated work of the communicators, not whether the medium is used at the same time. In this paper, when referring to whether a medium is used at the same time, or not at the same time by the sender and receiver(s), we use the terms synchronous, or asynchronous, respectively. However, when discussing the "shared patterns of coordinated behavior with a common focus," and the medium's ability to facilitate such behavior, we then use the term media synchronicity. Therefore, media synchronicity is defined as "the extent to which the capabilities of a communication medium enable individuals to achieve synchronicity" (Dennis, Fuller, and Valacich, 2008, p. 581). Because conveyance is focused on the delivery of (relatively) large amounts of information, there is less need for the communication to be synchronous. There is even an advantage for asynchronous communication in that the receiver has time to assimilate the new information. On the other hand, convergence with its focus on achieving a shared meaning requires relatively smaller amounts of information transmission and benefits from frequent and closely spaced (high velocity), back-and-forth communication. Media that supports asynchronous communication should result in better conveyance of information than highly synchronous communication.

Whether a communication medium supports media synchronicity or not is determined by the media capabilities inherent in the medium. MST identifies five core capabilities that are the most important in assessing the ability of the medium to support synchronicity; (1) reprocessability, (2) rehearsability, (3) symbol sets, (4) transmission velocity, and (5) parallelism. An asynchronous medium has the potential for higher reprocessability (the medium's capability to the let receivers reexamine the message) than does a synchronous medium if the receiver is able to replay the messages. An asynchronous medium also supports rehearsability (the ability of the sender to "rehearse" his/her message prior to sending it) (Dennis, Fuller, and Valacich, 2008).

The symbol sets that a medium can transmit refers to the content of the message. Does the medium only transmit text, like in a written letter? Alternatively, can it include images, video, or some other symbology that carries meaning? An asynchronous medium that supports reprocessability likely also provides for more symbol sets (Scott and Sarker, 2010). A synchronous medium can also carry many symbol sets. Consider a F2F conversation where, in addition to the textual content of the spoken word, the speaker's body language, the tone of their voice, and their gestures all provide add some layers of meaning. Transmission velocity refers to the speed at which the message is transmitted from the sender to the receiver and corresponds to Daft and Lengel's "immediacy of feedback" (1986). In fact, transmission velocity also impacts the speed at which feedback occurs. For example, when in a classroom, a student may ask a question and receive an immediate answer (feedback); however, this is balanced by lower parallelism as the instructor may hear and answer only one question at a time.

2.1 Conveyance and Convergence in IS Education

Effective communication requires both conveyance and convergence. Nonetheless, some tasks need more conveyance and less convergence while other tasks require more convergence and less conveyance. For instance, in an educational situation the instructor (sender) needs to convey the information so the students (receivers) understand what the instructor is saying. However, once that information has been conveyed convergence is required so the students and instructor can verify that the students have achieved an appropriate understanding of the material. A medium that supports convergence--by providing transmission velocity and symbol sets--will likely result in more effective communication. The other side of that coin might be a situation where the instructor provides results/grades to the students. The primary communication process needed in such a situation is to convey the status information. Some convergence may be required, but could still be achieved using a medium that supports conveyance by providing the instructor with rehearsability to ensure that the grade and feedback provided are more likely to be received accurately by the students. Students could use reprocessability as they review the grade and/or comments to verify their understanding of the communication. But, the role of the medium in facilitating conveyance and/or convergence is not well-understood.

Media such as telephone or F2F conversations provide more effective convergence because the communication occurs synchronously, which allows for higher transmission velocity and faster information processing (Dennis, Fuller, and Valacich, 2008). Likewise, media such as email, letters, and memos provide more effective conveyance because the communication can occur asynchronously, which provides for higher quality transmission and allows for more retrospection with slower processing characteristics (Dennis, Fuller, and Valacich, 2008). In an educational context, a medium that supports convergence by providing transmission velocity is the traditional F2F lecture, whereas conveyance providing reprocessability is achieved with technologies such as Blackboard or Canvas, and even other documents and presentations such as PowerPoint slide decks.

MOOCs are an example of education delivery media that rely on reprocessability to compensate for the minimal support for convergence provided through group discussion threads. However, empirical evidence is mixed with respect to the role of the medium in supporting convergence or conveyance. In a study evaluating instant messaging (IM), participants viewed IM as a synchronous medium, but they found IM to be more suited for conveyance rather than convergence (Hung et al., 2008). Muhren, Van Den Eede, and Van de Walle examined the use of media in humanitarian crises and found that, at odds with MST's predictions, low media synchronicity media (voice mail, documents, and fax) are insufficient for conveying the information necessary to coordinate disparate groups in crisis situations (2008). These two studies illustrate the need for clearly specifying the relationship between the task at hand (e.g., negotiation, status-report) and understanding which combinations of communication processes (conveyance and convergence) will most effectively support that task.

The focus of MST is communication performance. That is, how can the "fit" between the medium's capabilities, the communication processes required, and appropriation factors (familiarity, training, experience, etc.) influence communication performance (see Figure 1). While MST does not predict how a sender will choose a medium, media choice is an important factor in that a poor medium choice could reduce fit, thereby reducing the effectiveness of the communication. George, Carlson, and Valacich (2013) found that when a manager is asked to strategize which communication process they will favor, conveyance or convergence, their choice of medium aligns with those suggested by MST. When respondents had chosen a conveyance strategy, they were more likely to choose email, memos, and letters to accomplish the task. When they had chosen a convergence strategy, they were more likely to choose F2F or the phone to accomplish the task.

To test the role of medium capabilities on communication performance in an information systems educational task, Scott and Sarker (2010) argued that teaching activity diagramming constituted a conveyance task. The media capabilities that supported information processing (symbol sets and reprocessability) were best suited for the receiver to be able to incorporate the new information into their mental models. Their experimental conditions manipulated the levels of symbol sets and reprocessability and found that media with more symbol sets and more reprocessability facilitated better learning. However, they did not account for any convergence processes in the receiver's ability to understand the material conveyed. This leaves open the question, could a more synchronous medium be as effective as an asynchronous medium for a primarily conveyance task, when convergence is included as well? In Scott and Sarker's (2010) experiment, the receivers were not able to receive any feedback, especially instant feedback (transmission velocity) or engage in discussions with other receivers of the information or the instructor (parallelism).

3. HYPOTHESIS DEVELOPMENT

Following any instruction or training, we expect student performance to improve. Performance was measured by the number of correct answers, or scores, students selected from the pretests and posttests they took. Therefore, we hypothesize that:

H1 and H2. Students will score better on a test over the content of the instruction after receiving either a synchronous F2F lecture, or an asynchronous multimedia tutorial, than before receiving the information in either medium.

Furthermore, if students receive both delivery formats, the order in which they are presented should not affect their pretest nor posttest performance. Therefore, we hypothesize that:

H3. The order in which the synchronous F2F lecture and asynchronous tutorial are presented will not affect the pretest nor posttest scores.

The advantage that students gained in Scott and Sarker's (2010) study was attributed to the medium's increased reprocessability and symbol sets. However, they assumed the task to be primarily a conveyance task and did not provide for any convergence. Considering the importance of feedback and discussion in learning, we expect that a F2F lecture vs. the instructional, yet static, tutorial provided in Scott and Sarker's study would provide increased performance. Therefore, we designed our tutorial to provide feedback to the student during the instruction. Therefore, we hypothesize that:

H4. Students receiving the asynchronous tutorial will score the same as students receiving the synchronous F2F lecture.

As the students receive the instruction and incorporate the new knowledge into their existing mental models, the opportunity to reprocess, or revisit the material provides learning benefits (Scott and Sarker, 2010). Through repetition, we expect that students will achieve higher pretest and posttest scores than when only having the instruction once. Therefore, we hypothesize that:

H5a. Students receiving both the asynchronous tutorial and the F2F lecture will score higher than students receiving only the synchronous F2F lecture; or H5b. only the asynchronous tutorial.

4. METHODS

We chose to conduct an experiment because this allowed us to investigate the actual impact of the asynchronous tutorial on student learning and compare this impact to that from a more traditional synchronous lecture-based format. We gave the tutorial to undergraduate students from three semesters of a web development class. Course content and delivery methods were the same in each semester, which ensured that students in all semesters had the same minimal level of understanding of the course content leading into the study. Additional details of the study are provided in the following sections.

4.1 Participants

All 49 participants were undergraduate students majoring in Information Technology Management (ITM) and taking a web development course required by the major. Of the 49 participants, 40 (82%) were male and 9 (18%) were female. The ratio of males to females in the study is consistent with the male to female ratio in the ITM major. In addition, 43 (87%) of the participants were seniors and 6 (13%) were juniors. The age range of the participants was 21 to 53, with an average age of approximately 27. Also, 26% of the participants received an A for the course, 17% received a B, 38% received a C, and 19% received an F. The distribution of grades for the study participants is approximately the same as the overall grade distribution for the course. All students had previously completed an introductory programming course in Java. While all students were asked to participate in the study, it was stressed that participation was optional. Participants received no benefits (e.g., extra credit points, compensation, etc.) and nonparticipants received no penalties. In a few cases students had to repeat the class. Data from those students were only used from their first semester. Seventy-three percent of the students elected to participate. The students had not previously had class material provided via e-learning technologies. The research was approved by our university's Internal Review Board.

4.2 Experimental Tasks

Students in all three classes completed the same set of tasks, which were to complete a pretest, complete the asynchronous tutorial, receive the synchronous F2F lecture, and complete the posttest. What differed between classes was the order in which these tasks were completed. Table 1 lists the ordering for each class.

By varying the order, we were able to construct and test hypotheses comparing the asynchronous delivery method (the tutorial) to the synchronous delivery method (F2F instruction) with respect to student learning. Table 2 lists the hypotheses, the datasets being compared, and the conditions tested, e.g., received F2F instruction, received tutorial, received both, etc. The following provides the details for each experimental task.

4.2.1 Pretest and posttest: Both the pretest and posttest first presented a snippet of HTML code that creates a form. Following this were five questions relating to the HTML form and how the form posts data entered to a PHP script for server-side processing. The pretest and posttest questions were essentially the same; however, the HTML code snippets were different resulting in different answers to the pretest and posttest questions. The pretest and posttest are presented in Appendices A and B. A student's score is the total number of questions answered correctly.

4.2.2 Asynchronous tutorial and synchronous F2F instruction: The tutorial and the lecture focused on creating an HTML form and then posting data from the form to a PHP script. Included in each was the material tested in the pretest and posttest. The asynchronous tutorial was developed using PowerPoint and Visual Basic. Every PowerPoint slide required the student to provide the correct response for a specific concept relevant to the stated problem domain. As can be seen in Figure 2, each slide provided a partial solution and required the student to provide additional detail. At the bottom of the slide were six buttons corresponding to six potential solutions. To incorporate transmission velocity into the tutorial, we sought to provide immediate feedback to the student by having two different experiences depending on the student's responses. If the student pressed an incorrect button, we provided automated feedback by taking the student to a slide that indicated they had chosen an incorrect response and provided a link back to the corresponding concept slide. If the correct button was pressed, the student was notified of this and allowed to advance to the next slide. Following each mode of instruction, the students were given up to a week to complete the posttest.

5. RESULTS

The data used in this analysis were pretest and posttest student scores (i.e. the number of correct answers) from three separate semesters for a total of six datasets. More specifically, each data point was the number of correct responses to the five test questions for a specific student. As such, the data is ordinal and ranges from zero to five. For each semester, unique identifying numbers were assigned to each student for matching pretest scores to their corresponding posttest scores. The test scores for a student were removed from the pretest and posttest datasets if the student did not take both the pretest and the posttest or if the student did not complete the entire tutorial.

We tested the hypotheses by statistically comparing means from the six datasets. Where possible, we tested the means using paired samples t-tests. All other tests used independent samples t-tests without assuming equal variances. We conducted six hypothesis tests, therefore we used Bonferroni's correction to set the significance value to 0.05/6=0.008 (Cohen, 2001). Because the data was not normally distributed, we validated the parametric tests using nonparametric algorithms. We validated the paired samples t-tests using the Wilcoxon Signed-Rank nonparametric test with the null hypothesis that the sample populations are the same. Likewise, we validated the independent samples t-tests using the Mann-Whitney-Wilcoxon nonparametric test with the null hypothesis that the sample populations are the same (Cohen, 2001). The descriptive statistics for the six datasets are displayed in Table 3.

5.1 Hypothesis Tests

In this section, we provide a detailed discussion of the results from the t-tests conducted to test the six hypotheses discussed above. We also provide nonparametric test results because, as Siegel (1956) noted, sample sizes as small as six may be accurately tested with nonparametric statistics. They support the t-test results that we show in Table 4.

Hypotheses 1 and 2 predicted that students would perform better after receiving instruction (whether F2F or the tutorial) than students who had yet to receive any instruction. We tested both hypotheses using a one-tailed independent samples t-test without assuming equal variance. As we expected, students who received instruction (whether F2F or the tutorial) performed significantly better than students who had not received any instruction. This confirms that the material was new to the students. Hypothesis 1 compared the pretest scores from Semester 1 and Semester 2 (see Table 2) and the students who received only the F2F instruction scored (m = 3.83) significantly higher (p < 0.001) than the students who had not received the F2F instruction (m = 1.80). Hypothesis 2 compared the posttest score from Semester 2 to the pretest score for the same semester (see Table 2) to test that students who received the tutorial would perform better than students who had not received any instruction. Semester 2 posttest mean score (m = 3.60) is significantly greater than (p < 0.001) the Semester 2 pretest mean score (m = 1.80).

We predicted that the order of the presentation of instruction (F2F first, then tutorial; or tutorial first, then F2F) would have no effect (Hypothesis 3). We compared the posttest score from Semester 1 to the posttest score for Semester 3 (see Table 2) by performing a two-tailed independent samples t-test without assuming equal variance. Semester 1 posttest mean score (m = 4.08) is not significantly different from the Semester 3 posttest mean score (m = 3.53). Supporting our prediction, the null hypothesis of equal means could not be rejected (p = 0.176).

Hypothesis 4 predicted that there would be no difference between student performance for those receiving only the F2F instruction and those receiving only the tutorial. We compared the pretest score from Semester 1 to the posttest score for the Semester 2 (see Table 2) to test H4 by performing a two-tailed independent samples t-test without assuming equal variance. Semester 1 pretest mean score (m = 3.83) is not significantly different from the Semester 2 posttest mean score (m = 3.60). The null hypothesis of equal means could not be rejected (p = 0.550), therefore supporting our prediction.

We predicted that students who received both modes of instruction would perform better than students who only received the one mode of instruction. Hypothesis 5a tested students' test scores after receiving both modes of instruction with test scores after receiving only F2F instruction (the pretest score from Semester 1 and the posttest score from Semester 1). Semester 1 pretest mean score (m = 3.83) is not significantly greater than the Semester 1 posttest mean score (m = 4.08). The results of the test did not support the alternative hypothesis that the mean of the posttest is higher than that of the pretest (p = 0.169).

Hypothesis 5b compared students who received both modes of instruction to students who only received the tutorial. We compared the posttest score from Semester 1 to the posttest score for Semester 2 (see Tables 1 and 2) to test H5b. Semester 1 pretest mean score (m = 3.83) is not significantly greater than the Semester 2 posttest mean score (m = 3.60). The results of the test did not support the alternative hypothesis that the mean of the Semester 1 posttest was greater than the mean of the Semester 2 posttest (p = 0.118). We tested H5a and H5b by performing a one-tailed independent samples t-test without assuming equal variance.

6. DISCUSSION

Our study addresses the research question "can an asynchronous medium be designed to provide media synchronicity and achieve similar communication effectiveness as the synchronous F2F instruction." The results of H1 and H2 support our expectation that after some instruction, regardless of the medium, the recipients of the training would show improved pretest and posttest scores. We were working with one medium, the F2F lecture, which has the ability to support more media synchronicity than the other, the asynchronous tutorial. While the F2F lecture is synchronous, meaning that the sender and the recipient are working together at the same time, the tutorial is asynchronous, meaning that the sender and recipient are not working together at the same time. Media synchronicity does not necessarily require that the sender and recipients are working together simultaneously, but it does require that they are working with "... a shared pattern of coordinated synchronous behavior with a common focus" (Dennis, et al., 2008, p. 581). We sought to provide that experience of working together toward a common focus to the recipients by manipulating the recipient's perception of transmission velocity (feedback) in the automated tutorial.

As predicted (H3), we found that order of presentation did not make a difference to posttest scores. Formally, we failed to find a difference for the order of presentation. That is, it did not matter whether the students viewed the tutorial first or second, the outcomes were the same. This supports the idea that adequate media synchronicity can be incorporated into an asynchronous tutorial to compensate for the lack of F2F interaction. Further supporting our research question that an asynchronous medium can achieve similar communication effectiveness as the F2F medium, H4 shows that students' performance was equal whether they learned via the F2F or tutorial method.

Somewhat surprisingly, H5a and H5b were not supported. We found that after initial instruction using one medium, additional instruction with the other medium did not improve test scores. It does provide further indirect evidence that the two instructions modes, by themselves, were equivalent with respect to content and effectiveness. This result is counterintuitive with respect to Ebbinghaus (1964), and years of research that has shown that repetition improves learning outcomes at least in word recall studies. However, recently some research has found that repetition is not universally helpful and can even result in decremental recall (English and Visser, 2014; Kuhl and Anderson, 2011). Possibly the failure to support H5 was because there was an insufficient gap between when the subjects heard the lecture or used the tutorial and when they took the posttest (Cepeda et al., 2008). It is also possible that the initial instruction was sufficient for the students to master the material and therefore subsequent instruction did not result in improved test scores.

By using the multiple samples and comparisons among the various pretests and posttests, we were able to isolate the effects of two different media that instructors have available to them. Our results indicate that there are no differences in outcomes between the F2F lecture and the automated tutorial (H4). This provides evidence that technical education can be delivered successfully to large numbers of students through the asynchronous approach. This has considerable benefits to providing IS education asynchronously with the increasing prevalence of distance and distributed education, and yet provide the student the experience of high transmission velocity with their instructor and therefore experience greater convergence during the learning (communication) task. It takes more time to prepare the asynchronous material than it takes to deliver a F2F lecture, however, once the asynchronous material is developed it can be used repeatedly with no additional time from the instructor. The F2F method requires that the instructor repeat the material for every training session. Also, while not tested, the tutorial should scale to large numbers of students. For F2F instruction to have media synchronicity there needs to be the potential for interaction between the student and instructor. For very large F2F classes that potential decreases considerably.

7. LIMITATIONS AND FUTURE RESEARCH

The generalizability of our results must be considered. Participation was voluntary although most students elected to use the tutorial. The topic was one being covered in class with a subsequent for-credit assignment (not a part of this research). Therefore, the subjects were highly motivated, as one would expect in a non-experimental setting. However, the topic of the experiment was one specific topic in PHP coding. This is just one of many types of technical education topics. It may be that not all topics are equally suitable to asynchronous learning.

A further limitation of our study is the small sample sizes. While significant differences were found for Hypotheses 1 and 2, we cannot rule out the possibility that for Hypothesis 3 the failure to find a difference with order of presentation may be the result of the small sample size, although the finding is consistent with our prediction. The small sample sizes also hindered our ability to test hypotheses based on demographic data. This was especially true for gender differences where the small number of women compared to the number of men was particularly problematic.

We focused on only two aspects of media synchronicity, automated feedback (increasing transmission velocity) and the potential for reprocessability. Additional constructs in MST should be explored. Different types of training tasks should be explored. Moreover, the different aspects of MST need to be examined in combination with other types of tasks. It is possible that tasks that are more complex will require more media synchronicity to perform as well, or better than F2F training.

Since scalability is potentially a large benefit to the asynchronous tutorial, further research needs to be done to test that empirically. With the large sample sizes necessary for testing scalability, it might also be possible to explore individual differences among subjects concerning cognitive differences. Finally, it would be good to test the long-term retention of new material. It may be that one medium is more supportive of long-term memory than another.

Tim Chenoweth

Karen Corral

Kit Scott

Information Technology and Supply Chain Management Boise State University Boise, Idaho 83725, USA

timchenoweth@boisestate.edu

8. REFERENCES

Cepeda, N. J., Vul, E., Rohrer, E., Wixted, J. T., & Pashler, H. (2008). Spacing Effects in Learning a Temporal Ridgeline of Optimal Retention. Psychological Science, 19(11), 1095-1102.

Cohen, B. H. (2001). Explaining Psychological Statistics. New York: John Wiley.

Daft, R. L. & Lengel, R. H. (1986). Organizational Information Requirements, Media Richness and Structural Design. Management Science, 32(5), 554-571.

Dennis, A. R., Fuller, R. M., & Valacich, J. S. (2008). Media, Tasks, and Communication Processes: A Theory of Media Synchronicity. MIS Quarterly, 32(3), 575-600.

Ebbinghaus, H. (1964). Memory: A Contribution to Experimental Psychology. New York: Dover.

English, M. C. W. & Visser, T. A. W. (2014). Exploring the Repetition Paradox: The Effects of Learning Context and Massed Repetition on Memory. Psychonomic Bulletin & Review, 21(4), 1026-1032.

George, J. F., Carlson, J. R., & Valacich, J. S. (2013). Media Selection as a Strategic Component of Communication. MIS Quarterly, 37(4), 1233-1251.

Hung, Y-T. C., Duyen, N. T. T., Kong, W-C., & Chua, A-L. (2008). Reexamining Media Capacity Theories Using Workplace Instant Messaging. IEEE Transactions on Professional Communication, 51(4), 352-368.

Kuhl, B. A. & Anderson, M. C. (2011). More is not Always Better: Paradoxical Effects of Repetition on Semantic Accessibility. Psychonomic Bulletin & Review, 18(5), 964-972.

Muhren, W. J., Van Den Eede, G., & Van de Walle, B. (2008). Making Sense of Media Synchronicity in Humanitarian Crises. IEEE Transactions on Professional Communication, 52(4), 377-397.

Scott, C. K. & Sarker, S. (2010). Examining the Role of the Communication Channel Interface and Recipient Characteristics on Knowledge Internalization: A Pragmatist View. IEEE Transactions on Professional Communication, 53(2), 116-131.

Siegel, S. (1956). Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill.

AUTHOR BIOGRAPHIES

Tim Chenoweth is an associate professor of Information Systems at Boise State University. He received his PhD from Washington State University in 1996. His research interests include issues related to big data and data analytics as well as IT education and security.

Kit Scott is an Assistant Professor of IT Management in the Information Technology and Supply Chain Management ddepartment at Boise Stat University. He received his doctorate from Washington State University. His research focuses on electronic media and communication effectiveness, human-computer interaction, and innovation and creativity. His work appears in such journals as IEEE Transactions on Professional Communication, Communications of the AIS, and The DATABASE for Advances in Information Systems. He teaches introduction to programming and project management.

Karen Corral is an Associate Professor in the Department of Information Technology and Supply Chain Management at the College of Business and Economics at Boise State University. She holds a BA in English from the University of Michigan, an MS in Computer Information Systems from Arizona State University, and a Ph.D. in Business Administration from Arizona State University. Her research interests are in the area of data and knowledge management as related to decision support. Her work has been published in journals such as Decision Sciences, Information Systems Frontiers, and Decision Support Systems.

Table 1: Experimental Task Ordering for Each Dataset

Dataset      Task Order

Semester 1   1) Students receive F2F
             instruction.
             2) Students take the pretest.
             3) Students complete the tutorial.
             4) Students take the posttest.

Semester 2   1) Students take the pretest.
             2) Students complete the tutorial.
             3) Students take the posttest.

Semester 3   1) Students take the pretest.
             2) Students complete the tutorial.
             3) Students receive F2F
             instruction.
             4) Students take the posttest.

Table 2: Hypotheses and treatments

Hypothesis                      Dataset Comparison

H1: Students who have not       Semester 1 pretest is greater
received the lecture or the     than Semester 2 pretest
tutorial will score better
after receiving only the
lecture.

H2: Students who have not       Semester 2 posttest is
received the lecture or the     greater than Semester 2
tutorial will score better      pretest
after viewing only the
tutorial.

H3: The order in which the      Semester 1 posttest is equal
lecture and tutorial are        to Semester 3 posttest
presented will not affect
pretest nor posttest scores.

H4: Students receiving the      Semester 1 pretest is equal
tutorial will score the same    to Semester 2 posttest
as students receiving the
lecture.

H5a: Students receiving         Semester 1 posttest is
both the tutorial and the       greater than Semester 1
lecture will score higher       pretest
than students receiving only
the lecture.

H5b: Students receiving         Semester 1 posttest is
both the tutorial and the       greater than Semester 2
lecture will score higher       posttest
than students receiving only
the tutorial.

Table 3: Descriptive Statistics

Dataset      N     Mean   Std. Deviation

Semester 1   12    3.83   1.03
Pretest
Semester 1   12    4.08   1.08
Posttest
Semester 2   20    1.80   0.89
Pretest
Semester 2   20    3.60   1.10
Posttest
Semester 3   17    3.24   1.30
Pretest
Semester 3   17    3.53   1.01
Posttest

Table 4: Hypothesis Tests

Hypothesis   t-stat   Sig.    Nonparametric   Effect
                              Test            Size

1            5.685    P <     sig. < 0.001    2.02
                      0.001
2            6.493    p <     sig. < 0.001    2.01
                      0.001
3            1 396                            0 51
                      0.176
4            0.606    P =     sig. = 0.554    0.22
                      0.550
5a           1.000    P =     sig. = 0.187    0.39
                      0.169
5b           1.217    P =     sig. = 0.117    0.43
                      0.118

Note: Hypotheses 1, 3, 4, and 5b were tested with
independent samples t-tests; Hypotheses 2 and 5a were
tested with paired samples t-tests.
COPYRIGHT 2016 Journal of Information Systems Education
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

 
Article Details
Printer friendly Cite/link Email Feedback
Author:Chenoweth, Tim; Corral, Karen; Scott, Kit
Publication:Journal of Information Systems Education
Article Type:Report
Date:Jan 1, 2016
Words:5650
Previous Article:Editor's message.
Next Article:Design and emergence of a pedagogical online InfoSec Laboratory as an ensemble artefact.
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters