Printer Friendly

Evaluation of alternative media used with a groupware editor in a simulated telecommunications environment.

Writing efficiency using a groupware editor was evaluated in six configurations by factorially combining three communication media (audio only, audio plus video, and face-to-face) each with and without groupware editing in a within-subjects design. Twelve subjects familiar with journalistic writing were matched into dyads and wrote news articles based on sets of questions about video clips shown to them. The quality of the writing task products was consistently high and showed no differences among conditions. Results indicated that face-to-face conditions resulted in significantly less time to complete the task than did other communication conditions. When groupware editing was used, however, all communication media were equally efficient. The results were discussed in terms of the benefits of using the groupware editor to increase efficiency when face-to-face communication is not practical. Methodological procedures were discussed as a means of reducing team variability in evaluating groupware using telecommunications.

INTRODUCTION

A new type of software, termed a groupware editor, allows multiple users to create and edit a single document simultaneously (Olson, Mack, and Wellner, 1990). This software is a member of the groupware family of tools (Gibbs, 1989) used for some types of computer-based telecommunications environments. It has ostensibly been developed to increase efficiency in coauthoring tasks in which users may not be co-located. However, questions as to the effectiveness of this type of cooperative work aid still need to be evaluated empirically.

Little objective data have been published concerning usability of particular groupware editors or interactions between such systems and other telecommunication media. Most previous studies describe only observational data of teams at work. Gale (1991) noted excessive variability and no significant differences in user performance in a field study of a group whiteboard system versus other communication conditions. Smith, O'Shea, O'Malley, Scanlon, and Taylor (1991) developed a video tunnel designed to convince users that they were communicating face-to-face through a video system used for team problem solving. However, they reported high variability in their data and no significant performance effects.

A few publications have described research methodologies for evaluating types of group work similar to the groupware editor. Ishii (1990), in his discussion of a groupware package called TeamWorkstation, mentions that the video and audio media help with the establishment of floor control (i.e., who has the permission to edit) and other social protocol issues. Tang and Minneman (1990) mention the usefulness of the display of hand gestures in floor control. Watabe, Sakata, Maeno, Fukuoka, and Ohmori (1990) cite anecdotal results concerning both audio and video channels in their system, MERMAID. The audio channel was the most frequently used, and the video channel helped participants become more familiar with one another and accelerated informal discussions.

Work on Teamworkstation by Ishii and Miyake (1991) does provide some objective statistical data concerning their system. Two different modes of work space were compared: an overlaid picture mode and a teledesk mode. The overlaid picture mode is an analog combination of video signals from the two workstations, whereas the teledesk mode is a digital representation of the signals from the workstations. These two modes were compared in a mixed-factor design with order of mode usage and instruction in a calligraphy task. The overlay mode was found to be subjectively preferable and resulted in reduced task completion time.

The high levels of variability in the experimental data shown in previous group work studies influenced the methodology developed for use in the present experiment. Group writing has not been addressed previously because of problems in evaluating writing and the lack of sufficient examples of groupware editors. Our study was designed to develop a method for investigating groupware editors and to evaluate writing efficiency for telecommunications environments that include a groupware editor.

METHOD

Subjects

Twelve college students with intermediate expertise in journalistic writing were matched into six dyads. Every subject was a communication major at Virginia Tech and was familiar with the news article reporting structure. Members of a dyad were unfamiliar with each other and had similar scores on dominance (Jackson, 1987) and writing ability (Woodcock and Johnson, 1990) pretests. Subjects were matched in an attempt to reduce variability resulting from an unequal distribution of work within a dyad. Matching of unfamiliar subjects was also necessary to ensure equal initial levels of subject communication.

Task

Each dyad was asked to write six short news articles based on observations of six different video clips containing actors and objects derived from Video Clue II(TM) by Parker Brothers. A set of questions developed for each video clip and pretested for difficulty was given to the dyad after viewing the video clip. Depending on each participant's availability, dyads completed one or two news articles a day with fewer than three days between sessions. A news article style of reporting was used because it is highly restrictive. We hoped this type of task would reduce variability resulting from writing styles of the individual team members. This task was pretested with two dyads to refine the experimental procedures.

Experimental Design

We examined the writing efficiency of two-person teams using a groupware editor, Aspects(TM) from Group Technologies, in conjunction with three communication media. The treatment conditions in this experiment were chosen to emulate possible real-world uses of groupware and communication media. The factor of Groupware Editor had two levels: groupware editing and single-user editing. In single-user editing, only one member of the dyad used the groupware editor. The factor of media type had three levels: audio only, audio plus video, and face-to-face. A two-factor, within-subjects analysis of variance design was used.

The 720 possible orders of the resulting six treatment presentations in this within-subjects design could not be totally counterbalanced because of the large number of orders. Alternatively, a balanced Latin square, as shown in Table 1, was used for ordering treatment conditions. The six different video clips were completely confounded with presentation order. Note that this balancing scheme allows each of the six experimental conditions to occur once in each order across the six dyads. In addition, each of the six conditions preceded and followed each of the other five conditions once across all dyads.

TABLE I

Counterbalancing of the Six Treatment Conditions Using a Balanced Latin Square

                     Six Dyad Groups

Treatment Order   1   2   3   4   5   6

       1          1   2   3   4   5   6
       2          2   3   4   5   6   1
       3          6   1   2   3   4   5
       4          3   4   5   6   1   2
       5          5   6   1   2   3   4
       6          4   5   6   1   2   3




Note. Condition 1 = single-user editing and audio-only communication, condition 2 = single-user editing and audio-plus-video communication, Condition 3 = single-user editing and face-to-face communication, Condition 4 = groupware editing and audio-only communication, Condition 5 = groupware editing and audio-plus-video communication, and Condition 6 = groupware editing and face-to-face communication.

Apparatus

Two offices connected by simulated computer-based telecommunications were used. Each office workstation had a computer running the groupware editor, a closed-circuit standard headset telephone, and a monochrome video monitor and camera, as shown in Figure 1. All of this equipment was connected to the other office and the experimenter monitoring station. The video equipment was positioned in an attempt to provide the illusion of face-to-face contact, as suggested by Smith et al. (1991). For face-to-face sessions with groupware editing, two computers in one office were utilized. Only those communication devices necessary for a given session were active. All sessions were recorded on videotape and timed. In addition, all written articles were saved on computer disk.

Procedures

Subjects were pretested for matching, trained with the groupware editor, and given a practice session on the writing task. No discussion was allowed within a dyad during observation of the video clips, nor was it allowed outside the experimental setting. This was enforced to aid in maintaining constant levels of within-dyad familiarity. One member of each dyad was randomly chosen to be the typist in those conditions that used single-user editing. Only one subject could enter the text of the document in these conditions.

The content of the news article to be produced for each session was structured through sets of questions. These questions were developed to be of equal difficulty and presented in random order within each set. This was done to ensure comparable product content across teams and conditions. Subjects were instructed to embed answers to the provided questions in their article in the chronological order in which the events involved occurred in the video, and all conventions of journalistic style were to be followed. Subjects were instructed to produce the highest-quality articles possible and, secondarily, to complete them in the shortest time possible. Subjects also completed a questionnaire about the experimental conditions and software following completion of the study. Subjects were paid $5 for each experimental session.

RESULTS

Quality of the written news articles was assessed in terms of grammatical, factual, and chronological accuracy. Grammatical quality of the written articles was found to be at a level of less than three unique errors per document using the computerized grammer checker (Language Systems, 1990). Therefore, no statistical analysis was warranted. Standard questions were developed so that the correct answers and the order of events would be unambiguous. Factual correctness of the news articles was measured by the number of facts placed in the written news article in response to questions that required single-fact answers. Likewise, chronological accuracy was measured by the number of these same facts that were in the same chronological order as shown in the video clip. These two measures of news article quality showed no significant differences (p > 0.05) on any of the six F tests according to analysis of variance (ANOVA). Consequently, the resulting consistent quality of documents allowed time to complete the articles to be used as the primary measure of work efficiency under the assumption that product quality divided by writing time equals efficiency.

A single-observation Latin-square ANOVA showed significant effects of both trial presentation order, F(5,20) = 5.27, p < 0.05, and team differences, F(5,20) = 7.01, p < 0.01, in terms of task completion time. A Greenhouse-Geiser correction was used on all subsequent ANOVAs of the task completion time data because of heterogeneity of covariance. All post hoc paired comparisons on significant main effects and interactions were evaluated by the least significant difference test (p < 0.05).

Results from the task completion time data indicated a significant main effect of communication media, with the face-to-face communication alternative significantly more efficient than either of the other two media alternatives F(2,10) = 6.33, p < 0.05. Mean times (in minutes) to complete the task for the three levels of this factor were: 31.57 (audio only); 31.50 (audio plus video); and 25.54 (face-to-face).

No significant differences in completion time were found between the non-face-to-face conditions with groupware editing and the face-to-face communications with single-user editing, F(2,10) = 0.73, p < 0.20. A subsequent F[MAX] test was conducted to determine differences in variance, F(5,6) = 32.58, p < 0.05. The audio-only and the audio-plus-video communication conditions that used groupware editing both had significantly less variable performance than the audio-plus-video communication condition with single-user editing (critical difference: 18.7, p < 0.05). Both mean times and the [plus or minus] 1 standard deviation error bars of task completion time for the Groupware Editor X Communication Media interaction are shown in Figure 2.

Subjective data obtained from questionnaires, including Likert scale ratings and open-ended comments, supported the objective results of task completion time. Video communication was considered unnecessary, face-to-face communication was preferred, conditions with the groupware editor were considered more productive than other conditions, and simultaneous typing was not considered distracting. Responses supporting these statements were significantly different from a uniform distribution by a Kolmogorov-Smirnov analysis (C[MAX] = 0.60, p < 0.05). ANOVAs revealed that subjects preferred trials with the groupware editor, F(1,11) = 67.31, p < 0.01, and that a feature supporting text messaging between members of a dyad was significantly less "essential to work" than other features, F(3,33) = 8.32, p < 0.01. Open-ended comments by subjects indicated major problems of slow system response and a lack of keyboard cursor movement control.

DISCUSSION

Overall, face-to-face communications were more efficient than the simulated telecommunication environments. This finding agrees with early communications research (e.g., Weeks and Chapanis, 1976). Interestingly, the lack of difference shown between the face-to-face condition with single-user editing and other groupware editing alternatives indicates that writing efficiency can be maintained by the use of a groupware editor in computer-based telecommunication situations in which users are not colocated. More important, merely allowing auditory communications among team members performing groupware editing resulted in task efficiency equal to face-to-face communication. Consequently, high bandwidth video was not necessary in this particular telecommunication task, which involved highly structured, collaborative writing.

The reduced variability in task completion time for certain groupware editing conditions may have been attributable either to communication structure or to the leveling effects of group writing. The significant effect of differences among teams may have been attributable to consistent style differences among dyads that affected efficiency levels. When the groupware editing was not allowed, the user selected to type inputs completed a larger share of the work. The randomly chosen lead typist may not have been the most efficient member of the team.

Positive opinions about the groupware editor support the assertion that a significant effect of groupware editing was obscured by the time data variability. Other opinion data concerning the media conditions agree with results from the time data. Improving the system response time and cursor movement problems noted by the users may help in specifying an improved interface for groupware editing.

One main conclusion gained from this study is that it is practical to perform objective research on groupware. A structured writing task proved useful in conducting this type of research. The consistent levels of quality throughout the study indicate that no special training is needed to prevent quality decrements from this new writing paradigm. From a methodological point of view, the use of a within-subjects design, matching of subjects, and the restrictive journalistic writing style all contributed to reducing variability in the time data to the extent that certain effects could show significant differences. Even so, variability was still high. Perhaps a larger sample of dyads is needed to provide the statistical power to demonstrate differential effects of the groupware editor alternatives.

The generalizability of this type of structured experimental task and results from this particular groupware editor also need to be extended across a variety of team communication tasks and various features of groupware editors. This research used a highly structured coauthoring task that emphasized verbal collaboration. Other communication tasks (e.g., decision making, problem solving, consensus building, crisis management, persuasive presentations) in which the participants are not co-located may require more emphasis on other channels of telecommunication to support group writing. For example, taxonomies of various office automation tasks similar to those in Czaja (1987) need to be extended, and the value of alternative telecommunications media across these tasks needs to be evaluated empirically. Nonetheless, the task developed in this study may aid in establishing a general methodology for evaluating groupware editors and group writing in computer-based telecommunication environments.

Evaluation also needs to be extended beyond group work involving dyad coauthoring to multiauthoring environments. Of particular importance is the situation in which two or more individuals are co-located in a face-to-face conference and the remaining team member is remotely located but participates in the coauthoring through computer-based groupware support. Video access may be necessary in this more complex scenario in order to facilitate effective communication. However, providing accurate eye contact for teams consisting of three or more persons at different locations using video presents significant challenges (e.g., Bocker and Muhlbach, 1993). Consequently, video communication in telecommunications environments needs additional research in order to optimize the presentation and use of this medium.

In conclusion, the overall results of this study suggest that groupware editing may result in efficiency and reductions in variability for highly structured coauthoring tasks when face-to-face communication is not practical or is not possible. For example, electronic communication is necessary for remote office work (Olson, 1983; Olson and Bly, 1991), and electronic alternatives to face-to-face communication are necessary to allow participation of individuals with mobility impairments who cannot physically travel to group meetings (Williges and Williges, 1995). Consequently, additional research is necessary to enhance groupware for work alternatives in which electronic communication provides the only means of participation.

ACKNOWLEDGMENTS

This study was based on a master's thesis conducted by the first author. The authors thank Beverly H. Williges for her technical advice and support. In addition, the authors acknowledge the helpful suggestions provided by David Green and the two technical reviewers on a draft of the article.

REFERENCES

Bocker, M., and Muhlbach, L. (1993). Communicative presence in video communications. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 249-253). Santa Monica, CA: Human Factors and Ergonomics Society.

Czaja, S. J. (1987). Human factors in office automation. In G. Salvendy (Ed.), Handbook of human factors (pp. 1587-1616). New York: Wiley.

Gale, S. (1991). Adding audio and video to an office environment. In J. M. Bowers and S. D. Benford (Eds.), Studies in computer-supported cooperative work: Theory practice and design (pp. 49-62). Amsterdam: Elsevier.

Gibbs, S. J. (1989). LIZA: An extensible groupware toolkit. In Proceedings of the ACM Conference on Computer-Human Interaction (pp. 29-35). New York: Association for Computing Machinery.

Ishii, H. (1990). Teamworkstation: Towards a seamless shared workspace. In CSCW '90: Proceedings of the Conference on Computer-Supported Cooperative Work (pp. 13-26). New York: Association for Computing Machinery's Special Interest Groups on Computer-Human Interaction and Office Information Systems.

Ishii, H., and Miyake, N. (1991). Toward an open shared workspace. Communications of the ACM, 34(12), 42-50.

Jackson, D. N. (1987). Personality research form (3rd ed.). New York: Research Psychologists.

Language Systems [Computer software]. (1990). CorrecText. New York: Houghton Mifflin.

Olson, M. H. (1983). Remote office work: Changing work patterns in space and time. Communications of the ACM, 26,182-187.

Olson, M. H., and Bly, S. A. (1991). The Portland experience: A report on a distributed research group. International Journal of Man-Machine Studies, 34, 211-228.

Olson, J. S., Mack, L. A., and Wellner, P. (1990). Concurrent editing: The groups interface. In Human-computer Interaction -- INTERACT'90. New York: Elsevier.

Smith, R. B., O'Shea, T., O'Malley, C., Scanlon, E., and Taylor, J. (1991). Preliminary experiments with a distributed, multi-media, problem solving environment. in J. M. Bowers and S. D. Benford (Eds.), Studies in computer supported cooperative work: Theory practice and design (pp. 31-48). New York: Elsevier.

Tang, J. C., and Minneman, S. L. (1990). VideoDraw: A video interface for collaborative drawing. In Proceedings of the ACM Conference on Computer Human Interaction (pp. 313-320). New York: Association for Computing Machinery's Special Interest Group on Computer-Human Interaction.

Watabe, K., Sakata, S., Maeno, K., Fukuoka, H., and Ohmori, T. (1990). Distributed multiparty desktop conferencing system: MERMAID. In CSCW '90: Proceedings of the Conference on Computer-Supported Cooperative Work (pp. 27-38). New York: Association for Computing Machinery's Special Interest Groups on Computer-Human Interaction and Office Information Systems.

Weeks, G. D., and Chapanis, A. (1976). Cooperative versus conflictive problem solving in three telecommunication modes. Perceptual and Motor Skills, 42, 879-917.

Williges, R.C., and Williges, B.H. (1995). Travel alternatives for the mobility impaired: The surrogate electronic traveler (SET). In A. D. N. Edwards (Ed.), Extra-ordinary human-computer interaction. Cambridge, England: Cambridge University Press.

Woodcock, R. W., and Johnson, M. B. (1990). Woodcock-Johnson psycho-educational battery -- revised. Allen, TX: DLM Teaching Resources.

(1) Request for reprints should be sent to Robert C. Williges, 302 Whittemore Hall, Virginia Tech, Blacksburg, VA 24061-0118.
COPYRIGHT 1995 Human Factors and Ergonomics Society
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1995 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Special Issue: Telecommunications
Author:Green, Charles A.; Williges, Robert C.
Publication:Human Factors
Date:Jun 1, 1995
Words:3285
Previous Article:Using natural language conventions in the user interface design of automatic speech recognition systems.
Next Article:Telepresence in videocommunications: a study on stereoscopy and individual eye contact.
Topics:

Terms of use | Privacy policy | Copyright © 2022 Farlex, Inc. | Feedback | For webmasters |