Evaluating new technology for staff development.

Abstract

This study evaluated professional development workshops delivered to eight partner school districts by interactive video conferencing (IVC), a technology new to these districts. This study focused on attendees' (a) beliefs about their IVC workshops and outcomes of attendance, (b) their attitudes toward having attended the workshops, and (c) their intentions to attend future IVC workshops. Data were collected by mailed questionnaire and analyzed by correlation and multiple regression. Results show overwhelmingly favorable attitudes toward having attended the workshops and beliefs closely related to those attitudes.

Introduction

Interactive video conferencing (IVC) is not an especially new technology, but until fairly recently its costs have limited its use to the large organizations that could afford it (Bums, 2000). Falling prices and state and federal grants have greatly increased the number of IVC users, especially in education, and therefore, the importance of both formative and summative evaluations of IVC delivery. Fortunately a number of useful studies of distance education in general, and IVC in particular, were available to inform the present evaluation.

A recent text on distance learning (Simonson, Smaldino, Albright, & Zvacek, 2003) summarized the research on distance learning and reported its relative effectiveness-compared with traditional learning--for student achievement, and the importance of interaction. With regard to video conferencing specifically, Venn, Moore, and Gunter (1999) found it to be a low-cost alternative to off-campus travel to rural schools. Numerous studies have confirmed the importance of interaction (e.g., Shrestha & Sutphin, 2000; Carville & Mitchell, 2000). Further, video conferencing courses can be instrumental in the knowledge gain of attendees (Henning, 2001).

Project Background and Evaluation Rationale

The educational benefits of modern information and communications technology are well recognized. School districts across the United States either have acquired, or are trying to acquire, these technologies for their teachers, staff, and students. Interactive video conferencing (IVC) is a technology that is believed to have great potential for professional development particularly in rural and less-advantaged areas. A grant from the Texas Education Agency to the Region 5 Educational Service Center enabled eight southeast Texas school districts to establish two-way interactive video classrooms and connections to an extant educational telecommunications network. The goal was to provide professional development to teachers and other school personnel in the partner districts by an IVC technology delivery system. Most of the partner districts are small, rural, and low-income. These districts have few of the technological, professional development, or other resources typically available in larger, metropolitan districts. This grant has provided the partner districts access to the conferencing technology to access high quality professional development workshops previously unavailable to them.

Evaluation Objectives

Given the newness of IVC in these districts, it was believed essential to evaluate the use of this technology for professional development workshops, and attendees' perceptions of their IVC workshops were of special interest. An informal qualitative evaluation elicited workshop attendees' beliefs about the workshops shortly after each was concluded. The present study measured these beliefs, and other variables, after the newness of the technology had dissipated, and the attendees had had an opportunity to apply what they might have learned in their workshops.

Theoretical Framework

This study's investigation of beliefs, attitudes, and behavioral intentions, was guided by a theory of attitude formation and behavior decision-making (e.g., Fishbein, 1963; Fishbein & Ajzen, 1975). The theory has been applied to changing beliefs and attitudes (e.g., Lutz, 1973), and beliefs, attitudes, and actual behavior (e.g., McArdle, 1972). The theory has been applied to educational decision-making (e.g., Pryor, 1990; Thornburg & Pryor, 1998; Meyer, 2002; Kim, 2003), and used as a guide to the evaluation of distance learning (e.g., Pryor & Bitter, 2001). The theory has recently been made accessible for use by school leaders (Pryor & Pryor, 2005). Briefly, the theory holds that attitude toward an object--such as an IVC workshop--is formed by a set of beliefs about attributes of the object, and by a corresponding evaluation of each attribute (e.g., Fishbein, 1963). Attitude toward a behavior--such as attending a future IVC workshop--is formed by a set of beliefs about outcomes of performing the behavior and a corresponding evaluation of each outcome. Intention to perform a behavior--such as attending a future IVC workshop--is formed by attitude toward performing the behavior, and a perception of normative pressure concerning the behavior. More detail on the theory, as applied in this study, is given below in the Results section.

Method

Study participants were 108 employees of school districts in the project's service area, who had attended an IVC workshop in the past year. The largest group of participants was administrators (34 percent) both building- and district-level, followed by teachers (23 percent), support staff (19 percent), librarians (16 percent), and others (9 percent). Participants reported a mean level of 3.45 (out of 5 possible points; standard deviation of 1.21) on technology experience previous to their workshop. Our sampling plan was purposive: Only those workshops which had met multiple times over the course of a semester (usually five or more, but at least two) were selected for inclusion in the study. We believed that attendees of these workshops would have more certain beliefs about their workshops, and a greater likelihood of having learned and applied new knowledge. Of the 201 attendees invited to participate in the study, usable responses were received by 108 for a response rate of 53.7 percent.

To assess non-response bias, responses were divided into two groups, early and late responders, as suggested by Armstrong and Overton (1977). A T-square test found no significant overall difference between the groups, and t tests found no significant differences between the groups on five major variables, suggesting that non-response bias was minimal. (Major variables were perceptions of the utility of IVC, knowledge gain, attitude toward having taken the IVC workshop, intention to take another IVC workshop in the future, and technology experience prior to the IVC workshop.)

Data were collected by mailed questionnaire. The instrument consisted of 18 single-item, five-point, bipolar probability and evaluative scales, and five open ended items. The reliability of single-item scales, such as those used in this study, has been demonstrated (e.g., Pryor, 1999). Data were analyzed primarily by correlation and multiple regression.

Results and Discussion

The results are reported and discussed in three main areas: (a) participant beliefs about their workshops, (b) their attitudes toward the workshops, and (c) their intentions to attend a future IVC workshop. Study variables are italicized below for clarity. Although correlations between the variables are reported, no attempt is made to argue that these are all causal relationships. In certain instances, however, a reasonable argument for causality can be made, and readers must judge the plausibility for themselves. Generally, only correlations equaling or exceeding .50 (accounting for at least 25 percent of the variance) are reported.

Beliefs

Attribute Beliefs

The theory of reasoned action discriminates between beliefs about attributes of an object (e.g., an IVC workshop) and beliefs about outcomes of a behavior (e.g., attending an IVC workshop). The qualitative responses collected immediately after each workshop were primarily beliefs about attributes of the IVC workshops. The convenience, and savings in time and cost from the use of IVC were the benefits most salient to attendees. In addition, many attendees commented that they would not have been given release time by their districts to attend a traditional face-to-face training. The only problem noted by attendees was the occasional interruption of transmission by technical problems. These qualitative responses, and other beliefs, were content analyzed into belief statements that were then attached to scales in the present study. The responses to these quantitative items are reported in Table 1. See issue website http://rapidintellect.com/AEQweb/sum2006.htm

Over 78 percent of participants believed four aspects of their workshop related to the greater efficiency of IVC over traditional professional development (saved driving time, more convenient than traditional workshop, saved time away from school, and got questions answered easily). Over 76 percent of participants believed they had learned information useful for their jobs, and nearly 60 percent of participants believed they could not have attended the workshop if it had been offered as traditional face-to-face professional development.

Outcome Beliefs

The theory of reasoned action is also concerned with beliefs about likely outcomes of performing a behavior (e.g., Fishbein & Ajzen, 1975). Three outcomes were not elicited from workshop participants, but were included in the questionnaire because of their importance to the study: knowledge gain, performance gain, and student performance gain.

Knowledge gain. For instruction, the amount students learn is--arguably--the primary evaluative dimension. Although self-report measures of knowledge gain are not so widely accepted as actual measures of gain, there is evidence of their validity. Campbell and Stanley (1966) suggested that self-report measures of knowledge gain tend to disguise rather than exaggerate, actual knowledge gain, and this has been confirmed in studies of technology mediated staff development (e.g., Al-Mekhlafi, 2000; Bitter, 1993). Henning (2001) found that video courses can be instrumental in knowledge gain, and that finding was confirmed in this study. Nearly 77 percent of participants believed they had learned information at the workshop that was useful to their jobs. Participants rated their knowledge gain at a mean of 4.32 (standard deviation of .86), which is rather high given the maximum possible score of 5 points. A correlation analysis investigated the relative strength of possible predictors of knowledge gain, and 16 variables correlated significantly (with a p value of less than .01). Not surprisingly, learning information that improved job performance was most closely related to knowledge gain (correlation of .66), followed by the quality of the speaker (correlation of .61), and by the comfort of the physical facilities (correlation of .59). Concerning physical facilities, it is important to note that each of the attending school districts had dedicated a classroom to distance learning, renovated the room, and installed new technology and furniture, all to better serve interactive video conferencing. The fourth variable related to the first and was improved job performance resulted in improved student performance (correlation of .58).

Performance gain. Although knowledge gain is important, in professional development it could be argued that improved on-the-job performance is the ultimate dimension for evaluation. This study also examined whether participants believed their knowledge gain had been translated into improved performance on the job. Although 77 percent of participants believed they had learned job-useful information, understandably fewer participants (just under 67 percent) reported improved job performance resulting from this learning. A correlation analysis investigated the relative strength of possible predictors of performance gain, and 16 variables correlated significantly (with a p value of less than .01). Six of these correlated at .50 or higher. Of these, four were plausibly related in a causal manner: (a) I learned information useful for job (correlation of .66), (b) the quality of the presenter (correlation of .62), (c) the comfort of the physical facilities (correlation of .59), and (d) IVC workshop saved driving time (correlation of .52).

Student performance gain. Although performance gain is important, in educational professional development it could be argued that improved student performance is truly the ultimate dimension for evaluation. This study also examined whether employee performance gain had been translated into improved student performance. Although just under 67 percent reported improved job performance, only 50 percent of all participants reported that this resulted in improved student performance. A correlation analysis investigated the relative strength of possible predictors of student performance gain, and 16 variables correlated significantly (with a p value of less than .01). Of these, five were plausibly related in a causal manner: (a) I learned information that improved job performance (correlation of .88), (b) I learned information useful for job (correlation of .58), (c) the quality of the presenter (correlation of .58), (d) the comfort of the physical facilities (correlation of .56), and (e) IVC workshop saved driving time (correlation of .50).

Attitudes

Attitude toward Having Attended the IVC Workshop

Attitude is positive, negative, or neutral affect toward a given object or behavior. The theory that guided this study holds that attitude toward performing a future behavior is formed by a set of beliefs about likely outcomes of the behavior. An attitude toward having already performed a behavior is, therefore, formed by a set of beliefs about outcomes of having performed the behavior. More than 81 percent of participants had a favorable attitude toward having attended the workshop. The mean score on the five-point bipolar scale of attitude toward having attended the IVC workshop was 4.34 (with a standard deviation of 0.90), which is between favorable (4.00) and strongly favorable (5.00). It was beyond the scope of this study to elicit and measure the outcomes participants' believed followed their participation in the IVC workshops, and therefore, formed their attitudes toward having attended the workshops. Some useful understanding of their attitudes, however, might be obtained by knowledge of the extent to which they agreed or disagreed with the beliefs elicited from them after their workshops, as reported in Table 2. See issue website http://rapidintellect.com/AEQweb/sum2006.htm

To determine how well these 13 beliefs explained attitude toward having attended the workshop, attitude was regressed on the 13 significant predictors reported in Table 2. The regression analysis yielded a multiple correlation of .832, accounting for just over 64 percent of the variance in attitude toward having taken the workshop (the F value was 13.888; degrees of freedom were 13,80; and the p value was less than .001).

Two variables obtained significant beta weights (a) IVC workshop more useful than traditional workshop (beta was .376), and (b) IVC workshop more convenient than traditional workshop (beta was .266), and two more obtained the next highest beta weights (c) improved job performance resulted in improved student performance (beta was. 180, p was .065), and (d) got useful information for job (beta was .180, p was .085). Using just these four variables as predictors in a second regression analysis produced a multiple correlation of .790, accounting for just over 60 percent of the variance in attitude toward having taken the workshop. Three of the four predictors obtained significant beta weights in this analysis: (a) IVC workshop more useful than traditional workshop (beta was .429), and (b) IVC workshop more convenient than traditional workshop (beta was .274), and (c) got useful information for job (beta was .206). This analysis suggests that the perceived utility and convenience of IVC workshops might have influenced the very favorable attitudes of participants toward having attended their workshops.

Attitude toward Attending a Future IVC Workshop

Nearly 78 percent of participants had a favorable attitude toward attending a future IVC workshop, slightly lower than the 81 percent with a favorable attitude toward having attended the initial IVC workshop. This difference might be due to the fact that the "future workshop" was entirely hypothetical at the time of data collection: no information on its content, presenter, or dates was provided to participants. The mean score on the five-point bipolar scale of attitude toward attending a future IVC workshop was 4.32 (standard deviation of 1.06), which is between favorable (4.00) and strongly favorable (5.00).

Intentions to Attend Future IVC Workshop

This study also investigated the participants' intentions to participate in a future professional development workshop offered by IVC, and some possible influences on those intentions. Although IVC can provide high quality professional development for school personnel in rural and low-income districts that would otherwise be available, little is known about how these personnel respond to such workshops, in particular, whether or not they will continue their participation after the initial experience. This aspect of the study was also guided by a theory of attitude and decision-making, described above. Nearly 43 percent of participants were likely to attend a hypothetical future IVC workshop. The mean score on the intention measure was 3.53 with a standard deviation of 1.25, the largest standard deviation of those on all variables but one. (The hypothetical nature of the workshop might explain why very favorable attitudes and norms predicted less-favorable intentions.) Three analyses were conducted.

The first analysis was the examination of variables that correlated significantly with intention to participate. As the decision-making theory predicts, the variables that correlated most highly with intention were attitude toward attending (correlation of .71; p value less than .01), followed by normative pressure to attend (correlation of .57; p value of less than .01) ("People important to me think I should attend ..."). The next most highly correlated variables were less strongly related (both correlated at .47): (a) belief that attending an IVC workshop (as opposed to a traditional workshop) saved the participant from taking time away from school, and (b) attitude toward having taken last spring's IVC workshop.

The second analysis was the investigation of the relative influence of intention's theoretical predictors, attitude toward attending, and perceived normative pressure to attend. The measure of intention was regressed on measures of attitude and norm, resulting in a multiple correlation of .722, accounting for just over 51 percent of the variance in intention (F value of 57.064; degrees of freedom were 2,105; p value of less than .001). Attitude achieved a beta weight of .601 (p value of less than .001), norm achieved a non-significant beta weight of .166 (p value of .067). Although normative pressure failed to achieve significance in the regression, the alpha level was sufficiently low that it would not be unreasonable to expect that norm might achieve significance at a later time when participants had more information about their opportunities to attend future IVC workshops.

Given the importance of attitude toward attending, the next step was the examination of variables that correlated significantly (p value less than .01) with attitude. The most highly correlated variable was intention to attend (correlation of .71), followed by norm (correlation of .67), and by attitude toward having attended last spring's workshop (correlation of .60). The remaining three correlates suggest beliefs that might have been outcomes of having attended the initial IVC workshop: (a) IVC workshop saved time away from school (correlation of .55), (b) IVC workshop more convenient than traditional workshop (correlation of .53), and (c) I got questions answered as easily as in a traditional workshop (correlation of .50). The fact that attitude toward having taken last spring's workshop correlated as well as it did was not surprising; a previous evaluation study (Pryor, 1999) had found the same phenomenon. It seems intuitive that when former attendees at a workshop are asked about their attitudes toward attending an unspecified future workshop by the same sponsor, they are likely to base their attitudes largely on beliefs about outcomes of having attended the previous workshop. The scores of those with the most favorable attitudes toward future participation (scores of 5) were compared with those whose scores were less than five. On the remaining 17 variables, there were 14 significant differences between the scores of the two groups. On each of the 14 variables the highly favorable group had the higher score.

Conclusion

In a time of budget and time constraints in education, distance learning by IVC appears to be an effective alternative to traditional professional development, especially in isolated rural, disadvantaged areas. This study provides insight into the perspectives of new attendees in IVC professional development workshops, including the likelihood of their future attendance at such workshops. The study examined relationships between the direct predictors of future attendance, and beliefs about the initial workshop attended, to suggest some possible influences on intentions to attend. These results provide strong support for the use of IVC technology for professional development of these and other users new to the technology. The participants in this study believed overwhelmingly that IVC was more effective than traditional programming, more convenient, that it saved them driving time, and time away from school, and that they learned information useful for their jobs.

The educational importance of this project is that it is provides much needed access to training and technological resources to school districts that were previously underserved technologically. The educational importance of this evaluation is that it provides an insight into the perspectives of these new users of IVC and suggests the attendees' satisfaction with IVC workshops, and perception of benefits. These results also provide useful information for future program development and promotion of technology-mediated professional development. Future research and evaluation of future new technologies can benefit by including the variables and methods applied in this study to gain important understandings of new user perspectives. When feasible, future studies should apply the entire theory of reasoned action (see Pryor & Pryor, 2005), so that the specific beliefs underlying attitude and norm can be investigated in more depth.

References

Al-Mekhlafi, A. G. (2000). Instructional media for teachers' preparation. International Journal of Instructional Media, 28(2), 191-207.

Armstrong, J. S., & Overton, T. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 14, 396-402.

Bitter, G. G. (1993). Teaching mathematics methods using interactive videodisc: Research results from the TMMUIV project. Monograph of Technology Based Learning and Research, Arizona State University (Serial No. 5).

Burns, J. T. (2000). Toward an effective virtual learning environment. The Quarterly Review of Distance Education, 1(2), 119-127.

Campbell, D., & Stanley, J. (1966). Experimental and quasi-experimental designs for research. Boston, MA: Houghton Mifflin.

Carville, S. & Mitchell, D. (2000). It's a bit like Star Trek: The effectiveness of video conferencing. Innovations in Education and Training International. 37, 42-49.

Fishbein, M. (1963). An investigation of the relationships between beliefs about an object and attitude toward that object. Human Relations, 16, 233-240.

Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.

Henning, M. (2001). Closing the gap: Using conferencing software to connect distance education students and faculty. Journal of Library Administration. 32, 233-246.

Kim, H. (2003, April). Factors of forming teachers' intention to use the internet in class. Paper presented at the meeting of the American Educational Research Association, Chicago.

Ludlow, B. L. (I 994). A comparison of traditional and distance education models. Proceedings of the Annual National Conference of the American Council on Rural Special Education, Austin, TX.

Lutz, R. J. (1973). Cognitive change and attitude change: A validation study. Unpublished doctoral dissertation, University of Illinois, Urbana.

McArdle, J. B. (1972). Positive and negative communications and subsequent attitude and behavior change in alcoholics. Unpublished doctoral dissertation, University of Illinois, Urbana.

Meyer, L. (2002). Applying the theory of planned behavior: Nursing students' intention to seek clinical experiences using the essential clinical behavior database. Journal of Nursing Education, 41(3), 107-116.

Pryor, B. W. (1990). Predicting and explaining intentions to participate in continuing education: An application of the theory of reasoned action. Adult Education Quarterly, 40, 146-157.

Pryor, B. W. (1999). Three longitudinal impact evaluations of continuing library education: Participant satisfaction, program effects, and future participation. Journal of Education for Library and Information Science, 40(1), 10-26.

Pryor, B. W., & Bitter, G. G. (2001, February). A Theory of Attitude Formation as a Guide for Educational Research: Examples from Studies of The e-Learning Network. Paper presented at the meeting of the Southwest Educational Research Association, New Orleans, LA.

Pryor, B. W., & Pryor, C. R. (2005). The school leader's guide to understanding attitude and influencing behavior: Working with teachers, parents, students, and the community. Thousand Oaks, CA: Corwin.

Shrestha, G. & Sutphin, H. D. (2000). Relationship between interaction and acceptance in satellite video-conferencing. Journal of Educational Technology Systems. 28, 43-58.

Simonson, M., Smaldino, S., Albright, M., & Zvacek, S. (2003). Teaching and learning at a distance: Foundations of distance education (2nd Edition). Columbus, OH: Merrill Prentice Hall.

Thornburg, G. E., & Pryor, B. W. (1998). Attitudinal and normative predictors of continuing library education: An application of the theory of reasoned action. Journal of Education for Library and Information Science, 39 (2), 118-133.

Venn, M., Moore, R.L., Gunter, P. (1999). Using audio/video conferencing to observe fieldbased practice of rural teachers. Rural Educator, 22 (2), 24-27.

Brandt W. Pryor, Pryors Educational Services, Edwardsville, IL

Carolyn H. Crawford, Lamar University, TX

Desmond V. Rice, Lamar University, TX

Caroline R. Pryor, Southern Illinois University Edwardsville 

Dr. B. Pryor specializes in attitude-behavior research. Drs. Crawford and Rice are associate professors in the Department of Educational Leadership. Dr. C. Pryor is assistant professor, Department of Curriculum and Instruction, and Wye Fellow of the Aspen Institute. The Pryors recently wrote The school leader's guide to understanding attitude and influencing behavior (Corwin Press).