Effective models for assessing the costs of educational technologies.

Abstract

Better information about how to use new technologies more effectively in higher education is essential, especially given the current rapidly changing environment in higher education. This paper summarizes the results of two field tests of cost assessment models at Washington State University (1) a focused project using the Flashlight Cost Model to analyze three technology strategies along the dimensions of cost efficiency, pedagogical effectiveness and flexibility; (2) a study of the costs of designing, developing, and delivering distance courses delivered on the WWW using new technologies. The study was conducted using the Technology Costing Methodology developed in conjunction by the WCET and NCHEMS. The information each model provides has given WSU key insights into new educational technologies.

Introduction

Since ignoring new technologies in higher education is not an option, educational institutions across the country are now searching for effective strategies for investing in new technologies and analyzing the cost of courses which use the new technologies. No strategy promises to be cheap. The challenge is to find or develop approaches that are effective relative to cost. This paper outlines Washington State University's (WSU) experience field testing two models designed specifically to analyze costs, especially costs of new technologies in education. The first model tested was the Flashlight Cost Model which was developed by the Flashlight Program of the Teaching, Learning and Technology Group of the American Association of Higher Education. The second model applied was the Technology Costing Methodology (TCM) model. The National Center for Higher Education Management Systems (NCHEMS) is participating with the Western Cooperative for Higher Education (WCET) to develop the Technology Costing Methodology. The Project will produce a costing methodology (and related procedures) for calculating technology costs both within an institution and across institutions. Both models apply an accounting concept called Activity-based Costing (ABC) to analyze new technologies and/or delivery techniques in education.

Related Works

The work summarized in the following paper relies heavily on two handbooks, The Flashlight Cost Analysis Handbook - Version 1 (Ehrmann and Milam, 1999) and a preliminary draft of the Technology Costing Methodology Handbook (Jones, 2000). The Flashlight Cost Model handbook is available for a small fee. At the time of this writing, a second edition was in process. Information about the model can be found from the TLT Group web site: http://www.tltgroup.org/. The TCM manual was in the process of being finalized at the time this article was submitted for review. When the TCM Manual is completed it will be available on the World Wide Web. The following URL should contain information about the manual: http://www.wiche.edu/telecom/Projects/tcm/index.htm. The paper below summarizes some of the differences between the two handbooks. However, the authors experience indicates that the handbooks compliment each other very well. The WCET plans to make available a casebook as well as the cost handbook summarizing field tests by institutions from around the U.S. The casebook, in addition to the Flashlight handbooks, when completed, will contain about 20 case studies. These case studies should provide potential users very valuable insights into cost studies that adapt ABC.

It is interesting that most of the applications of activity-based costing to education are occurring in the English-speaking world, specifically; Australia, Canada, Great Britain, Hong Kong and the U.S. An excellent review of cost models can be found in a study conducted at Sheffield-Hallam University (Bacsich, et. al., 1999). The focus of the study was to determine the hidden costs in networked learning. The study is extensive (100 pages) and valuable' in that it provides an excellent literature review and bibliography and contains other interesting material such as case studies, survey results, etc. The Costs of Networked Learning (CNL) project is beginning a new phase and people interested in such studies may want to check the web site for further analysis and materials. The authors recommend ordering a copy of the original study.

Readers might also be interested in a grant report detailing an application of ABC to a single department of a major U.S. university (Granof, Platt & Vaysman, 2000). The grant was funded by the PricewaterhouseCoopers Endowment for The Business of Government. The report details a case study that applies ABC to a specific department. However, readers interested in other applications of ABC may also fin this report useful. The authors found a section titled "Lessons Learned" to be particularly interesting and insightful. The Flashlight Cost Analysis Handbook - Version One is available to the public at a cost (currently $35.00) while the other studies cited above are or will be available on the World Wide Web.

Overview of the Flashlight Cost Model and the Technology Costing Methodology

Both the Flashlight Cost Model and the TCM were developed for the purpose of analyzing the costs of new technologies being used in education. Traditional accounting methods summarize costs but do not go into detail about the activities that the costs were used to produce. ABC is a method to relate costs to specific outputs and enable managers to better analyze that relation. In the PricewaterhouseCoopers study it is stated "ABC should be used to provide decision-useful information, not to develop a conceptually pure measure of costs" (Granoff, et al., 2000)

The Flashlight Cost Model used in this study is a "... decision support tool which utilizes activity-based costing techniques to analyze the costs of campus outputs" (Thomas and Lovrinic, 1994). The goal of the Technology Costing Methodology (TCM) project is to develop an authoritative costing methodology (and related procedures) for calculating costs:

   * within an institution to determine if proposed instructional approaches,
   that make heavy use of technology, actually do serve to contain costs; and

   * across institutions allowing data to be compared legitimately for
   different instructional or technological approaches, which will benefit
   legislatures, state-governing boards, state coordinating boards, and
   federal agencies.

The two models have similarities but some different focuses. The WSU experience indicates that familiarity with one model helps understand the other model. The differences are highlighted in Table 1. See issue's website <http://rapidintellect.com/AEQweb/win01.htm>.

Activity-based Costing

Both of the models adapt Activity-based Costing systems or techniques. An ABC system models the usage of an organization's resources on the activities that use these resources and then links the cost of the activities to outputs or products (Lewis, 1995).

The major difference between activity-based costing (ABC) and other accounting systems is that in an ABC model the resource costs are linked to activities rather than some other unit. For instance, teaching is an activity, and salary is the traditional cost measure associated with teaching. In education, however, just teaching may not accurately reflect the endeavor of developing and delivering an online course. ABC focuses on isolating the appropriate activities involved within a targeted study--such as integrating technology into teaching--and thereby provides organizations with a better understanding of the particular costs involved in the process and a how to link those costs to the particular outputs being studied. By isolating the particular activities within a process, ABC also provides improved metrics and a better understanding of complex business processes and the associated costs.

A traditional accounting system might give the total costs of a project; say $50,000 for payroll and $10,000 for goods and supplies. Activity-based costing would first determine the major activities of the project (such as design, development, delivery and assessment of a course) and then determine the cost of each separate activity.

Costs of activities will usually need to be estimated. Most university accounting systems do not use activity-based costing. In this way, ABC does not necessarily, (as one concerned faculty member once objected), "bury the audience in minutiae of detailed data." The amount and detail of the data can be tailored to the kinds of decisions the data and the report targets. Issues such as this are summarized in Figure 2 (Cokins 1996). See issue's website <http://rapidintellect.com/AEQweb/win01.htm>.

Based on experience, after the activities of the project have been determined, it may help to prepare an "activity dictionary". The TCM includes an activity dictionary, but some activities may be added or modified. The Flashlight Cost Model did not include an activity dictionary in their original handbook. An "activity dictionary" lists the major activities and defines each activity. The definitions can be a useful resource for staff that are trying to determine where to classify their time or other expenses. Edward Forrest gives an example of a format for creating an activity dictionary (Forrest 1996).

Case 1 -- Using The Flashlight Cost Model To Assess Three Technology Development Strategies

A study was initiated during the fall 1997 term to evaluate costs of three different development strategies, all of which used new technologies. It must be recognized that each of the technology approaches reflected a correspondingly distinct instructional implementation strategy that informs the analysis as well:

The modular approach

Some educational technology is developed to teach a particular point and is designed to be used only during specific class periods during a term. A product developed at WSU called "InterFon[TM]" is an example. It targets a key course concept and is designed for a single unit within a few Anthropology courses that focuses on teaching phonemes, the fundamental building blocks of language. The project would be used, by design, for only a few weeks during a single semester in any relevant course or courses. It was assumed it would cost more per student unit of use, but it was hoped this study would provide a metric of magnitude.

The web-based conferencing template

The most common example of this template may be a threaded discussion list used by students to communicate asynchronously via the World Wide Web. "The Speakeasy Studio and Cafe" is a course delivery and management tool built around threaded discussion list software. The tool was initially designed for and implemented in WSU's Freshman Seminars, which included 25 introductory classes that were linked to general education courses. The linked courses, as it turned out, provided a good method for expanding the use of the web-based computer conferencing software because the companion courses, learning of the technology, often adopted the technology.

The class conversion approach

Many new educational technologies begin as content specific replacements for certain classes. They sometimes evolve into more adaptable tools. "T&L 445" was developed by an instructor of educational technology who asked students to read essays and then e-mail responses to him from a designated window on the Web. The course design included graded responses, assignment sorting by dates, and other course specific features. The notable aspect of the class conversion approach in this study was that the design of this educational technology course for the Web integrated the course content with the technology being studied. Senior education majors studying new teaching technologies were actually using new technologies when they took the course about new technologies.

The Method: The Flashlight Economic Model

This cost study applied the Flashlight Cost Model (Ehrmann and Milam, 1999). The model has been developed as an evaluative tool as part of the Flashlight Project. The major question of the study was to determine which of the three general approaches yielded efficiencies relative to other concerns such as effectiveness and resource allocation.

Overview of Results

InterFon[TM] was partially funded by a grant from Boeing and Speakeasy was largely funded by grants from Microsoft and Boeing. The costs shown below are the total costs of the projects (no grant revenues have been deducted from the total cost to develop each project).

The cost of the web-based conferencing template, The Speakeasy Studio and Cafe, was substantially more expensive to develop than the other two approaches, costing roughly $180,000 to the point of instructional implementation and reliable use. The module project, InterFon[TM], was the next most expensive development strategy, costing $40,000 to develop to the point of reliable implementation and classroom use. Finally, the cost of converting an entire class was the least expensive strategy, estimated at $10,620.

However, the total cost of the projects can be a misleading way to compare the cost efficiency of the three development strategies. A more accurate measure may be the total cost per unit of use. For this study we used a measure that we called "student weeks of use." We took the number of students in each course times the number of weeks each course used the technology to determine "students weeks of use."

The estimated usage of the Speakeasy Studio and Cafe was 24,000 student weeks of use. (Note: that usage has increased substantially since the original analysis; there are over 200 courses using the Speakeasy Studio and Cafe for the fall 2000 term. Neither of the other two investments is in use at this time.) To put the costs to develop each technology and the amount of student use into perspective, the estimated cost to develop each was divided by the number of student weeks of use. Table 3 illustrates the estimated costs, student weeks of use and cost per student week of use for each of the three projects. See website <http://rapidintellect.com/AEQweb/win01.htm>.

Summary

Two findings from the study were especially salient.

   1. Even though the Speakeasy Studio and Cafe cost substantially more to
   produce, its cost per student week is comparable to the course conversion
   approach used for Teaching and Learning 445 and much less expensive when
   compared to the InterFon[TM] module.

   2. The Flashlight Cost Model pinpointed another important issue, that is,
   as a new technology continues to be used, the proportion of total expenses
   used to revise, support, and maintain it increases. The Speakeasy Studio
   and Cafe had been used for several semesters and had (and continues) to go
   through several revisions. Figure 1 shows that the ratio of expenses for
   the "Revise and Maintain" activity which appears to be increasing
   proportionally with the maturity of the product. See issue's website
   <http://rapidintellect.com/AEQweb/win01.htm>.

This final observation may have major implications for institutions developing new technologies. The cost of the new technologies will continue to rise as they are used. Revisions and maintenance costs seem inevitable and may become a major cost of the product as it matures. This data clearly highlighted that developing a technology requires funding the entire apparatus, not just development costs, but also training, maintenance and costs of revisions. This observation may not have been so apparent if ABC had not been used.

Conclusion

The results of this cost assessment using the Flashlight Cost Model make two points clear. (1) Educational investments are costly, both the initial investment as well as the cost to maintain and upgrade the courseware products. (2) The cost per unit of use will drop significantly if a product is "flexible" enough to be used and re-used by many disciplines. The surfacing and articulation of these reasonably obvious conclusions has proven vital for shaping policy and pointed to the next study summarized in Case 2.

Case 2 -- Using the Technology Costing Methodology to Assess the Cost of Designing, Developing and Delivering Courses to be Taught via the World Wide Web

Introduction

The lessons from the Flashlight Cost Model helped target future analyses of technology developments, but it made sense to follow with a study focusing on the process of implementing the educational technologies into courses. More emphasis would be placed on people resources and processes than on the technology. As a result, The Center for Teaching, Learning & Technology (CTLT) and the Distance Degree Program (DDP) at Washington State University (WSU) collaborated to use the Technology Costing Methodology (TCM) to analyze and improve WSU's pilot "Program Development Process".

WSU is in the process of implementing a five-step assessment process for all courses developed using the Program Development Process. Instructors are surveyed to determine their goals for the course, two formative surveys are conducted of students, to determine student goals and to determine students' experiences with interaction. Instructors participate in a focus group to assess the job of the course development team and finally, the costs of each course are assessed. A sixth step is in design; the CTLT hopes to encourage instructors to use an online survey tool to administer Classroom Assessment Techniques to distance students.

The Washington State Higher Education Coordinating Board asked WSU to do a field test of the TCM. The TCM model was particularly appropriate for the study being planned on the costs of courses being developed and taught online and provided an opportunity to deepen our understanding of cost methods. The TCM model also applies Activity-based Costing, although the focus is more ambitious and targets national norms. A preliminary draft of the TCM handbook states "An objective of this manual is to present a costing methodology that responds to this internal need..... Therefore, the second purpose of this document is to propose a set of procedures by which data from multiple institutions can be placed in a reasonably common framework and exchanged or compiled." (Jones, 2000).

Overview of the Study

An interdepartmental, multidisciplinary team composed of members from the Distance Degree Programs (DDP), the Center for Teaching, Learning, and Technology (CTLT) and the Student Advising and Learning Center (SALC) helped develop the courses. The courses analyzed were from eight departments. Time estimates were gathered from 40 faculty and staff. 169 students completed the courses in the analysis. All or some of the students in each of the ten courses were distance learners. All courses relied on technologies for student access and were developed for first time delivery; both the cost analysis and course development process were prototypes.

The activities for this project were chosen to correspond in general to a "courseware development cycle", similar in many ways to software process models (Pressman, 2001). Software process models help a team identify a development strategy that envelops the needed tools and methods into general, or in Pressman's words "generic phases". The software development process has been applied to courseware engineering and models are continuing to evolve (Bostock, 2000). Table 4 summarizes the activities used for the TCM project. See issue's website <http://rapidintellect.com/AEQweb/win01.htm>.

Summary of Results

Table 5 illustrates estimated average expenditures of the ten courses by major activity. Some of the expenditure types accrued to specific activities while other expenditures occurred in almost all activities for all the courses (e.g., salary & wages). See issue's website <http://rapidintellect.com/AEQweb/win01.htm>.

Discussion and Next Steps

The inverse correlation between design and development throws into relief the importance of planning and, to a considerable extent, the costs of not doing so. It is also interesting to note that the estimated costs of the design and development phases, on average, consumed more than half (59%) of the resources. Anecdotally, we know that delivery is the hidden variable - teaching time during delivery tends to be underestimated and appears to be more contingent on design than is generally recognized. Further TCM studies may provide information to support policy changes that will encourage more design time.

Several of the courses that were analyzed in the fall 1999 study will be taught again. WSU plans to track costs of subsequent offerings of the courses. It is expected that design and development -- or redesign and redevelopment -- will be less intense and faculty will optimize their instructional techniques.

Overall Summary

The combination of these cost studies surfaced more immediate and potentially important implications on teaching and quality of students' learning experiences that we anticipated. We hope to refine our focus in future studies on the constituent activities, design and delivery in particular, as keys for controlling costs and improving the quality of learning.

References

Bacsich, P., Ash, C., Boniwell, K., Kaplan, L., Mardell, J., and Caven-Atack, A. (1999) The Costs of Networked Learning. Sheffield: Sheffield Hallam University. http://www.shu.ac.uk/virtual_campus/cnl/ (August 16, 2000).

Bostock, Steven (1998) Courseware Engineering - an overview of the courseware development process http://www.keele.ac.uk/depts/cs/stephen_bostock/docs/atceng.htm#4 (December 15, 2000).

Cokins, Gary (1996). Activity Based Cost Management - Making it Work, A Manager's Guide to Implementing and Sustaining an Effective ABC System, Irwin, p. 200.

Evans, J. Stuart. (1991). "Strategic flexibility for high technology maneuvers: A conceptual framework" Journal of Management Studies, 28(1): 69-89.

Ehrmann, Stephen (1995). "Asking the right questions: What does research tell us about technology and higher learning?" Change, 27(2), 20-2x.

Ehrmann, S. & Milam, J. Jr. (1999). "Modeling Resource Use in Teaching and Learning with Technology", a Corporation for Public Broadcasting.

Forrest, Edward (1996). Activity Based Management: A Comprehensive Implementation Guide. McGraw-Hill, pp. 49-58.

Granof, M., Platt, D., and Vaysman, I. (2000), "Using Activity-Based Costing to Manage More Effectively" http://endowment.pwcglobal.com/grants/mgdpiv.asp (August 30, 2000).

Jones, Dennis (2000), Technology Costing Methodology Handbook - Preliminary Draft, National Center for Higher Education Management Systems (NCHEMS) in conjunction with the Western Cooperative for Educational Telecommunications (WCET), pp. 3 & 5.

Lewis, Ronald (1995). Activity-Based Models for Cost Management Systems. Quorum Books, p. 50.

Lovrinic, Joe (1994). "Economic Model (technology version) Development Guidebook - draft version 1.1," Indiana University.

Pressman, Roger (2001). Software Engineering, A Practitioner's Approach. 5th Ed., McGraw Hill. pp. 23-47.

Thomas, B. & Lovrinic, J. (1994). Creating an Economic Model for Your School or Support Unit - a Guide for the Project Team. Indiana University,

Dr. Henderson, Assessment Coordinator at Center for Teaching, Learning, and Technology, received Ph.D. from Washington State University, MBA in Finance from the University of Washington. Dr. Brown, Director of Center for Teaching, Learning, and Technology, received Ph.D. from Washington State University, MA in English from San Diego State University.