Sustaining innovation in technological education.

Community colleges are faced with continuing to expand both academic and vocational offerings, and they have a unique role in providing and validating skills (Carnevale & Desrocher, 2001). Often a prominent component of community colleges, occupational and technical programs are especially important today in our rapidly changing job market. Recently, technology and health care programs were voted "hot" by community college administrators (Coley, 2000). The top five "hottest" programs were digital systems, facilities technology, dental hygiene, manufacturing process technology, and telecommunications. So how are community colleges responding?

The U.S. national need for more technicians was anticipated in the early 1990s in reports such as Technology for All Americans: A Rationale and Structure for the Study of Technology (International Technology Education, 1996) and Gaining the Competitive Edge: Critical Issues in Science and Engineering Technician Education (National Science Foundation, 1993). Congress passed the Scientific and Advanced-Technology Act of 1992 (U.S. Congress, PL 102-476), which called for the National Science Foundation (NSF) to improve education for technicians in advanced technology fields. In response to the Act, NSF developed the Advanced Technological Education (ATE) Program. The long-term goal of the ATE program is to develop projects that will be sustained after the ATE funding stops. These college-based programs are in competition of sorts with certifications offered by national industry, trade and professional associations and company vendors, such as Microsoft, Cisco, and others. The number of other organizations offering certifications has grown substantially from about 120 in 1965 to more than 1,600 in 1996 (National Organization for Competency Assurance, 2000). To be competitive, community colleges need to sustain innovation in technology education.

Sustainability can be simply defined as the ability to prolong or to supply with sustenance, but the issues surrounding sustainability are complex. It can take many forms but, in general, implies that the valuable portions of a new project will be supported and therefore continued in some form. In our experience with evaluating projects, the continuation can be supported by funding from internal or external sources. Furthermore, internal funding could mean that more money is made available or that priorities are shifted within existing funding levels. For example, if a valuable portion of a new project was more experiential classes through the use of collaborations with local businesses, this project could be sustained in a variety of ways. Continuous sources of external funding could be found to support the increased faculty effort to arrange and monitor the experiences. The college could support the innovation through providing continuous extra funding or through its existing course support structures by not supporting some other kind of learning. The instructor could sustain the change by using his or her time for experiential learning instead of preparing lectures.

The research reported here uses the ATE projects as a data source for examining the potential for sustainability that generally exists in technical education projects. In other words, the research question was: Which characteristics related to sustainability are exhibited by a set of technical education projects?

Literature Review

Despite its importance, there is little research directly related to sustainability, although some generic strategies have been suggested (Cuban, 2000). The research on sustainability most relevant to technical education is the literature related to sustainability of organizational change. Generally, the literature is based on the authors' experiences with sustaining change rather than empirical studies.

Howard and Howard (2000) presented the Self-Determination/Reliance Model (SDR). This model is based on psychosocial development and associated with the monitoring and assessing of the processes and outcomes of community groups. Critical movement on the identified dimensions indicates progress toward sustainability. These dimensions are accountability, decision making, information, knowledge and skills, and resource mobilization. Flower (1996) described five fundamentals for organizational change: husbanded resources, abundant relationships, abundant information, distributed power, and a clear sense of purpose. Schwartz (1994) listed several guidelines for managing and sustaining change. He stated that successful change requires employee participation, training, provision of continuous feedback, a reward system, and development of group norms. Gabelnick, MacGregor, Matthews, and Smith (1990) offered a checklist for sustainable learning communities in colleges. The list includes (a) determining focus and design, (b) using faculty resources, (c) coordinating with current institutional initiatives, (d) obtaining and maintaining administrative support, (e) having needed resources, (f) using promotion and marketing, (g) having institutionalization of the concept, and (h) using program improvement mechanisms. Senge, Kleiner, Roberts, Ross, and Smith (1994) have suggested that in order to support organizational change an organization should practice five learning disciplines. These include personal mastery (encouraging all members to develop themselves), mental models (reflecting upon how to view the world), shared vision (building commitment within a group), team learning (contributing to each other), and systems thinking (recognizing the interrelationships that shape systems).

In addition to these various lists, other articles referenced the need for organizational cultural compatibility and the ability of programs to create and command value. Schneider, Brief, and Guzzo (1996) reported that the cultural aspect suggests that change is not sustained when (1) the change is inconsistent with existing climate and culture and (2) changers fail to build a climate and culture to support the change. Schneider et al. go on to state that changes are more likely to be sustained when the organization is ready and prepared for the change and when the change permeates all levels and functions of the organization. In order to help build a culture for change, the organization needs to value the innovation. Cropper (1996) suggested that the creation of value falls into two domains: (1) consequential value such as legitimacy, security, and efficiency; and (2) constitutive value, which is based on efforts being valued (i.e., expressions of purpose and fit with institutional context, capacity, and conduct). For example, in the ATE program, consequential value could be provided in terms of legitimization from business and industry collaborations or the use of standards. Security could be provided through the use of existing programs and professors and emphasis on how the new programs will increase the security of their jobs and the security of the institution in terms of new students.

The Advanced Technological Education Program

The ATE Program focuses its funding efforts at the community college level and has four objectives:

* Develop model instructional programs in advanced technology fields

* Provide professional development of faculty and instructors in advanced technology fields

* Establish innovative partnership arrangements

* Develop and disseminate instructional materials

Since its inception in 1994, more than 400 ATE grants have been made. While most have gone to two-year colleges, ATE also offers outreach to secondary and baccalaureate institutions. Grant awards are made in two categories, centers and projects, with centers receiving substantially more funds and having a broader scope. Each center or project uses these funds to facilitate the development of technicians for a particular field. While a project tends to focus on only one or two of the above objectives, centers typically address most or all of the objectives. Centers always receive funding for multiple years, serve as model programs for other institutions and organizations, and disseminate information to a region (e.g., several States or the nation as a whole).

About 200 projects (centers and projects will be referred to as projects) are currently active across the United States. ATE supports education in a broad range of technology fields including agricultural technology, biotechnology, chemical technology, civil and construction technology, computer and information technology, electronics, environmental technology, geographic information systems, manufacturing and engineering technology, marine technology, multimedia technology, telecommunications, and transportation technology. ATE also supports education in the basic sciences needed to understand the technologies (e.g., physics, chemistry, and mathematics). Descriptions of some of the projects are provided in The Learning Edge: Advanced Technological Education Programs at Community Colleges, edited by Mahoney and Barnett (2000).

ATE program activities include: (a) the adaptation of exemplary educational materials, courses, and curricula in new educational settings; (b) the design and implementation of new educational materials, courses, laboratories, and curricula; (c) the preparation and professional development of college faculty and secondary school teachers; (d) internships and field experiences for students, faculty, and teachers; and (e) the broad dissemination of exemplary educational materials and pedagogical strategies that have been developed through funded ATE projects. Activities may have either a national or a regional focus, but not a purely local one (NSF, 2001).

Methodology

The data used to examine the issue of sustainability were gathered as part of an NSF-funded evaluation of the ATE program housed at Western Michigan University's Evaluation Center. A Web-based survey was administered to all current ATE projects. After the survey, a subset of 13 projects was selected for in-depth site visits. The quantitative and qualitative data collected from the survey and the site visits were combined and compared to an analytic framework developed from the literature review to assess the potential for sustainability within the projects. Using a predetermined framework centers this study within a postpositivistic philosophy. In short, the study used a sequential, nested, mixed-methods design and viewed the data within a postpositivistic paradigm.

The Web-based survey was conducted in May 2000 (113 projects, with a response rate of 73%). All current projects were included in the sample. See http://ate.wmich.edu for a complete copy of the survey and its accompanying reports. Approximately 20 items on the survey related to sustainability. Eight of these were items that rated project growth, and the rest pertained to the different project activities such as collaboration or materials development. For growth items, projects were asked to rate their current status against their status the previous year in terms of number of collaborations, financial support, participation by other institutions, use of project products, numbers of students enrolled, placement, and graduation rates, and numbers of professional development attendees and offerings. Items within the areas included specific questions about numbers and other aspects of the projects rather than ratings of change. The survey data were summarized through the creation of percentages, mean scores, and medians.

Site visits were made to 13 ATE projects (3 centers and 10 projects) to provide more in-depth data. Visits were conducted from September 2000 through February 2001. Judicious mixes of knowledgeable evaluators and content experts conducted the visits. Sites were selected using a purposive sampling technique based on recommendations from NSF, thus ensuring a cross section of project types in the ATE program. Sites were primarily selected based on maturity (time conducting the project), technological disciplines, funding levels, geographic dispersion across the United States, major ATE objectives the projects addressed, and type of sponsoring organization. Specifically six projects were in their 2nd or 3rd year of funding, and 7 projects were in at least their 4th year of funding. The technological disciplines represented were: Chemistry/ Environmental (3), Manufacturing Technology (3), Information Technology (3), GIS/Multimedia (2), Engineering Technology (1), and Biotechnology (1). Seven projects were funded for a total of $750,000 or less, while six projects received a total of more than $750,000. Of the 13 projects, 4 were located on the West coast, 4 in the middle section of the United States, and 5 on the East coast. All the projects addressed at least two of the four primary objectives of the ATE program (i.e., program improvement, materials development, professional development and collaboration). Seven projects addressed all four primary objectives. The projects included 11 community colleges with technical training programs, one professional organization developing materials for use at community colleges and high schools, and one 4-year school which also offered associate degrees. Prior to the visits, site visit team members were asked to review packets of information (e.g., agenda, project abstract, survey, evaluation reports, proposal, annual reports) and were provided with site visit procedures and interview protocols. The authors or a "trained" person (someone who had attended another site visit with one of the authors) participated in all of the visits to help further guarantee consistency.

To minimize the burden on the sites, site visits were usually one full day in length for projects and two full days for centers. Activities included interviews with key personnel (e.g., principal investigators, coprincipal investigators, business and industry collaborators, faculty, students), review of materials (e.g., modules), tours of facilities (e.g., labs), and observation of activities (e.g., classes). Each site received a report that was for its use only. This described the project through the eyes of the site visitors and detailed impressions by work category and other important issues (e.g., goal attainment, sustainability, evaluation, and such). The site visit protocols contained questions pertaining to sustainability of the innovations inherent in the project and were directed to the chief administrator of the ATE institution (a president or dean), the principal investigator, and others involved in the project. Based on the answers to these questions, the site visit report outline asked the site visitors to comment on the potential for sustainability of the project overall, and for additional comments about sustainability in the individual sections related to the ATE project activities. To analyze the data, all of the site visit reports were read and any comments related to sustainability were marked. These were then reread, organized, and considered in conjunction with the survey data.

Results

The survey and site visits produced a large amount of data. Crossing the existing literature with the nature of the ATE program produced an analytic framework from which to access the data. The suggestions listed in the literature review about what would produce sustainability were compared to the data from the ATE projects. The suggestions and data were consolidated into the analytic framework. The resulting framework showed three substantive areas where educational technology projects could focus on sustainability: collaboration, program improvement, and professional development. The collaboration area relates to the recommendations in the above literature for wide participation with clear focus and group norms as well as the notions of shared decision making and the distribution of power. This area also includes ideas of administrative support. The program improvement area incorporates the ideas of marketing, flexibility, resource mobilization, coordination with existing initiatives, and effective use of resources. Also included here are the notions of accountability, feedback, and continuous improvement associated with evaluation. Professional development is related to reward systems, knowledge, and skills. The data from the survey and site visits relating to each of these areas are considered separately below.

Collaboration

Collaboration is critical if an innovation is to be sustained, since the collaborators provide the support necessary for sustaining the innovation. Collaboration is a widespread and fundamental characteristic of the ATE projects. On the survey, a majority of the projects reported increasing or substantially increasing direct participation by other institutions and organizations. The median project listed at least 15 collaborative efforts, and each engaged slightly more than 2 people per collaboration. The most prevalent collaborator reported by projects was business and industry, followed by educational institutions. The site visits showed that the business and industry collaborators provided valuable information about their needs as well as educational opportunities.

Consortia of two-year colleges existed as well as programs that articulated with four-year programs and had connections with high schools. The site visits showed that the articulation programs were usually very specific to the institutions involved. Often general education courses transferred to four-year institutions, but few of the technical courses transferred. Although reverse transfer was evident at some of the sites where students working on four-year degrees take technological courses at the community colleges, these courses often did not count in the four-year programs. Relationships with secondary schools were common although these were more generally for the purposes of recruitment than for course development. Many of the sites offered community college credit for courses taken as part of a high school diploma. Some sites also had collaborations with "corporate universities"--the educational arms of many businesses.

The observed collaborations varied widely in terms of duration and formalization. The range was from short ad hoc relationships of a few days to long-term sustained relationships based on formal agreements. The principal investigator survey responses about the factors which affect the quality of collaborative efforts fell into five areas: (1) quality and enthusiasm of the people involved, (2) commitment and interest, (3) mutual benefit, (4) common purpose and vision, and (5) communication including clear expectations. The sites showing the strongest collaborations were characterized by regular meetings which appear necessary to sustain enthusiasm and progress. The visits showed that it was also important that the collaboration have a win-win orientation. Although altruistic participation produced some benefit, stronger relationships resulted from all parties gaining something. It also was beneficial if the collaborators had formal agreements about what the roles of each were. Roles within the collaborations included giving advice, providing assistance, assessing progress, and being advocates. Additionally, the expected outcomes of participation in the collaboration needed to be specified. In other words, the benefit to be obtained from cooperating had to be clearly identified and understood by all.

The collaborations were successful in providing support for the projects. The survey data showed that for every dollar provided by NSF for the duration of projects' grant periods, the projects increased their working resources for the ATE program by one-half dollar. Connections across peer institutions also provided monetary support. The projects reported themselves as increasing their receipt of financial support from other organizations.

Program Improvement

Strong programs are a critical element of sustainability. Numbers of students in a program are strong indicators of a program's sustainability. Because of the different emphases of the ATE projects, these enrollment numbers varied substantially. Some had very high numbers of students (e.g., 700) while others had small numbers (e.g., 7-10). Generally, the number of students involved in a given program at the visited sites were modest (i.e., 10-30 students). The projects reported themselves as "increasing" on items about numbers of students enrolled, students placed in technical jobs, and students graduating or completing programs. When viewed from the perspective of a specified program conducted during the previous 12 months, the average number of program completers was 43. When examined from a per course perspective, the average number of students enrolled in a course in a specified program was seven.

Another aspect of strong programs is meeting the needs of the field. The site visits showed that the changes in the different technology fields occurred very quickly with little lead time. Therefore, the most successful programs were those that were attuned to the future and flexible enough to respond quickly. The site visits also showed that the updating often required new and expensive equipment. For example, geographic information system (GIS) is a valuable tool that can be used in a variety of occupations and settings. To realize its potential fully, however, powerful computers are necessary. These are expensive and the need for even more computing capacity grows rapidly so the cutting edge computers purchased need to be replaced relatively quickly.

Many of the changes required in technological programs involve changes in curriculum and instructional approaches. The site visits showed that the curricula were grounded in work based standards and, if these were not available, the programs worked with businesses and industries to identify them. The technological programs also reflected the use of mathematics, science, and communications across the technical curriculum components and integrated soft skills such as teamwork and critical thinking. Many programs incorporated work based education, but this aspect was not always required of the students. Work experiences were more often presented as opportunities in which the students could participate. The visits showed that there were quite often multiple educational tracks within a general area to meet student needs both in terms of types of knowledge and outcomes. The projects reported themselves as increasing their receipt of income from the use of developed products. Notwithstanding this increase, they reported lack of time, money, and other resources as barriers in terms of producing quality programs.

More than 80% of the projects reported use of a project evaluator. However, based on the site visits, it appears that only modest data collection for evaluative and accountability purposes is undertaken and that the evaluation may be concentrated on only one specific aspect of the project. Without more critical information (e.g., collection of number of students: enrolled, completing programs, gaining credit for articulated courses, participating in mentoring programs, and going on to four-year colleges), projects are most likely operating in the dark, not knowing how to improve activities and programs or reward effort, not knowing if the goals of their activities are being attained, or not being able to track their progress toward sustainability.

Professional Development

Professional development is directly related to program quality. Providing adequate and timely professional development to meet the needs of technological faculty poses difficult problems. The site visits showed that many programs made extensive use of adjunct faculty and that these faculty were well qualified for their jobs. On the other hand, the regular faculty routinely reported that they received very few opportunities for professional development and little financial support to participate in it. Some faculty even said that they were encouraged not to participate in professional development because it would take them away from their classes. Perhaps this lack of encouragement can be understood. Some faculty told about colleagues who had been lured away into industry once they had attained advanced skills.

Despite the problems with professional development, one goal of the ATE program is to provide professional development. The survey results showed that ATE-provided professional development opportunities were well attended and received. Conferences, workshops, and in-service courses were the most popular formats. Course offerings were well attended with a median of about 20 individuals for projects and about 130 for centers. Most participants were from associate degree-granting institutions. Regarding attendance at their professional development opportunities, more than 75% of the projects reported they were at least at 75% capacity, and more than 90% of centers reported this level of capacity.

The limited nature of evaluation referred to under "program improvement" above is also relevant here. Although a great deal of professional development takes place in the ATE projects, little is known about the implementation of the education received except that the participants found it valuable. Also, although the professional development is rightfully focused on content upgrading, other types of knowledge such as instructional materials development are less apparent.

Implications

Based on the ATE program, sustainability of innovations in technical education appears to be an attainable goal. In particular, the ATE projects were quite successful at developing and maintaining collaborations. This finding is consistent with the findings of Mahoney and Barnett (2000), who reported that long-standing connections were strengthened and new ones created as part of ATE. Through collaborations, the projects were able to coordinate current initiatives, obtain external and administrative support, mobilize resources, and distribute power. The distribution of power helps to make more people responsible for the success of an innovation, and the more ownership people feel the more they will be advocating for sustainability. The collaborations were also developed with varied organizations; this breadth of contact also helps to promote sustainability. Additionally, many of the collaborations were mutualistic, and therefore the continuation of the innovation was of benefit to many.

The data showed that business and industry collaborators were the most common, and their involvement focused on assuring accurate representations of the workplace. This finding is not too surprising since the nature of technical programs requires partnerships with business and industry (AACC, 1996). Most technical programs have lay advisory committees which are mandated by the community college system in almost all states. These committees meet to assist programs from as frequently as quarterly to only annually (Reed, 2002).

The collaborations with other educational institutions were revealing. The articulation agreements were very specific to the institutions involved. This finding is consistent with the findings of Coley (2000). The few relationships found with for-profit educational entities showed good promise. This possibility was also suggested by Carnevale and Desrocher (2001). This is also consistent with other research which shows that nearly two-thirds of the 1,600 "corporate universities" have alliances with educational institutions that offer their workers training and degrees (Meister, 1998; Cobb & Nelson, 1999).

One of the realities in the technology fields is change. The keywords are invention and innovation. This constant change not only provides an exciting and dynamic culture and cutting-edge jobs, but it also requires the constant updating of programs and materials and the concurrent professional development of faculty members. We found that the need to update occurred with little prior warning and that the update had to be accomplished quickly in order to meet that need. We also found that the processes in place in the community colleges are not designed to be responsive to this rapid change. Many colleges have a cyclical approach to change, and the cycles tend to be much slower than the technological occupations they support. The change cycles are necessary, because community colleges cannot afford to revise and refurbish all of their programs simultaneously. There also needs to be some mechanism for assuring that there is equality among the programs. In one sense, this situation is fair, but certainly disadvantageous to the technology programs.

Leading-edge technology programs are also very high risk. The materials necessary for an innovative program may become out-of-date quickly and have little opportunity for alternative use. These cutting edge programs can draw students and provide the students with lucrative employment but are very costly. Sometimes these types of programs are used as model programs where the risk is justified in the hope of other programs moving forward. In these cases, however, it is unlikely that the community college would be able to continue to support the program at this level. As has been the case in the past, we found that the business and industry partners often provide equipment or access to the equipment at their sites. This help is very useful but constricts equipment use and sometimes ties the community college program more directly to a particular business than might be desirable.

In order to have cutting-edge programs, the instructors must have cutting-edge skills. Some of the demand is met by hiring adjunct faculty who have the necessary expertise and recent experience. This is a good strategy, but it leaves the program without internal institutional support and can result in a lack of continuity in the program and lack of integration of different courses into a coherent course of study. It also can raise the potential for controversy between the adjunct and permanent faculty. The other alternative is to upgrade the experience of the permanent faculty through professional development. Because of the changing needs, this development has to occur quickly. Unfortunately, the cyclical nature of opportunity discussed above is present for faculty development as well as for program support. Additionally, most staff development programs are not designed to provide rapid faculty skills development. According to McKenzie (1991), traditional professional development programs have many flaws. They are generally short term and somewhat ad hoc. They often do not provide sufficient time to practice new skills or conduct long-term follow up and support.

Furthermore, there is generally little money to develop faculty inservice programs because the public and policymakers fail to make professional development a high priority (Gold, 2002). Based on the interviews during the site visits, community colleges tend to undervalue professional development, making faculty members responsible for their own continuing education. The interviewees said that community colleges have little or no money routinely available for faculty development and little money to support faculty while they attend and begin to implement new approaches and skills. They noted that even release time, money for books, or travel is often difficult to obtain. It appears there are few processes in place for providing the continuous upgrading and current knowledge necessary to maintain excellence in technology. Persky (1990) suggests that if staff development for instructors is to be truly effective, administrators must take supporting action. Colleges and the public need to recognize the value of continuing education and incorporate the funding necessary to support it. These new skill levels need to be rewarded as well, or these highly trained faculty members will be lured away by significantly higher industry salaries.

The two least evident elements of potential sustainability were available information and marketing. Information about the strengths and opportunities inherent in a program is necessary to market it successfully. Marketing is necessary to help provide the funds required to support the innovations both internally and externally. Improvement in these two elements would help to ensure the sustainability of future innovations.

In order to help achieve this sustainability, we suggest the development of multiyear plans which outline the process for sustaining the innovation. Elements of this plan, including implementation strategies and schedules, could include:

* Ongoing vision and goals (i.e., where a project sees itself and its partners in the future)

* Methods and time-tables for collecting data critical for accountability and decision making and means for sharing and using this information with key stakeholders 

* Identification of and strategies for obtaining additional funding, revenue sources (e.g., product income), and other support (e.g., dollars for training, release time for faculty)

* Descriptions of collaborations or partnerships and what the contributions of these partnerships will be in concrete terms (e.g., specific goals, in-kind services, dollars, dissemination of materials, written commitments and for specified time periods)

* A depth chart for personnel and partnerships, including contingencies for critical personnel and partnership changes (e.g., retirement, key business and industry partner struggling through an economic downturn, loss of a key funder)

* Description of the strategies for the institutionalization of the project (e.g., approval of a tech program by a specified date, plan for written commitments from administration over time)

* Promotion and marketing plan that outlines the various means (e.g., Web, conferences, publications, professional organizations, trade shows) to be used to raise awareness and acceptance of a project and update and disseminate its products (e.g., materials)

In summary, in order to help ensure the sustainability of technical education innovations, several steps should be taken. The existing structure of internal and external collaboration should be strengthened in both breadth and depth so that more people are involved and at least some are involved more substantially. These collaborations need to be symbiotic and involve truly shared power. The programs developed need to be progressive but carefully designed so that any expensive equipment can have multiple or future uses. The programs of study need to have the depth required to fit into the high-need areas but also have enough basic understanding that the students and the program itself can be flexible in responding to the changing job market. The professional development of faculty needs to be supported in terms of opportunities and in terms of reward. Finally, the programs must determine the data necessary to market themselves effectively to both internal and external audiences.

References

American Association of Community Colleges. (1996). Interdependence through partnerships: Transforming education, Improving science, mathematics engineering and technology instruction. Strategies for the Community College, 64. Washington, DC: Community College Press.

Carnevale, A., & Desrocher D. (2001). Help wanted, credentials required: Community colleges in the knowledge economy. Washington, DC: Educational Testing Service.

Cobb, J., & Nelson, M. (1999). Blurting the lines: The convergence of higher education and the corporate universities. Retrieved from www.academyonline.com/at_issue/issue_0799.html

Coley, R. (2000). The American community college turns 100: A look at its students, programs and prospects. Washington, DC: Educational Testing Service.

Cropper, S. (1996). Collaborative working and the issue of sustainability. In D. Huxham. (Ed.), Creating collaborative advantage (pp. 80-100). London: Sage. Cuban, L. (2000). Fads and fireflies: The difficulties of sustaining change. Educational Leadership, 57(7), 7-9.

Flower, J. (1996). The five fundamentals of dealing with change. Physician Executive, 22(8), 37.

Gabelnick, F., MacGregor, J., Matthews, R., & Smith, B. (1990). Making learning communities work: Issues of implementation and sustainability. In R. E. Young (Ed.), Learning communities: Creating connections among students, faculty and disciplines. (pp. 39-51). New Directions for Teaching and Learning, no. 41. San Francisco: Jossey-Bass.

Gold, N. (2002). Assessing the impact and effectiveness of professional development in the advanced technological education (ATE) program. Kalamazoo, MI: University of Western Michigan, The Evaluation Center.

Howard, D. C., & Howard, P. A. (2000). Towards sustainability of human services: Assessing community self-determination and self-reliance. Canadian Journal of Program Evaluation, 15(1), 25-40.

International Technology Education. (1996). Technology for all Americans: A rationale and structure for the study of technology. Reston, VA: Author.

Mahoney, J., & Barnett, C. (2000). The learning edge: Advanced technological education programs at community colleges. Washington, DC: Community College Press.

McKenzie, J. (1991). Designing staff development for the information age. The Educational Technology Journal, 1(4).

Meister, J. (1998). Corporate universities: Lessons in building a world class workforce. New York: McGraw-Hill.

National Organization for Competency Assurance. (2000). Retrieved from www.noca.org.

National Science Foundation. (1993). Gaining the competitive edge: Critical issues in science and engineering technician education, NSF 94-32. Washington, D.C.: Author.

National Science Foundation. (2001). Advanced Technological Education [ATE] program solicitation, NSF 01-52. Washington, DC: Author.

Persky, S. (1990). What contributes to teacher development in technology? Educational Technology, 30(4), 34-38.

Reed, L. (2002). Collaboration efforts for the ATE program: ATE centers' and projects' approaches, use, and effectiveness of collaboration. Kalamazoo, MI: University of Western Michigan, The Evaluation Center.

Schneider, B., Brief, A., & Guzzo, R. (1996). Creating a climate and culture for sustainable organizational change. Organizational Dynamics, 24(4), 7-19.

Schwartz, A. (1994). Eight guidelines for managing change. Supervisory Management, 39(7), 3.

Senge, P. M., Kleiner, A., Roberts, C., Ross, R. B., and Smith, B. J. (1994). The fifth discipline fieldbook: Strategies and tools for building a learning organization. New York: Doubleday.

U. S. Congress. Scientific and Advanced-Technology Act of 1992 (PL 102-476). Washington, DC: Author.

Frances Lawrenz is a professor and chair of the Department of Educational Psychology at the University of Minnesota in Minneapolis. [e-mail]

Nanette Keiser is a senior research associate at The Evaluation Center at Western Michigan University in Kalamazoo. [e-mail]

Bethann Lavoie is a Ph.D. candidate in the Department of Curriculum and Instruction at the University of Minnesota in Minneapolis. [e-mail]