Printer Friendly

An organizational approach to understanding evaluation in extension.

Introduction

Existing in some capacity in every state and national territory, Extension is a large, educationally-focused organization based within the land-grant university system. Having developed and grown in educational capacity over the past hundred years, Extension offers some unique challenges when addressing evaluation concerns. Nongovernmental funds including grants from public and private agencies, such as the W. K. Kellogg Foundation, assist in the development and delivery of unique programs within specific state systems; however, the majority of funding for Extension programs comes from local, state, and federal dollars (Rasmussen, 1989). Therefore, the primary driver for program evaluation within Extension is accountability for public funds.

While the federal government has rapidly increased Extension accountability requirements through passed legislation including the Government Performance and Results Act (GPRA) in 1993 and the Agricultural Research, Extension, and Education Reform Act (AREERA) in 1998, Extension continues to exist with little data showing programmatic worth. This may be attributed to a broadly held belief that evaluations are never used by decision makers. Patton (2008) argued the majority of evaluation efforts are conducted in vain. Wholey, Scanlon, Duffy, Fukumotu, & Vogt (1970) stated that "the recent literature is unanimous in announcing the general failure of evaluation to affect decision making in a significant way" (p. 46). In fact, very little evidence shows evaluations have succeeded in effecting government planning (Patton, 2008). Weiss (1972) stated "a review of evaluation experience suggests that evaluation results have not exerted significant influence on program decisions" (p. 10).

Other research shows this situation is sometimes different when it comes to Extension. By placing an emphasis on learning along with accountability, embedding in evaluation system--wide policy, Extension evaluation efforts can make an impact on decision making and enhance support for state--level funding. During the 1990's, Ohio State University Extension used a proactive approach to working with their legislators and decision makers resulting in significant growth in appropriations from their state government (Jackson & Smith, 1999). Their basic philosophy was guided by striving to stay accountable through open communication with their funders regarding how resources were used in the past, how continued and increased funding would be used in the future, and openly sharing their evaluation plans and results exhibiting public value throughout the programmatic process (Jackson & Smith, 1999). By documenting positive impacts on clients to those making funding decisions, this state was able to increase their county Extension budgets above and beyond the rate of inflation.

Unfortunately, the amount of effort the state of Ohio puts in to its accountability efforts is the exception and not the rule (Fetsch & Gebeke, 1994). Even with the assistance of Extension evaluation specialists, supportive evaluation cultures within state Extension systems are limited (Radhakrishna & Martin, 1999) despite research showing evaluation is an essential Extension employee competency (Harder, Place, & Scheer, 2010). Due to an initial push to measure short term changes, the majority of Extension professionals utilize posttests given at the conclusion of their educational activities to assess the level of success obtained (Franz & Townson, 2008). While low level reactions and some knowledge and skills gained are accounted for with this method, long-term outcomes recording actual behavior changes along with social, economic and environmental impacts of Extension programs are lacking. Without enhanced evaluation driven environments at the organizational level, the state and federal Extension systems will continue to be inadequate at reporting programmatic successes, resulting in a lower perceived public value of Extension programs.

Therefore, questions exist as to how an enhanced evaluation driven environment can be established. What is the best organizational structure within Extension to promote and enhance Extension professionals' evaluation behaviors? Does professional development training for Extension professionals emphasizing evaluation make a difference? What can be done to enhance and promote evaluation behaviors leading to credible data on the programmatic impacts and public value needed to continue government funding? Since identifying and using evaluation systems to assess program impact in Extension is part of the National Research Agenda: Agricultural Education and Communication, 2007-2010 (Osborne, 2007), a study exploring the ways in which Extension systems can be adjusted to encourage the creation and use of evaluation systems can yield valuable data providing direction for future practice.

Theoretical Framework

A review of the evaluation literature, the Burke-Litwin model of organizational performance and change (Burke & Litwin, 1992), and the Theory of Planned Behavior (Ajzen, 1985) led to the creation of a conceptual model of organizational evaluation (Lamm, Israel, & Irani, 2010) used as the theoretical framework for this study (Figure 1). The conceptual model of organizational evaluation is divided into three sections: transformational factors, transactional factors, and individual performance factors. Similar to the BurkeLitwin model, this model suggests changes within the transformational factors influencing evaluation practices will require the entire system to adjust with individuals across the organization exhibiting new behaviors (Burke & Litwin, 1992). The transactional factors of the model represent the everyday transactions occurring within an organization that affect evaluation. Changes to these factors are considered systematic improvements, evolutionary rather than revolutionary, and specifically selected (Burke, 2008). The individual performance factors within the model are derived from both the Burke-Litwin model and the Theory of Planned Behavior and represent the individual's needs and values, skills and abilities, attitude, subjective norm, and perceived behavioral control influencing individual behavior choices related to evaluation (Ajzen, 2006; Burke & Litwin, 1992). Each of the transformational, transactional, and individual performance factors will be described in more detail below.

[FIGURE 1 OMITTED]

Transformational Evaluation Factors

Leadership is the primary driver in the conceptual model of organizational evaluation (Lamm et al., 2010) having a strong influence on evaluation behavior (Burke, 2008). Arguments have been made against the impact leadership has on organizations from a social perspective (see Salancik & Pfeffer, 1977; Zacarro, 2001; Bourgeois, 1985), but in this case Extension administration is a hierarchical structure with a director for each state program making system-wide decisions; therefore their choices and opinions weigh heavily on the entire organization. Preskill and Boyle (2008) determined that leaders who are willing to seek information when they make decisions, are open to feedback from others, and reward employees for engaging in evaluation work will have a positive influence on increasing evaluation activities and the development of longer lasting impacts of evaluation within the organization.

In addition, an organizational culture designed to allow Extension professionals to share information freely, trust one another, encourage asking questions, and take risks was more likely to breed successful evaluation efforts (Preskill & Boyle, 2008). While studying changes in organizational cultures, examining over 200 companies including Con Agra, GE, SAS, British Airways, and Bankers Trust, Kotter, and Heskett (1992) found that without rooting change in the culture of the organization it was nearly impossible to sustain the change over time. Their primary finding was that those organizations with the highest performance and ability to make changes were the companies that embraced an adaptive culture (Kotter & Heskett, 1992).

Transactional Evaluation Factors

An organizational culture seeking out new information will establish policies and procedures that reward the practice of evaluation (Burke, 2008). Therefore, using evaluation as a benchmark in performance reviews and promotion and tenure will influence employees to engage in evaluation behaviors (Bess, 1998). Burke (2008) also found organizational culture influenced the climate within the work unit and individuals' needs and values. Both organizational culture and work unit climate are social context variables within the conceptual model of organizational evaluation. If the overall organizational atmosphere encourages the use of evaluations, work unit climate will follow suit with an emphasis on the need for evaluation use as part of the social atmosphere (Burke, 2008).

Since evaluations are most effective when management clearly communicates why they are necessary and how they are going to be used (Preskill & Boyle, 2008), management is essential to successful organizational evaluation. Currently, management of evaluation activities within Extension varies from state to state. In some cases evaluation activities are coordinated by county or regional directors, while other states manage evaluation efforts through the use of state wide Extension evaluation specialists located closer to state administration (Warner, Rennekamp, & Nall, 1996). Through clearly created and communicated objectives, and tasks to accomplish those objectives, management is easier to understand and tasks are more easily accomplished by employees (Preskill & Boyle, 2008).

The evaluation system includes all aspects of the policies and procedures put into place to encourage, enhance, and assist with the function of evaluation within an organization (Burke & Litwin, 1992). High quality evaluation efforts require materials, personnel, time, and financial resources (Arnold, 2006; Volkov & King, 2007). The ways in which a state Extension system sets up its reporting procedures, the clarity of goals for evaluation, whether evaluation weighs on performance appraisals or the tenure process, the rewards Extension professionals receive for evaluating their programs, and financial allocations set aside to allow for evaluation impact how much Extension professionals evaluate their programs (Arnold, 2006).The way a systems policies and procedures are structured also influenced professional development efforts as dictated by the structure of an organization (Burke, 2008). In addition, Arnold found the opportunity to learn about and increase skills in the area of evaluation through professional development efforts impacted individual Extension professionals' evaluation skills and abilities.

Along with professional development, structure includes defining levels of evaluative responsibility, communication regarding evaluation practices within the system, and how individuals within the organization interact in implementing evaluation (Burke & Litwin, 1992). Evaluation behaviors are strongly dictated by how well an organization is structured to create, capture, store, and disseminate data (Preskill & Boyle, 2008).

A positive organizational work unit evaluation climate is measured by the ways Extension professionals choose to talk about evaluation, their willingness to interact about and ask evaluative types of questions, their interest level regarding the use of evaluative data, and a group commitment to conducting meaningful and timely evaluations (Boyle, Lemaire, & Rist, 1999; Huffman, Lawrenz, Thomas, & Clarkson, 2006; McDonald, Rogers, & Kefford, 2003). The work unit evaluation climate is a key factor in establishing subjective norms around evaluation within individuals. The work unit climate is the primary social influence on an individual's belief structure, so how people the individual respects and of whose opinion holds value for the individual view evaluation (including peers, management, and leadership) will weigh heavily on how that same individual established their subjective norm of evaluation (Ajzen, 1991). In addition, if the work unit accepted and incorporated evaluation into the established climate, the individual feels more support, and therefore more control over using evaluation, enhancing their perceived control of engaging in evaluation behaviors (Ajzen, 2006).

Individual Performance Factors

In order to sustain individual evaluation behaviors within an organization long term, opportunities must be available for employees to learn from and about evaluation (Preskill & Boyle, 2008). The structure of the organization, including professional development efforts, directly impacts the individual evaluation skills of Extension professionals (Burke, 2008). Extension professionals must feel they have the competencies necessary to perform evaluation tasks in order to carry them out (Ghere, King, Stevahn, & Minnema, 2006). Extension professionals are hired for their subject matter knowledge (Rasmussen, 1989) and very rarely come with the competencies needed to properly evaluate. The amount of professional development and the skills/competencies Extension professionals within a state system have developed in regards to evaluation will impact their level of evaluation efforts (Arnold, 2006).

Beyond this, Extension professionals need to feel evaluating their programs holds personal and organizational value in order to be motivated to engage (Burke, 2008). Past research has shown evaluation use positively influences behavior regarding developing future evaluations because it offers a feeling that the work is truly valued and needed (Mackay, 2002; McDonald et al., 2003; Patton, 2008). The creation of a culture that uses the results of evaluations, thereby altering an individual's need to evaluate, can alter a person's attitude towards evaluation (Burke, 2008).

If an Extension professional feels the value of evaluating outweighs the cost and time associated with it, they will engage in evaluation practices (Ajzen, 2006). However, people experience tension when they are confronted with information that challenges their attitudes (Eagly & Chaiken, 1993). This is due to a feeling that their self-concept or personal identity is being challenged (Eagly & Chaiken, 1993). Individuals want to be seen a certain way, so the work unit evaluation climate (social atmosphere surrounding evaluation the individual works in) will have an influence on an individual's attitude toward evaluation (Ajzen, 2006). Research has shown that work unit evaluation climate, including talk about evaluation in the workplace, team members asking questions about each other's work, group interest in using data to make decisions, and the creation of group commitments to evaluation, directly impacted the individuals' attitudes about evaluation in a positive way (Boyle et al., 1999; Huffman et al., 2006; McDonald et al., 2003).

In addition to attitude, previous research has shown how powerful the need for conformity is on altering human behavior (Asch, 1956; Crutchfield, 1955; Sherif, 1935). The more difficult or ambiguous a task is, the more people rely on one another to develop and interpret the task at hand (Deutsch & Gerard, 1955; Kelley & Lamb, 1957). In addition, the more attractive belonging to a specific group is, the more likely an individual is to conform to their expectations (Festinger, Gerard, Hymovitch, Kelley, & Raven, 1952). An individual's established attitude is a measurement of how much they favor or disfavor an activity, therefore the attractiveness of a group has a strong reciprocal relationship with the subjective norm. It is expected that if evaluation practices are a norm within the work unit climate the individual is a part of and attracted to then they will feel more control over the practice, align with what is expected, and pursue engaging in evaluation behaviors (Ajzen, 2006).

Purpose and Objectives

The purpose of this research was to use a system-wide approach to examine how organizational evaluation structure influences the evaluation behaviors of Extension professionals in the field. The study was guided by the following objectives:

1. To identify Extension professionals' personal and professional characteristics, evaluation behavior, and perceptions regarding transformational evaluation factors, transactional evaluation factors, and individual performance evaluation factors.

2. To determine if, and how, Extension professionals' perceptions regarding transformational evaluation factors, transactional evaluation factors, individual performance evaluation factors, and personal and professional characteristics explain variation in Extension professionals' evaluation behavior.

Methods

Participants

The participants for this study consisted of the 1795 field-based Extension professionals employed by the University of Arizona, University of Florida, University of Maine, University of Maryland, Montana State University, University of Nebraska, North Carolina State University, and University of Wisconsin in 2010. Field-based Extension professionals were chosen as the target population due to their direct programming responsibilities, and therefore direct evaluation expectations. It is recognized that in some states, state Extension specialists conduct educational programming similar to field staff, and as such the exclusion of these individuals is a limitation of this study. Specific state systems were targeted to incorporate differences in organizational size, location, and hiring process. Responses were received from 1223 of the participants. Only 1173 were complete, resulting in a response rate of 65.2%.

A Chi-square test was run on the gender variable in order to identify if any significant differences existed between those who responded and the entire sample population based on a p value of <.05 established a priori. Differences in gender were non-significant with a Chi-square value of 2.00 and a p value of .16. A Chi-square test was also run on the gender variable to determine if any significant differences existed between respondents and non-respondents. Differences in gender were non-significant with a Chi-square value of 1.72 and a p value of .19.

Instrumentation

Due to a lack of previous research on the topic, an instrument measuring the variables of interest was not available, therefore the researcher created a 110 item instrument. The conceptual model previously described was used as the basis for specifying items for the instrument.

Participants were asked to report on how they evaluated their best or most important program by marking whether or not they had engaged in 29 specific data collection or data analysis methods during the past year. To assess participants' perceptions regarding the level their organization addresses transformational and transactional evaluation factors they were asked to rate their perception of specific items related to each factor on a Likert-type scale. The scale ranged from 1 - Strongly Disagree, 2 - Disagree, 3 - Neutral, 4 - Agree, 5 - Strongly Agree. The survey included questions regarding evaluation leadership, organizational evaluation culture, the management of evaluation, evaluation policies and procedures, work unit climate regarding evaluation, and structure pertaining to evaluation.

To assess each participant's individual performance evaluation factors, several types of questions were used (Ajzen, 2006). On questions related to individual needs and values regarding evaluation and individual evaluation skills and abilities the instrument requested participants rate their perception on a Likerttype scale. The scale ranged from 1 - Not at all true for me, 2 - Slightly true for me, 3 - Somewhat true for me, 4 - Mostly true for me, 5 - Completely true for me. Participants were asked to rate their perceptions of items associated with their attitude, subjective norm, and perceived behavioral control of evaluation on a semantic differential scale (Ajzen, 2006). This set of questions asked participants to rate their level of agreement between a set of bipolar adjectives. The instrument ended with a series of questions related to the individual's personal and professional characteristics.

The instrument was reviewed by a panel of experts specializing in survey design, evaluation, and familiarity with Extension organizations for internal validity and pilot tested on the Extension professionals employed by Colorado State University to check reliability. All of the final indexes created were considered acceptable with Cronbach's alpha coefficients ranging from .70 to .86 (Table 1).

Procedure

The instrumentation for the study was distributed using an online survey. The target population's access to the Internet and use of e-mail as a communication tool enabled the use of an online survey instrument (Dillman, Smyth, & Christian, 2009). The researcher designed and implemented the online survey by contacting the participants via e-mail using Dillman et al.'s (2009) Tailored Design Method including an initial request to complete the survey and weekly reminders for four weeks. A list of Extension professionals' names, e-mail addresses, and gender were generated by each state system. The state lists were combined, with state coded by the researcher to create a database of 1795 Extension professionals. A participant number was assigned to each individual. All correspondence related to the actual survey were sent by the researcher with approval from each state's Extension administration. Each state Extension director sent out an e-mail systemwide two weeks prior to the initial contact alerting the Extension professionals in their system they would be receiving an important survey.

Data Analysis

Quantitative research methods were used to achieve the research objectives. Data analysis for the study was completed using SPSS18 statistical software package for Windows. Descriptive statistics were used to accomplish the first objective. There were two stages to calculating the level of evaluation behavior dependent variable. First, the responses to the three program participation record items (keep records, track gender, track race/ethnicity) were summed. The sum score of these three items were multiplied by the response to the participation accuracy response (0 - not at all accurate, .25 - slightly accurate, .5 - somewhat accurate, .75 - mostly accurate, and 1 - completely accurate) to create a participation records score. Second, responses to the 26 remaining behavior items were summed to create an evaluation item score. The participation records score and the evaluation item score were then summed to create a score (with a possible range of 0 to 29) used as the evaluation behavior score in further data analysis.

Participant responses to the set of statements designed to measure each transformational, transactional, and individual performance evaluation factor were summed and averaged to create an overall score for each factor. Consistency of measurement within each factor was determined by using Cronbach's alpha coefficient. All of the final indexes created were considered acceptable, ranging from .70 to .86. Frequencies were reported for responses to each of the personal and professional characteristic items. Multiple regression analysis with hierarchical order of entry of predictor variables was performed to accomplish the second objective.

Results

Objective 1

Descriptive analysis of the personal and professional characteristic data collected showed there were 751 female (64.0%) and 422 male (36.0%) participants. The large majority (87.6%, n = 1027) of participants were Caucasian/White with African Americans representing 4.1% (n = 48). Hispanic, Native American, and Other categories were represented minimally. The majority of respondents (70.1%, n = 822) had obtained a Master's degree while 19.0% (n = 223) had a Bachelor's degree. All programmatic areas were represented with 27.1% (n = 318) of respondents focusing on Family and Consumer Sciences/Nutrition, 23.4% (n = 275) on 4-H Youth Development, 24.6% (n = 289) on Agriculture, and 11.2% (n = 131) on Horticulture. Almost half of the participants (43.1%, n = 505) were in tenure-tracked positions with 62.6% (n = 316) having already achieved tenure.

When the participants' evaluation behaviors were reviewed, it was discovered that 13.6% (n = 163) did not engage in the practice of evaluation. The majority of participants who did evaluate kept program participation records (n = 966, 82.4%), tracked their participants' gender (n = 841, 71.7%), used posttests to evaluate specific activities (n = 830, 70.8%), tracked their participants' race/ethnicity (n = 805, 68.6%), and conducted interviews to evaluate their activities (n = 760, 64.8%). Examining data analysis/reporting methods revealed the majority of participants who evaluated were just reporting the number of customers attending their program (n = 966, 82.4%). When calculated, the overall evaluation behavior scores of the participants ranged from 1.00 to 27.25 (M = 11.83, SD = 6.20) and were normally distributed.

The transformational, transactional, and individual performance factors expected to influence evaluation behavior were also examined (Table 2). Of the individual performance factors, participants had the highest level of agreement with a perceived subjective norm around evaluation (M = 4.16, SD = .67). In addition, participants reported a higher level of agreement towards needing and valuing the practice of evaluation (M = 3.96, SD = .69). There were stronger levels of agreement on almost all of the individual performance evaluation factors than factors at the transformational and transactional levels within their organization.

Objective 2

Initially both transformative evaluation factors, leadership and culture, had a significant, yet small, influence on the participants' evaluation behavior score variation (see Model 1 column in Table 3). However, when mediating factors were introduced in Models 2 - 4, the influence of the transformative evaluation factors became non-significant. When transactional evaluation factors were added to the model, structure and work unit climate emerged as having a significant positive influence on evaluation behavior (see Model 2 column in Table 3). Upon the addition of the individual performance factors, both structure and work unit climate became non-significant and management emerged as having a significant negative effect on evaluation behavior. In this model, skills and abilities had a large positive significant effect (4.42) along with the subjective norm having a large positive significant effect (1.29). Changes occurred again in the final model, where personal and professional characteristics were added (see Model 4 column in Table 3). In the last model, the only transformational or transactional variable having a significant effect was work unit climate (-.11). Individual skills and abilities and subjective norm remained significant in this model showing the personal and professional characteristics did not mediate their effect on evaluation behavior. In fact, the only personal and professional characteristic having a significant effect on evaluation behavior was an individual's tenure status. If a participant was accruing tenure, their level of evaluation behavior score was 1.06 points higher than a participant in a non-tenure tracked position.

A very small amount of the variation ([R.sup.2] = .02) in participants' evaluation behavior scores was explained by the first model that only included the organizational transformational factors (Table 4). The addition of transactional factors made a small significant contribution to the explanation of variance ([R.sup.2] Change = .02). However, the inclusion of individual performance factors made a much larger significant contribution to the explanation of variance ([R.sup.2] Change = .17). The third model explained 21% of the variance in Extension professionals' evaluation behavior scores. When the personal characteristics were added to create the last model, the change in explained variance was not significant at an alpha level of .05.

Conclusions

This national study gives insight into the evaluation behaviors Extension professionals are engaging in. It also sheds light on how Extension professionals perceive organizational and personal factors associated with evaluation and how those factors are contributing to variation in their evaluation behaviors. This study has shown Extension professionals are engaged in a wide variety of evaluation behaviors. However, the majority are only fulfilling basic reporting requirements. This data supports Franz and Townson's (2008) claim that the majority of Extension professionals currently utilize posttests given at the conclusion of their educational activities to assess the level of success. In general, Extension professionals reported some agreement with the view that their organization addresses evaluation at all levels. It is interesting to note Extension professionals reported a higher level of agreement with the individual performance evaluation factors than evaluation factors at the organizational level although the mean differences were not great. While Extension professionals are not being hired for their evaluation expertise, many do need and value evaluation and associate themselves as having some evaluation skills and abilities.

In addition, the results from this study are aligned with previous research on organizational change theory showing transformational, transactional, and individual performance factors have an effect on behavior choices (Burke, 2008). However, the findings from this study are incongruent with previous research on individual behavior change theory showing attitude, subjective norm, and perceived behavioral control directly impact behavior (Ajzen, 2006). In this study, only the subjective norm had a significant effect on behavior.

At the transformational level, leadership and culture had a significant effect on evaluation behavior when other factors were not controlled. This result support previous research suggesting leaders open to discussions surrounding the practice of evaluation, which encourage professional development in this area, and use evaluation data when making decisions will have a positive influence on increasing evaluation activities (Preskill & Boyle, 2008). The positive effects of these transformational factors were mediated by transactional factors. Structure and work unit climate were the two transactional factors which appeared as fundamental to influencing evaluation behavior. This is consistent with previous research showing that evaluation behavior is enhanced when Extension professionals understand their evaluative responsibilities, perceive open communication within the system regarding the practice of evaluation, and are encouraged to interact while implementing evaluations (Boyle et al., 1999; Burke & Litwin, 1992; Huffman et al., 2006).

The most influential factors were at the individual level. In this study an individual's evaluation skills and abilities had the strongest influence, indicating that the amount of professional development an Extension professional receives in regards to evaluation has a large impact on their level of evaluation efforts (Arnold, 2006). In addition, the subjective norm contributed significantly to variation in evaluation behavior. This supports the view that Extension professionals are relying on one another to make decisions regarding evaluation behaviors (Deutsch & Gerard, 1955; Kelley & Lamb, 1957).

Implications and Recommendations

Extension leaders need to recognize how their actions will have effects throughout their Extension system, albeit these are relatively weak and mediated by other factors. Through intentional action emphasizing the importance of evaluation, these individuals can have a system--wide impact, literally transforming their organization to one that values evaluation. This intentional action should include the creation of a statewide reporting system for creating, storing, and disseminating evaluation data (Preskill & Boyle, 2008). Extension leaders should also work toward shifting the culture of their organization to be more pro-evaluation and add structures that support the development of evaluation skills and abilities. New structures might include creating an evaluation focused website displaying the answers to common questions regarding evaluation issues and contact information for peers and specialists who understand and are willing to share their evaluation expertise. Individuals need to feel supported by their peers in order to see value in evaluating and feel confident in their abilities as shown by the influence the subjective norm around evaluation had on evaluation behavior. By placing an emphasis on establishing networks Extension professionals can turn to when faced with difficult decisions regarding evaluation, they will be more likely to engage in evaluation behaviors (Ajzen, 1991).

Most importantly, at the individual level, Extension professionals must have the skills and abilities to evaluate their programs in order to engage in evaluation. Since Extension professionals are hired for their subject matter knowledge (Rasmussen, 1989), evaluation focused professional development opportunities are essential to gaining these skills. Increased financial and human resources should be allocated to the development of evaluation skills including training on defining measureable objectives, creating logic models, working with stakeholders to establish evaluation needs, data analysis, reporting results, and communicating results effectively (Ghere et al., 2006). With a strong emphasis placed on training, so that Extension professionals have the skills necessary to evaluate and report findings, state and federal Extension systems will be more likely to obtain data showing programmatic worth.

This study shows changes at all levels within an Extension system will have an effect on Extension professionals' evaluation behaviors. By making major changes at all levels of the organization regarding the practice of evaluation, the perceived public value of Extension programs can be strengthened as the ability to report programmatic success improves. With increasingly limited resources available for government programs, the federal Extension system will be better positioned to justify the continuance of the funds it receives.

DOI: 10.5032/jae.2011.04136

References

Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. In J. Kuhl and J. Beckman (Eds.). Action-Control: From cognition to behavior (pp. 11-39). Heidelberg, Germany: Springer.

Ajzen, I. (1991). The theory of planned behavior. Organizational behavior and human decision processes, 50(2), 179-211. doi:10.1016/0749-5978(91)90020-T

Ajzen, I. (2006). Behavioral interventions based on the theory of planned behavior. Retrieved from http://www.people. umass.edu/aizen/pdf/tpb.intervention.pdf

Arnold, M. E. (2006). Developing evaluation capacity in extension 4-H field faculty. American Journal of Evaluation, 27(2), 257-269. doi:10.1177/1098214006287989

Asch, S. E. (1956). Studies of independence and conformity: A minority of one against a unanimous majority. Psychological Monographs, 70(9).

Bess, J. L. (1998). Contract systems, bureaucracies, and faculty motivation: The probable effects of a notenure policy. Journal of Higher Education, 69(1), 1-22.

Bourgeois, L. J. (1985). Strategic goals, perceived uncertainty, and economic performance in volatile environments. Academy of Management Journal, 28(3), 548-573.

Boyle, R., Lemaire, D., & Rist, R. C. (1999). Introduction: Building evaluation capacity. In R. Bolye & D. Lemaire (Eds.), Building effective evaluation capacity (pp. 1-19). New Brunswick, NJ: Transaction Publishing.

Burke, W. W. (2008). Organizational change: Theory and practice, 2nd edition. Thousand Oaks, CA: Sage Publications.

Burke, W. W., & Litwin, G. H. (1992). A causal model of organizational performance and change. Journal of Management, 18(3), 523-545. doi:10.1177/014920639201800306

Crutchfield, R. S. (1955). Conformity and character. American Psychologist, 10, 191-198.

Deutsch, M., & Gerrard, H. B. (1955). A study of normative and informational social influences upon individual judgment. Journal of Abnormal and Social Psychology, 51, 629-636.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd ed.). Hoboken, NJ: John Wiley & Sons, Inc.

Eagly, A.H., & Chaiken, S. (1993). The psychology of attitudes. Orlando, FL: Harcourt Brace Jovanovich, Inc.

Festinger, L., Gerard, H. B., Hymovitch, B., Kelley, H. H., & Raven, B. (1952). The influence process in the presence of extreme deviants. Human Relations, 5(4), 327-346. doi: 10.1177/001872675200500402

Fetsch, R. J., & Gebeke, D. (1994). A family life program accountability tool. Journal of Extension, 32(1). Retrieved from http://www.joe.org/joe/1994june/a6.php

Franz, N. K., & Townson, L. (2008). The nature of complex organizations: The case of cooperative extension. In M.T. Braverman, M. Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 5-14. doi: 10.1002/ev.272

Ghere, G., King, J. A., Stevahn, L., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108-123. doi: 10.1177/1098214005284974

Harder, A., Place, N. T., & Scheer, S. D. (2010). Towards a competency-based extension education curriculum: A Delphi study. Journal of Agricultural Education, 51(3), 44-52. doi: 10.5032/jae.2010.03044

Huffman, D., Lawrenz, F., Thomas, K., & Clarkson, L. (2006). Collaborative evaluation communities in urban schools: A model of evaluation capacity building for STEM education. New Directions for Evaluation, 109, 73-85. doi: 10.1002/ev.179

Jackson, D. G., & Smith, K. L. (1999). Proactive accountability: Building relationships with legislators. Journal of Extension, 37(1). Retrieved from http://www.joe.org/joe/1999february/a5.php

Kelley, H. H., Lamb, T. W. (1957). Certainty of judgment and resistance to social influence. Journal of Abnormal and Social Psychology, 55, 137-139.

Kotter, J. P., & Heskett, J. L. (1992). Corporate culture and performance. New York, NY: Free Press.

Lamm, A. J., Israel, G. D., & Irani, T. (2010, November). Organizational impacts of extension state systems on evaluation practice in the field. Paper presented at the American Evaluation Association National Conference, San Antonio, TX.

Mackay, K. (2002). The World Bank's ECB experience. New Directions for Evaluation, 93. San Francisco: Jossey-Bass. doi: 10.1002/ev.43

McDonald, B., Rogers, P., & Kefford, B. (2003). Teaching people to fish? Building evaluation capacity of public sector organizations. Evaluation, 9(1), 9-29. doi:10.1177/1356389003009001002

Osborne, E. W. (Ed.) (2007). National research agenda: Agricultural education and communication, 2007-2010. Gainesville, FL: University of Florida, Department of Agricultural Education and Communication.

Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443-459. doi: 10.1177/1098214008324182

Radhakrishna, R., & Martin, M. (1999). Program evaluation and accountability training needs of extension agents. Journal of Extension, 37(3). Retrieved from http://www.joe.org/joe/1999june/rb1.php

Rasmussen, W. D. (1989). Taking the university to the people: Seventy five years of cooperative extension. Ames, IA: Iowa State University Press.

Salancik, G., & Pfeffer, J. (1977). Constraints on administrator discretion: The limited influence of mayors on city budgets. Urban Affairs Quarterly, 12, 475-798.

Sherif, M. (1935). A study of some social factors in perceptions. Archives of Psychology, 27, 1-60.

Volkov, B. B., & King, J. A. (2007). A checklist for building organizational evaluation capacity. Retrieved from http://www.wmich.edu/evalctr/checklists/ecb.pdf

Warner, P. D., Rennekamp, R., & Nall, M. (1996). Structure and function of the Cooperative Extension Service. Lexington, KY: Extension Committee on Organization and Policy Personnel and Organization Development Committee.

Weiss, C. H. (1972). Evaluating educational and social actions programs: A treeful of owls. In C. H. Weiss (Ed.). Evaluating action programs (pp. 3-27). Boston, MA: Allyn and Bacon.

Wholey, J. S., Scanlon, J. W., Duffy, H. G., Fukumotu, J. S., & Vogt, L. M. (1970). Federal evaluation policy: Analyzing the effects of public programs. Washington, DC: Urban Institute.

Zacarro, S. J. (2001). The nature of executive leadership: A conceptual and empirical analysis of success. Washington DC: American Psychological Association.

Alexa J. Lamm, Policy Research and Evaluation Specialist

Glenn D. Israel, Professor

University of Florida

ALEXA J. LAMM is a policy research and evaluation specialist at the National Public Policy Evaluation Center for Agriculture and Natural Resources at the University of Florida, G086A McCarty Hall, Gainesville, FL 32611, alamm@ufl.edu

GLENN D. ISRAEL is a Professor of Extension Education in the Department of Agricultural Education and Communication at the University of Florida, 218 Rolfs Hall, Gainesville, FL 32611, gdisrael@ufl.edu
Table 1

Number of Items in each Index and Final Index Reliability

                                         Number     Reliability
                                         of Items   Coefficient
Index                                    in Index

Evaluation Leadership                        6          .85
Organizational Evaluation Culture            5          .79
Management of Evaluation                     6          .85
Evaluation Policies and Procedures           6          .70
Work Unit Evaluation Climate                 5          .85
Structure Pertaining to Evaluation           5          .81
Individual Needs and                         6          .82
Values Regarding Evaluation
Individual Evaluation Skills/Abilities      11          .84
Attitude towards Evaluation                  7          .86
Subjective Norm of Evaluation                6          .79
Perceived Behavioral Control                11          .80
of Evaluation

Table 2

Participants' Perceptions Regarding Transformational, Transactional,
and Individual Performance
Evaluation Factors

Evaluation Factor                 M     SD

Transformational Factors
  Leadership                     3.58   .65
  Culture                        3.53   .69
Transactional Factors
  Structure                      3.73   .74
  Management                     3.56   .77
  Policies and Procedures        3.42   .63
  Work Unit Climate              3.40   .78
Individual Performance Factors
  Subjective Norm                4.16   .67
  Needs and Values               3.96   .69
  Attitude                       3.90   .67
  Skills and Abilities           3.63   .59
  Perceived Behavioral           3.60   .62
  Control

Table 3

Multiple Regression of Evaluation Behavior on Selected Factors and
Characteristics

                                 Model 1   Model 2   Model 3

Transformational Factors
Leadership                       .72 *     .24       .22
Culture                          84 **     -.14      -.28
Transactional Factors
Structure                                  1.17 **   .11
Management                                 -.44      -.58 *
Work Unit Climate                          .63 *     -.12
Policies and Procedures                    .38       .17
Individual Performance Factors
Needs and Values                                     .30
Skills and Abilities                                 4.42 **
Attitude                                             -.47
Subjective Norm                                      1.29 **
Perceived Behavioral Control                         .27
Personal Characteristics
Non Tenure Tracked
Accruing Tenure
Achieved Tenure

                                 Model 4 (a)

Transformational Factors
Leadership                       .11
Culture                          -.24
Transactional Factors
Structure                        .12
Management                       -.50
Work Unit Climate                -.11 *
Policies and Procedures          .20
Individual Performance Factors
Needs and Values                 .18
Skills and Abilities             4.51 **
Attitude                         -.48
Subjective Norm                  1.27 **
Perceived Behavioral Control     .31
Personal Characteristics
Non Tenure Tracked
Accruing Tenure                  1.06 *
Achieved Tenure                  .46

Note. aModel 4 also included gender, race/ethnicity, education level,
and programmatic area variables which were non-significant. * p <.05,
** p < .01.

Table 4

Hierarchical Regression of Evaluation Behavior on Selected Factors
and Characteristics

                              Adjusted    [R.sub.2]               Sig.
                                                                   of
Variable Entered              [R.sup.2]    Change     F Change   Change

Transformational Factors         .02        .02       13.11       .00
Transformational &               .04        .02        6.59       .00
Transactional Factors
Transformational,                .21        .17       50.61       .00
Transactional, & Individual
Factors
Transformational,                .22        .01        1.18       .29
Transactional,
Individual Factors,
& Personal
Characteristics
COPYRIGHT 2011 American Association for Agricultural Education
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2011 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Lamm, Alexa J.; Israel, Glenn D.
Publication:Journal of Agricultural Education
Article Type:Report
Geographic Code:1USA
Date:Oct 1, 2011
Words:6357
Previous Article:Perceptions and barriers of four female agricultural educators across generations: a qualitative study.
Next Article:Greenhouse facility management experts identification of competencies and teaching methods to support secondary agricultural education instructors: a...
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters