The evaluation of risky information technology investment decisions.
Keywords: decision; investment; prospect theory; risk.
Data Availability: Data are available from the first author.
Managers and other decision makers must often take gambles in expensive and sometimes unproven technologies to survive in competitive marketplaces. The evaluation of information technology (IT) investment decisions is a critical component of the IT investment process that has been largely ignored in extant research. Organizations regularly evaluate decision performance, and evaluation represents a key organizational control feature (Brown and Solomon 1993). Performance evaluation processes encourage and reward decisions that are viewed favorably by the firm and punish and discourage decisions that are viewed unfavorably. As a result, the evaluation of IT investment decisions drives current and future investment decisions. It is important to investigate evaluations of IT investment decisions because most companies make IT investment decisions, the risks are high, evaluation processes drive future decision making, and firms must guard against biased evaluation processes.
Prior research (see, e.g., Tan and Lipe 1997; Hershey and Barton 1992, 1995) demonstrates that providing evaluators with pre-decision information (i.e., the information that was used by a decision maker to reach a decision) and information about the decision process can have beneficial effects on evaluations of decisions. When evaluators have access to predecision information and decision process information, they often consider how they would have made the decision, and base evaluations on their own interpretation of the predecision information (Brown and Solomon 1993; Hershey and Barron 1995; Tan and Lipe 1997). As a result of this tendency to evaluate decisions based upon personal preferences, we expect that evaluators' personal risk preferences and use of decision domain data will drive their evaluations of others' decisions.
This research extends theories of risk preference and prospect theory to evaluation processes in order to investigate the effects of outcome information, decision domains, and risk preferences on evaluations of investment decisions. Using M.B.A. students as participants, we find that evaluations of IT investment decisions are influenced by decision outcomes, dispositional risk preferences of evaluators, and the decision domains (gain versus loss) present during IT investment decision making.
The primary purpose of this research is to describe the evaluation processes used to evaluate risky IT investment decisions. This is among the first investigations of the joint effects of decision domains (i.e., gains and losses) and dispositional risk preferences on evaluations of risky decisions. This is also among the first studies to apply prospect theory to IT investment evaluation processes. The study has important implications for both theory and practice. This research contributes to theory in three ways. First, decision domain information currently may be available and can certainly be made available to evaluators of decision performance, and it is necessary to understand how this information influences evaluations before making critical control system choices. Second, prior researchers have not considered the possibility that the decision domain influences others' evaluations of IT investment decisions. Future experimental designs may need to control for the effects of decision domains on evaluation processes, and firms need to be aware of the influence of decision domains on evaluation processes. Third, understanding how evaluators' risk preferences affect their evaluations is essential to accurately describing the IT investment evaluation process.
Pragmatically, the evaluation processes for monitoring and controlling IT investment decisions may be ineffective if risk preferences and decision domains lead to inaccurate evaluations of investment decisions. Inaccurate evaluation processes may reward and encourage suboptimal IT investment decisions. Further, if evaluator characteristics or decision domains influence evaluation processes and promote non-utility-maximizing decisions, then performance evaluation processes will not serve their intended functions.
Overall, our results indicate that individual risk preferences and decision domains influence evaluations of risky investment decisions, such as IT investments. The primary findings from our experiment indicate that: (1) risk-seeking evaluators give higher evaluations to IT investment decisions than risk-averse evaluators; (2) risk-averse evaluators give higher evaluations to decision makers having unfavorable outcomes in gain domains than they give to decision makers having unfavorable outcomes in loss domains. (risk-seeking evaluators do not so differentiate) and (3) risk-seeking evaluators give higher evaluations to decision makers having favorable outcomes in loss domains than they give to decision makers having favorable outcomes in gain domains (but risk-averse evaluators do not so differentiate).
The remainder of the paper describes the relevant literature and hypothesis development, followed by a description of the design, methods, and results. The final section includes discussion and conclusions, as well as limitations.
II. LITERATURE REVIEW AND HYPOTHESES DEVELOPMENT
Information Technology Investment Decisions
Over $2 trillion was spent in the year 2000 on investments in information technology (IT) (World Information and Technology Services Alliance [WITSA] 2000). The changing nature of hardware/software requires constant upgrades and investments in new technologies, and the substantial cost of recurring IT investments indicates that poor IT investment decisions can end the life of a business (Hinton and Kaye 1996). As a result, control of IT investment decisions is critical to firm survival and success. Prior research also finds that most large-scale IT investments have a champion who is held personally responsible for decision outcomes, and the success or failure of an IT investment decision often determines a champion's career path (Hinton and Kaye 1996). Consequently, the quality of IT investment decisions influences the future of firms and their decision makers.
Managers recognize that IT investment decisions impose significant risk. Firms typically require traditional cost/benefit figures and discounted cash flow analysis more often for IT investments than any other investment decision, and IT investment decisions are evaluated against these measures (Violino 1998; Hinton and Kaye 1996; Willcocks and Lester 1991). Further, to compensate for the high risk of IT investments, firms often set rate-of-return hurdles arbitrarily high (Clemons and Weber 1990). In sum, IT investment decisions occur frequently, are costly, significantly affect firm performance, affect decision maker's careers, and require the assimilation and use of accounting information. Given that IT investment decisions are among the riskiest decisions, theories of decision making under risk are of great relevance to the evaluations of IT investment decisions.
Risk Preferences and Prospect Theory
Numerous models describe decision making under ambiguity and risk. Most of these models focus on either the dispositional characteristics of individuals (e.g., Young 1985; Waller 1988) or the characteristics of the situational context (e.g., Kahneman and Tversky 1979). The dispositional view suggests that risk preference is a personality trait that remains constant over time and context (Young 1985; Waller 1988). Based strictly on dispositional characteristics, risk-seeking individuals should prefer risky choices, and risk-averse individuals should prefer riskless choices. Since evaluators base their evaluations on the choices they would have made themselves (Brown and Solomon 1993; Hershey and Barron 1995; Tan and Lipe 1997), the risk preferences of evaluators should influence evaluations of risky IT investment decisions. Risk-seeking evaluators will prefer risky choices, and give higher evaluations to risky decisions than risk-avoiding evaluators.
H1: Risk-seeking evaluators will give higher evaluations to risky IT investment decisions than will risk-avoiding evaluators.
Other research finds that: (1) risk preferences can change with decision context, and (2) risky choices are not solely a function of dispositional characteristics (e.g., Kahneman and Tversky 1979; Libby and Fishburn 1977). This is one basic premise of prospect theory--individuals seek risk in loss decision domains and avoid risk in gain decision domains. Kahneman and Tversky's (1979) prospect theory is a widely accepted behavioral model of risky decision making. Prospect theory suggests that individuals avoid risk when they perceive the current state to be positive (a gain decision domain), and individuals seek risk when they perceive the current state to be negative (a loss decision domain). Determination of the gain or loss decision domain depends on an individual's reference point. The reference point is determined by comparing the current state to some prior state, or, by comparing the current state to the state of a peer group (Fishburn 1977; Kahneman and Tversky 1979).
Tversky and Kahneman (1981) describe situations where a decision maker's current state places the decision maker in a gain or loss domain. For example, they describe this scenario: "Consider a person who has spent an afternoon at the race track, has already lost $140, and is considering a $10 bet" (Tversky and Kahneman 1981). The gambler has fallen behind an initial reference point of no losses and is now in a loss domain. The new reference point is a loss of $140, and the gambler will seek out long-shot bets to return to a neutral reference point. Conversely, a gambler who had prior winnings would be above a neutral reference point and would avoid long-shot bets. Tversky and Kahneman (1981) are asserting that the person's current situation causes them to frame the betting decision a certain way.
While research demonstrates different decision behaviors under gain and loss domains (see, e.g., Kahneman and Tversky 1979; Tversky and Kahneman 1981; Lipe 1993; Tan and Lipe 1997), we know of no research that examines how evaluators consider a decision maker's decision domain in evaluating a decision. Do evaluators recognize the decision domain and use this information in evaluations of decisions? In this study we do not manipulate the decision domain of the evaluators in the experiment. We instead reveal the decision domain of a decision maker. The concept that decision domains influence future evaluations of their decisions is a novel application of prospect theory that is uninvestigated in prior research.
When evaluators evaluate others' decisions, they base their evaluations largely on decision outcomes (e.g., Tan and Lipe 1997; Lipe 1993; Lipshitz 1989; Mitchell and Kalb 1981). Research (e.g., Hershey and Barron 1992; Fischhoff 1983) suggests that evaluators should focus on the decision processes employed by decision makers and the information that was available to decision makers during the decision-making process (we will term this information "predecision information"), rather than decision outcomes. We contend that providing predecision information to evaluators may influence evaluations in systematic ways, because of personal risk preferences and evaluators' use of decision domain data.
Evaluators often make personal assessments of the best decision and then evaluate decision makers based upon conformance with their personal beliefs (Brown and Solomon 1993; Hershey and Barron 1995). Evaluators "internalize" decision outcomes more when the evaluated decision does not match the evaluator's decision (Fisher and Selling 1993). That is, evaluators hold a decision maker more responsible for outcomes when the decision maker's decision does not match the decision that the evaluator would have made, given the same information. Research has also demonstrated that evaluators reward decisions with which they agree, and that agreement with decisions can attenuate the effects of outcomes on evaluations (Brown and Solomon 1993; Fisher and Selling 1993). The current research proposes that risk preferences and the decision domains faced by decision makers influence subsequent evaluations of those decisions largely due to the effects of agreement between evaluators and decision makers.
Kim (1992) finds that dispositional risk preferences and contextual risk preferences can interact to affect decisions. This leads us to expect that risk-seeking evaluators will agree more with risky decisions than risk-averse evaluators, but decision domains will also influence evaluations. Prospect theory indicates that decision makers with risky choices seek risk in loss domains and avoid risk in gain domains. When evaluators review risky IT investment decisions and the predecision information available to a decision maker, they will be more likely to agree with risky choices made in loss domains than risky choices made in gain domains. Therefore, when decision makers make risky choices in gain domains, evaluators will attribute more responsibility for outcomes to the decision makers than when risky choices were made in loss domains (i.e., internalize the outcome more).
In a review of the performance evaluation literature, Zuckerman (1979) finds that when outcomes conform to evaluators' expectations, the evaluators internalize the outcomes more than when outcomes deviate from their expectations. Therefore, if an evaluator expects an unfavorable outcome, and the actual outcome is unfavorable, the evaluator will place most of the blame on the decision maker. However, if evaluators expect a favorable outcome, then they place less blame on the decision maker for a poor outcome. Research into dispositional risk preferences indicates that risk seekers believe that risky situations can be controlled, while risk avoiders believe that risky situations are out of a decision maker's control. For decisions that involve high risk, risk seekers are more likely to believe that favorable outcomes will occur than risk avoiders (Krueger and Dickson 1994; Cooper et al. 1988; March and Shapira 1992). This results in an expectation of more favorable outcomes by risk seekers relative to risk avoiders. Similarly, risk avoiders have greater expectations of unfavorable outcomes than risk seekers.
Taken together, the prior studies on internalization of outcome expectations indicate an expected interaction of decision domains, decision outcomes, and risk preferences. Risk-averse evaluators internalize unfavorable outcomes more than favorable outcomes, and internalize outcomes more when decisions are made in gain domains relative to loss domains. Alternatively, risk-seeking evaluators internalize favorable outcomes more than unfavorable outcomes, but still internalize outcomes more when decisions are made in gain domains relative to loss domains. The interactive effects of outcomes, decision domains, and risk preferences lead to the following two hypotheses.
H2a: Risk-averse evaluators will less favorably evaluate decision makers with unfavorable outcomes in gain domains than decision makers with unfavorable outcomes in loss domains.
H2b: Risk-seeking evaluators will more favorably evaluate decision makers with favorable outcomes in gain domains than decision makers with favorable outcomes in loss domains.
III. RESEARCH METHOD
We conducted the experiment using 162 M.B.A. students enrolled in graduate accounting courses in an executive M.B.A. program. The participants acted as decision evaluators. The average participant age was 28 years, and the average total of professional work experience was 5.9 years. The average amount of professional work experience in a supervisory position was 2.0 years. Most participants indicated that they had previously been involved in the performance evaluation of subordinates and their decisions. Participants came from very diverse business and cultural backgrounds. As a result, the participant pool represented a wide cross-section of the population of experienced business people. These M.B.A. participants are appropriate for studying the effects of risk preference and prospect theory on evaluations because most participants had worked in management positions that involve supervising and evaluating subordinates, most participants had prior experience in evaluating others' decisions, and a majority of our participants were working in managerial positions while enrolled in the executive M.B.A. program. Further, M.B.A. students (and other graduate students) have been employed in several seminal studies of the evaluation processes employed by business professionals (see, e.g., Brown and Solomon 1987; Tan and Lipe 1997).
Two participants who did not complete the entire instrument were removed from the sample, resulting in a final sample size of 160 participants. In order to motivate student participants to perform during the task, the experiment was administered as a graded assignment. Participants were informed that although some portions of the task involved personal judgment, "successful and rigorous completion of all portions of the task" was required in order to receive course credit.
The design of the study was a 2 x 2 x 2 mixed factorial. The within-participants factors were decision outcome (favorable versus unfavorable) and decision domain (gain versus loss). We chose to implement decision domain and decision outcomes as within-participants variables to increase statistical power. Within-participant designs can generate demand effects if participants discover the research purpose (Pany and Reckers 1987). We took several precautions to avoid demand effects. Specifically, we randomized the order of experimental materials across participants and designed the instrument to disguise the purpose of the experiment. The between-participants independent variable was risk preference, which was measured using a lottery problem from Kim (1992) and an alternative risk preference measure. Risk preference measures were collected at the conclusion of the experiment. The dependent variable was the evaluation of IT investment decisions. Participants were the evaluators and decision makers were fictitious managers who had made IT investment decisions.
Task and Procedure
The task involved the evaluation of IT investment decisions. Participants acted as evaluators of the decision performance of IT investment decision makers. Each participant received four packets with predecision information, outcome data, and decision domain data for four IT managers of a fictitious company. (1) The order of the packets was randomized across participants. The predecision information included present value analyses that were originally conducted by the decision makers who made the investment decisions. (2) Outcome data consisted of actual cash inflows for the first year following the investment decision (ex post decision data).
Much prior research investigating the effects of decision information and outcome data on evaluations used expected value (EV) calculations to reveal good versus poor decision processes (Baron and Hershey 1988; Mowen and Stone 1992; Lipe 1993). In these previous studies, decisions that conformed to EV were said to be good decisions, and those that deviated from EV were poor decisions. All the decisions presented to evaluators in our task were consistent with EV. Therefore, all decisions would be deemed appropriate by EV calculations. Only the outcomes of the decisions and the decision frame differed between treatments. We assume that only these variables influence the evaluations.
Two fictitious decision makers had made IT investment decisions with favorable outcomes and two fictitious decision makers had made IT investment decisions with unfavorable outcomes. Favorable outcomes consisted of favorable variances from the original estimates of first year cash inflows, and unfavorable outcomes consisted of unfavorable variances. For each evaluation of a decision with a favorable outcome or an unfavorable outcome, participants also received additional information that was available to the decision maker at the time of the original IT investment decision. This information indicated whether the decision maker made the IT investment decision in a gain or loss decision domain as described in prospect theory. In our study, gain domains obtained when decision makers managed divisions that possessed superior technology relative to other divisions. Loss domains obtained when decision makers managed divisions with inferior technology.
Prior research finds that decision makers view the performance of peers as a reference point when making risky decisions (Fishburn 1977; Kim 1992). We employed the relative performance of a company's divisions as the reference point decision makers used when making investment decisions. That is, when a division was performing worse than other divisions in an organization, investment decisions were made from a loss domain. Similarly, when a division was outperforming other divisions on average, investment decisions were made from a gain domain. The manipulation of decision domain was accomplished by describing the current position of a division relative to other divisions in the firm at the time an IT investment decision had been made. This predecision information reveals the reference point, and therefore, the decision domain of the original decision maker. We did not directly state that the decision maker made the decision in a gain (loss) domain, because such a manipulation would be unrealistic and participants would likely not understand the terminology.
The IT investment decision makers in loss domains managed divisions that were described in terms such as "well behind many other divisions." Consequently, the IT manager was "losing" based on prior outcomes. The IT decision makers were behind the reference point of matching their competitor divisions. (3) We performed manipulation checks to verify that our evaluators recognized that a decision maker was either winning or losing based on prior outcomes.
Our manipulation of decision domains is unique in the prospect theory literature. Prior prospect-theory experiments that we are aware of manipulated the decision frame of the actual decision maker and then tested for adherence to prospect theory by the decision maker (see, e.g., Kahneman and Tversky 1979; Tversky and Kahneman 1981; Lipe 1993; Tan and Lipe 1997). We ask participants to evaluate the decisions of IT decision makers, and we manipulate the domain that the IT decision makers had faced when their decision was made.
For each decision maker from each of the four fictitious divisions, participants analyzed the predecision information, outcome data, and decision domain, and then rated each of the four decision maker's IT investment decision on an evaluation scale. (4) An example of the scale used in the experiment is as follows:
David's decision to invest in the new information technology was (make a slash on the scale below):
After evaluating all four decision makers, participants completed a demographics questionnaire. The questionnaire gathered information on work experience and attitudes toward evaluation processes. Finally, we measured each participant's risk-seeking propensity. The following question, from Kim (1992), measures an individual's risk-seeking behavior:
Would you be willing to participate in a gamble where you have a 50 percent chance of winning $10 and a 50 percent chance of losing $10, or would you prefer not to play the gamble? Indicate your willingness to play the gamble on the scale below: 100
Based strictly on utility, participants should be indifferent about the gamble. Therefore, the preference ratings capture differences in dispositional attitudes toward risk. Participants who are risk-seeking will indicate a preference level greater than 50, and risk-averse individuals will attempt to avoid the gamble (i.e., indicate a preference level below 50). We classified participants with scores over 50 as risk-seeking, and participants with scores of 50 or below as risk-avoiding. (5) The alternative measure of risk preference was a self-reported measure of risk preference. Participants rated their risk preference on a 0-100 bipolar scale anchored on risk avoiding and risk seeking. Classification of participants using this self-reported measure of risk preference was identical to the classification achieved with the gamble question for all but five of our participants (Pearson correlation between risk measures = 0.848).
A rating scale on the demographics questionnaire tested the manipulation of the gain versus loss domain. The final evaluation for half of the participants involved an IT investment decision made in a gain domain, and the final evaluation for the remaining participants involved a decision made in a loss domain. Participants rated the division's performance (from the last rating) relative to other divisions in the firm on a scale of 0 to 100. (6) Participants evaluating the gain domain decision gave a mean rating of 82, while participants evaluating loss domain decisions gave a mean rating of 41. The difference in means was significant (p < .01), indicating that the gain/loss domain manipulation was effective.
Table 1 presents descriptive analyses of the participant demographic data and risk preferences. There were no main effects of gender, age, work experience, or supervisory experience on performance evaluations. As a result, these variables were excluded from the statistical analyses. (7) There were no statistically significant differences in demographic data across the risk preference categories.
Twelve participants indicated no preference on the risk preference scale (i.e., a rating of 50). The analyses presented below classify these participants as risk-averse, following prior research (Kim 1992). One could argue that these participants are neither risk-averse nor risk-seeking. All analyses were also conducted with these participants removed; the results of all hypothesis tests remain unchanged.
Table 2 displays the mean evaluation ratings for all combinations of outcomes, decision domains, and risk preferences. The data indicate that evaluations are higher when outcomes are favorable relative to unfavorable and that risk-seeking evaluators give higher evaluations than risk-averse evaluators in most cases.
The first hypothesis posits that risk-seeking evaluators will give higher evaluations to decision makers choosing to make investments in risky technology than risk-averse evaluators. The means in Table 2 indicate that evaluations are higher for risk-seeking evaluators in three of the four combinations of decision domain and outcome. Across all outcome and decision domain conditions, the average evaluation score for risk-seeking evaluators is 67.95, while the average evaluation score for risk-averse evaluators is 63.45. We tested the first hypothesis using a 2 (favorable versus unfavorable outcome) x 2 (gain versus loss decision domain) x 2 (risk-seeking versus risk-avoiding) mixed ANOVA model. The dependent variable in the model is evaluation rating. The risk preference variable is statistically significant (F = 4.18, p < .043), providing support for the first hypothesis (see Table 3).
The remaining hypotheses predict joint effects of risk preferences and decision domains on evaluations of risky IT investment decisions. The means in Table 2 indicate that our participants with risk-avoiding preferences give higher evaluations in loss domains (mean = 57.41) relative to gain domains (mean = 47.37) for unfavorable outcomes. Hypothesis 2a posits that when outcomes are unfavorable, risk-averse evaluators will attribute unfavorable outcomes more to the decision maker when decisions were made in gain domains relative to loss domains. The means provide preliminary support for this hypothesis. Similarly, the means in Table 2 provide some support for H2b. When outcomes are favorable, risk-seeking evaluators will attribute the favorable outcomes more to the decision maker when decisions were made in gain domains (mean = 84.68) relative to loss domains (mean = 80.35).
We employed a series of planned orthogonal contrasts to test H2a and H2b. To test H2a, we conducted a planned contrast of the evaluations made by risk-averse participants in the unfavorable outcome condition across the gain and loss domain manipulations. The difference is statistically significant (t = 3.45, p < .001). A similar planned contrast of the evaluations made by risk-averse participants in the favorable outcome condition across the gain and loss domain manipulations finds no significant difference in evaluations across decision domains (t = 1.26, p <. 15). Risk-averse evaluators internalize decision outcomes more when outcomes are unfavorable and decisions were made in gain domains.
Finally, to test H2b, we conducted planned orthogonal contrasts of the differences in evaluations across decision domains for risk-seeking managers in both the favorable and unfavorable outcome conditions. The difference between evaluations of decisions made in gain versus loss domains is statistically significant when outcomes are favorable (t = 3.117, p < .005), but not when outcomes are unfavorable (t = .888, p < .25). Figure 1 displays the three-way interaction of outcomes, decision domains, and risk preference. Risk-averse individuals make large downward adjustments to evaluations of unfavorable outcome decisions in gain domains relative to loss domains, while risk-seeking individuals upwardly adjust evaluations of favorable outcome decisions in gain domains relative to loss domains.
[FIGURE 1 OMITTED]
V. MANIPULATION CHECKS AND VALIDATION TESTING
There remain two issues that could significantly affect the interpretation of our results. First, do our findings hold only for risky investment decisions, or to all investment decisions? lf we find precisely the same pattern of results for low-risk decisions and high-risk decisions, then it would appear unlikely that theories of decision making under risk provide reliable depictions of evaluations of risky IT investment decisions. Second, are IT investment decisions inherently different from any other form of investment decision? Prior research indicates that managers believe that IT investment decisions are more risky than most investment decisions because they require more financial data and set higher hurdle rates for IT investments than other investment decisions (Violino 1998; Hinton and Kaye 1996; Willcocks and Lester 1991; Clemons and Weber 1990). We performed additional experimentation to directly address managers' beliefs about IT investment risks. IT investment decisions were considered more risky than other investment decisions.
To test for differences between evaluations of risky and low-risk investment decisions, two additional treatments were examined. Twenty-four participants were given our loss domain/unfavorable outcome condition, but the investment decision was changed from a high-risk technology investment to a low-risk technology investment (i.e., an investment in personal computers). Another 22 participants evaluated the gain domain/unfavorable outcome decision, again with the investment decision changed from a high-risk technology investment to a low-risk technology investment. All participants also completed the risk preference measure. These new treatment groups made it possible to analyze the effects of risk preference on low-risk decisions when outcomes are unfavorable. Unlike the results of the main experiment, risk-averse decision makers did not differ in their evaluations of the gain or loss domains when the outcome was unfavorable. It appears that evaluators do not consider the decision domains of decision makers when evaluating decisions made under low risk. Prospect theory does not appear to apply to evaluations of low-risk investment decisions, and our results are specific to risky decisions.
Understanding how managers evaluate risky IT investment decisions is critical to improving organizational controls over decision making. Performance evaluation systems cannot be improved without understanding performance evaluation processes. For example, some researchers espouse the benefits of providing evaluators with predecision information (e.g., Tan and Lipe 1997; Hershey and Barton 1995), but the full effects of predecision information on evaluation processes have not been examined. Our findings indicate that providing decision domain information leads to systematic differences in evaluations that will influence future investment decisions. A key contribution from this study is the finding that prospect theory can be applied to evaluation processes, because evaluators recognize and use decision domain data in their evaluation process.
Results also suggest that future studies examining performance evaluation processes will need to control for the effects of decision domains and evaluators' personal risk preferences on performance evaluations. We find that risk preferences and decision domains interact to determine evaluations of high-risk decisions. The evaluation processes used to control IT investment decisions can result in disparate evaluations, depending on the risk preferences of the evaluator. As a result, future decisions by IT investment decision makers will be partially determined by the risk preferences of evaluators.
If evaluation processes are systematically influenced by the risk preferences of evaluators or the decision domains that IT managers face, then performance evaluation processes may fail to promote firms' investment objectives. For example, we find that risk-seeking managers reward risky decisions more than risk-averse managers under most circumstances. Firm-wide IT investment strategies may be partially driven by these individual risk preferences, and firms may need to monitor the risk preferences of evaluators.
Our research has three important limitations. First, M.B.A. students are imperfect surrogates for the managers who are likely to evaluate high-level decisions like decisions to invest in costly information technology. Managers who evaluate high-level decisions often will be experienced executives who may have learned how to override personal preferences in favor of corporate goals. Participants in controlled experimental settings will never behave identically to decision makers facing the myriad of competing forces and incentives present in the real world. Second, for the sake of experimental control and comparability to prior research, we provided financial-based outcome measures to indicate the success or failure of IT investment decisions. Information technology investments involve many intangible costs and benefits that cannot be measured with cash flows and expected value analyses (Hinton and Kaye 1996; Willcocks and Lester 1991). We did not provide participants with intangible cost or benefit data. Consequently, we are unable to determine the potential effects of intangible costs or benefits on decision evaluations based upon our findings. Finally, this is the first study to apply prospect theory to evaluation processes, and the experiment was specific to a small set of controlled decisions and outcomes. Additional research will be needed to validate the applicability of prospect theory to more complex evaluation processes.
The purpose of this research was to describe the effects of outcomes, decision domains, and risk preferences on decision evaluations. In addition, we extended prospect theory to the evaluation of risky decisions in order to explain how decision domains and risk preferences can influence evaluation processes. Evaluators who are provided with information that was used by a decision maker to make an investment decision are able to recognize the decision frame faced by the decision maker. The evaluators consider this decision frame and make personal assessments of the decision that should have been made, given the information that was available. The evaluators adhere to the predictions of prospect theory, and determine that they would be more likely to make risky IT investment decisions in loss domains. Finally, we find that in general, risk-seeking managers tend to reward risk-taking behavior more than risk-averse managers.
TABLE 1 Demographic Data Continuous Variables Min. Max. Median Mean S.D. Age 20 44 26 27.55 5.37 Years of Experience 0 38 4 5.92 6.95 Supervisory Experience 0 22 0.5 2.00 3.08 Risk Preference 0 100 65 58.82 31.31 Risk (a) Categorical Variables Averse Seeking Total Gender: Male 38 65 103 Female 23 34 57 Total 61 99 160 (a) Participants were classified as risk-seeking when they indicated a preference level greater than 50 on the gamble, and participants were classified as risk-averse when they indicated a preference level of 50 or below. TABLE 2 Mean Evaluation Ratings for All Combinations of Outcomes, Domains, and Risk Preferences Decision Domain (a) Outcome (b)/Risk Preference Gain Loss Favorable Risk-Averse 75.78 73.24 Risk-Seeking 84.68 80.35 Unfavorable Risk-Averse 47.37 57.41 Risk-Seeking 52.51 54.27 (a) The decision domain refers to the domains described in prospect theory. Gain domains exist when individuals perceive the current state to be positive, and loss domains exist when individuals perceive the current state to be negative. (b) Outcomes are the outcomes of the IT investment decisions. Outcomes could either be favorable (above expectations) or unfavorable (below expectations). TABLE 3 Mixed ANOVA: Effects of Outcomes, Decision Domains, and Risk Preferences on Evaluations Sum of Mean Source Squares df Square F Sig. Between Subjects Risk Preference (a) 705 1 705.00 4.18 0.043 Error 26630 158 169.00 Within Subjects Outcome (b) 91242 1 91242 274.21 0.000 Outcome * Risk 1708 1 1708 5.13 0.025 Error (Outcome) (c) 52578 158 332 Decision Domain (d) 212 1 212 1.24 0.267 Domain * Risk 876 1 876 5.12 0.025 Error (Decision Domain) 27026 158 171 Outcome * Domain 3031 1 3031 27.71 0.000 Outcome * Domain * Risk 367 1 367 3.35 0.069 Error (Outcome * Domain) 17285 158 109 (a) Risk preference was measured with a gamble preference from Kim (1992). Participants with preference scores over 50 were classified as risk-seeking, and participants with scores of 50 or below were classified as risk-averse. (b) Outcomes are the outcomes of the IT investment decisions. Outcomes could either be favorable (above expectations) or unfavorable (below expectations). (c) Separate error terms are used for the F-tests. Pooled error term tests are less conservative and more susceptible to violations of the assumption of homogeneity of error variances. The separate error term tests are generally preferred over pooled-error tests for within-participant designs with more than one within-participant factor (Maxwell and Delaney 1990, 507). (d) The decision domain refers to the domains described in prospect theory. Gain domains exist when individuals perceive the current state to be positive, and loss domains exist when individuals perceive the current state to be negative.
The authors thank Dan Stone, the anonymous reviewers, and participants of the 2003 Information Systems Midyear Meeting for helpful comments on earlier versions.
(1) Participants actually received six packets with information about six different IT investment decisions. Two of these packets contained probe evaluations that are not included in the analyses. The probe evaluations were included to prevent participants from recognizing the experimental manipulations and the purpose of the experiment. The probe evaluations described decisions to invest in technologies (personal computers and well-established accounting software) with more certain and easily measurable outcomes.
(2) Participants had received training on the use of present value analyses to make investment decisions early in the semester' and they were very familiar with present value concepts. In all four cases, present value analysis yielded a positive net present value after five years, and the total net present value was very similar across eases.
(3) Tversky and Kahneman (1981) indicate that people who have suffered losses from past outcomes attempt to increase their reference point to neutral by seeking risk. Therefore, loss domains produce more risk seeking than gain domains, and decision makers who have suffered losses in the past are more likely to take bets that they would normally find unacceptable (Tversky and Kahneman 1981). The authors describe similar scenarios for people in gain domains. Decision makers who are ahead based on prior outcomes tend to avoid risk to allow their reference point to remain positive.
(4) Participants evaluated the four managers in random order to prevent any bias in evaluations as a result of a learning curve or order effect.
(5) This classification method was employed by Kim (1992) in his study of risk preferences. Alternative methods of classifying participants were also employed and are discussed in the results section.
(6) We used the last rating for our manipulation check because participants completed the demographics questionnaire after the final evaluation. The final evaluation was still active in participants' memories, and they were able to recall the details of the last evaluation better than previous evaluations.
(7) When the years of experience variable is converted to a categorical variable based on a mean split, the categorical variable is statistically significant in the model (p < .10). All of the hypothesized results remain unchanged when the categorical variable is included as a between-participant variable in the mixed ANOVA model. Participants in both experience categories exhibited the hypothesized effect of loss domains on evaluations, but the effect was slightly stronger for the more experienced participants.
Baron, J., and J. Hershey. 1988. Outcome bias in decision evaluation. Journal of Personality and Social Psychology 54: 569-579.
Brown, C., and I. Solomon. 1993. An experimental investigation of explanations for outcome effects on appraisals of capital-budgeting decisions. Contemporary Accounting Research 10: 83-111.
Clemons, E., and B. Weber. 1990. Strategic information technology investments: Guidelines for decision making. Journal of Management Information Systems 7: 9-28.
Cooper, A., W. Dunkelberg, and C. Woo. 1988. Entrepreneur's perceived chances for success. Journal of Business Venturing 3: 97-108.
Fischhoff, B. 1983. Decision analysis: Clinical art or clinical science? In Human Decision Making, edited by L. Sjoberg, T. Tyszka, and J. Wise. Bodfors, Sweden: Doxa.
Fishburn, P. 1977. Mean-risk analysis with risk associated with below-target returns. American Economic Review 67: 116-126.
Fisher, J., and T. Selling. 1993. The outcome effect in performance evaluation: Decision process observability and consensus. Behavioral Research in Accounting 5: 58-77.
Hershey, J., and J. Barton. 1992. Judgment by outcomes: When is it justified? Organizational Behavior and Human Decision Processes 53: 89-93.
--, and --. 1995. Judgment by outcomes: When is it warranted? Organizational Behavior and Human Decision Processes 62: 127.
Hinton, M., and R. Kaye. 1996. Investing in information technology: A lottery? Management Accounting-London 74: 52.
Kahneman, D., and A. Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica 47: 263-291.
Kim, D. 1992. Risk preferences in participative budgeting. The Accounting Review 67:303-318.
Krueger N., and P. Dickson. 1994. Self-efficacy and opportunity recognition. Decision Sciences 25: 385-400.
Libby, R., and P. Fishburn. 1977. Behavioral models of risk taking in business decisions: A survey and evaluation. Journal of Accounting Research 15: 272-292.
Lipe, M. 1993. Analyzing the variance investigation decision: The effects of outcomes, mental accounting, and framing. The Accounting Review 68: 748-764.
Lipshitz, R. 1989. Either a medal or a corporal: The effects of success and failure on the evaluation of decision making and decision makers. Organizational Behavior and Human Decision Processes 44: 380-395.
March, J., and Z. Shapira. 1992. Variable risk preferences and the focus of attention. Psychological Review 99: 172-183.
Maxwell, S., and H. Delaney. 1990. Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
Mitchell, T., and L. Kalb. 1981. Effects of outcome knowledge and outcome valence on supervisor's evaluations. Journal of Applied Psychology 66:604-612.
Mowen, J., and T. Stone. 1992. An empirical analysis of outcome biases in constituent evaluations of public policy decision makers. Journal of Public Policy and Marketing 11: 24-30.
Pany, K., and P. Reckers. 1987. Within- versus between-subject experimental designs: A study of demand effects. Auditing: A Journal of Practice & Theory 7: 39-53.
Tan, H., and M. Lipe. 1997. Outcome effects: The impact of decision process and outcome controllability. Journal of Behavioral Decision Making 10: 315-325.
Tversky, A., and D. Kahneman. 1981. The framing of decisions and the psychology of choice. Science 211: 453-458.
Violino, B. 1998. ROI in the real world. Information Week 679: 60-72.
Waller, W. 1988. Slack in participative budgeting: The joint effect of a truth-inducing pay scheme and risk preferences. Accounting, Organizations and Society 13: 87-98.
Willcocks, L., and S. Lester. 1991. Information systems investments: Evaluation of the feasibility stage of projects. Technovation 11: 283-302.
World Information and Technology Services Alliance (WITSA). 2000. Digital Planet 2000. World Information and Technology Services Alliance. Arlington, VA: WITSA.
Young, M. 1985. Participative budgeting: The effects of risk aversion and asymmetric information on budgetary slack. Journal of Accounting Research 23: 829-842.
Zuckerman, M. 1979. Attribution of success and failure revisited, or: The motivational bias is alive and well in attribution theory. Journal of Personality 47: 245-287.
Jacob M. Rose
Anna M. Rose
Montana State University
Carolyn Strand Norman
Virginia Commonwealth University
|Printer friendly Cite/link Email Feedback|
|Author:||Rose, Jacob M.; Rose, Anna M.; Norman, Carolyn Strand|
|Publication:||Journal of Information Systems|
|Date:||Mar 22, 2004|
|Previous Article:||Development of the Journal of Information Systems from the editors' perspectives.|
|Next Article:||Computational models as a knowledge management tool: a process model of the critical judgments made during audit planning.|