Extending the forer test beyond face Validity: an experiential approach to Teaching social science methodology.INTRODUCTION
Many psychology students are familiar with the 'Forer Test'. In the test, a personality evaluation is administered and every participant gets the same personality profile as a result (Forer, 1949; Carroll, 2005). Most participants consider the profile to be fairly accurate, thinking it is a personalized profile based on the test. This face validity is due to the generally positive nature of the profile (Leung, Su & Morris, 2001), the accepted authority of the evaluator and the belief that the analysis was unique to them (Hannay, Arisholm, Engvik & Sj0berg, 2010; Dickson & Kelly, 1985). The test is used to show the problems with relying only on face validity to judge a measurement instrument or evaluation. In this exercise we go beyond the traditional 'Forer Test' lesson. We administer the 'personality test' online, and analyze the actual results from test using legitimate, established social science methods to see if indeed the test does reveal patterns that indicate latent constructs. In this way, an interesting but limited demonstration of a psychological phenomenon is used as a starting point for illustrating more sophisticated concepts. In addition to describing the basic exercise, we illustrate how it can be used in different but related behavioral disciplines within business education. Specifically we describe the basic exercise, and illustrate how it was successfully used in undergraduate and graduate management and marketing courses.
The paper begins with a description of the Forer test, its history and application. This is followed by a brief explanation of experiential exercises in general. A description of our exercise is presented next. After the basic exercise is presented, a specific implementation of the exercise will be examined in detail. The materials and procedures will be presented first, followed by the analysis methods and results. The presentation of the results in class and the ensuing discussion are explored next. The paper concludes with proposals of how to modify the exercise for different contexts.
BACKGROUND--THE FORER TEST
In 1949 Bertram R. Forer introduced "the fallacy of personal validation". Specifically he found that people had a tendency to accept vague, general descriptions of personality as very true for them even though the descriptions could apply to almost everyone. The personality evaluation given by Forer was as follows:
"You have a need for other people to like and admire you, and yet you tend to be critical of yourself. While you have some personality weaknesses you are generally able to compensate for them. You have considerable unused capacity that you have not turned to your advantage. Disciplined and self-controlled on the outside, you tend to be worrisome and insecure on the inside. At times you have serious doubts as to whether you have made the right decision or done the right thing. You prefer a certain amount of change and variety and become dissatisfied when hemmed in by restrictions and limitations. You also pride yourself as an independent thinker; and do not accept others' statements without satisfactory proof. But you have found it unwise to be too frank in revealing yourself to others. At times you are extroverted, affable, and sociable, while at other times you are introverted, wary, and reserved. Some of your aspirations tend to be rather unrealistic" (Forer, 1949).
Participants in Forer's test, performed in 1948, rated this passage as 4.26 on a scale of 0 to 5 with 4 indicating it was a 'good' assessment and 5 as 'excellent' (Forer, 1949; Hannay, Arisholm, Engvik & Sj0berg, 2010). The test has been repeated hundreds of times in the succeeding decades with the average remaining about 4.2 (Carroll, 2005).
This phenomenon of individuals tendency to accept 'bogus' feedback as accurate is also known as the 'Barnum Effect' (Meehl, 1956; MacDonald & Standing, 2002). The Barnum effect was initially used in classrooms to illustrate gullibility and deception; however it later was used to teach ethics (Beins, 1993). Those teaching ethics use the Barnum Effect to aid in discussions of the ethics of deception, the ethics of deception in research, the feelings of those who have been lied to (Beins, 1993). Recently Boyce & Geller (2002) found no studies that used the Barnum effect to 'promote a healthy skepticism of pseudoscience' or to teach research methods, therefore they used it to teach psychology research methods, ways of displaying and interpreting data, and to "highlight the pitfalls of pseudoscience" (Boyce & Geller, 2002).
Those who have conducted research into the phenomenon while using it class found that under certain circumstances people accept feedback rationally and not gullibly (Michels & Layne, 1980). For example, when presented with each, participants show the ability to discriminate between accurate, trivial and inaccurate feedback (Wyman & Vyse, 2008; Harris & Greene, 1984). Another interesting finding is that the Barnum effect is more prevalent in positive statements and evaluations than in negative leading researchers to conclude that the Barnum effect is somewhat cancelled by a self-serving bias (Leung, Su & Morris, 2001;
MacDonald & Standing, 2002).
THE EXPANDED FORER TEST EXERCISE
In this exercise we follow the lead of others who see the potential use of the Forer Test for teaching research methods (Boyce & Geller, 2002). We go further than using the Forer test to illustrate and start conversations about gullibility, ethics of deception and test validity. Furthermore, we use the same exercise to show how proper investigations can be done. The exercise was run in several classes in undergraduate and graduate organizational behavior, marketing and management courses. The exercise can easily be adapted for other situations.
Exercise Materials and Procedure.
Students were asked to complete a survey administered online in a course management shell. The survey consisted of what appeared to be a simple personality test. The items are shown in Table 1. They were asked to indicate which they preferred (or indicate 'no preference) for each of 15 sets of items. After being asked two demographic questions (age, gender), students were next shown a 'personality evaluation', presumably based on the answers given. They were then asked how accurate the description was from '1-very poor' to '5-excellent'.
Later in the week, in class, the 'truth' was revealed and the Forer effect was discussed. The instructors had already done a correlation and factor analysis on the items, and used these results to illustrate basic concepts of validity and latent constructs.
ANALYSIS OF RESULTS.
The data from all of the participating classes were combined and some simple analysis was conducted. The means and standard deviations of the variables were computed as well as statistics for kurtosis and skewness. Next significant correlations of the remaining items were identified. Table 2 shows the items that were correlated at .01 or better. An exploratory factor analysis was conducted to find interrelated patterns of relationships and identify latent constructs. Six groups were identified by the factor analysis (principle component, varimax rotation), of which only two had more than two items. Those two factors (Table 3) had alpha reliabilities that were <.5, which is unacceptable.
The procedure was explained to the classes. Students were asked to propose relationships between the items. The relationships were revealed and discussed with an emphasis on comparing proposed relationships with the actual relationships found. Next the two revealed patterns were presented. The discussion about the groupings centered on explanations of the groups and uses of this information. Next the marketing students were asked to identify ways that this information can be used to design products and marketing campaigns. The management students were asked how this information can be of value to managers. Both groups were asked to design a follow-up study related to their proposed uses of the information.
DISCUSSION AND CONCLUSIONS
The Forer test (Forer, 1949), or Barnum effect (Meehl, 1956) is often used to illustrate gullibility and ethics (Beins, 1993). Rarely is it used to explore research methods (Boyce & Geller, 2002). In this paper is description of how the Forer/Barnum phenomenon was presented in management and marketing course and used to introduce legitimate research methods.
Although the exercise was a success, we have identified several areas for improvement. First, the 'personality survey' can include a greater number of items and the items themselves can be more carefully chosen. The test given was designed to be nonsense; however that is not necessary at all and may have actually limited our analysis. The items themselves might be chosen to reflect specific concerns of disciplinary courses, such as consumer product or marketing related issues, or work related concepts to allow the final results to be applied to product design, marketing campaigns or staffing, job design, training and motivation. Traditional 'personality' items could be used if the exercise was to be used across classes in different disciplines and at different levels. A problem with the original items was a great degree of kurtosis and skewness. Scale items with greater variation would lead to a better analysis and still allow for discussion of skewness and kurtosis. Carefully chosen items would also allow for factor analysis to develop valid reliable factors. Factor analysis would be especially appropriate in a graduate level class. After the data is entered into a spreadsheet, the analysis can be done by students as an assignment, either with specific directions or leaving the students to find their own ways to use it. Discussions of validity at the graduate level are particularly important (Mundfrom, Young, Shaw, Thomas, & Moore, 2003) and better analysis results would facilitate such a discussion.
Once an instrument is developed future exercises can also be used for research purposes. It would be relatively easy to design experiments that extend the existing research. Graduate students and faculty could explore the dynamics of the Forer/Barnum effect on discipline specific tests (consumer related, job related, etc.) to see under what conditions people are rational or gullible (Michels & Layne, 1980), have the ability to discriminate between accurate, trivial and inaccurate feedback (Wyman & Vyse, 2008; Harris & Greene, 1984) and have the effect mediated by a self-serving bias (Leung, Su & Morris, 1980; MacDonald & Standing, 2002).
Beins, B.C. 1993. Using the Barnum Effect to Teach About Ethics and Deception in Research. Teaching of Psychology, 20(1): 33-35.
Boyce, T.E. & Geller, E.S. 2002. Using the Barnum Effect to Teach Psychology Research Methods. Teaching of Psychology, 29(4): 316-318.
Carroll, R.T. 2005. The Forer Effect. The Skeptic's Dictionary. Retrieved December 17, 2005 from HYPERLINK "http://skepdic.com/forer.html" http://skepdic.com/forer.html.
Dickson, D.H. & Kelly, I.W. 1985. The 'Barnum Effect' in Personality Assessment: A Review of the Literature. Psychological Reports, 57, 367-382.
Forer, B.R. 1949. The Fallacy of Personal Validation: A Classroom Demonstration of Gullibility. Journal of Abnormal and Social Psychology, 44: 118-123.
Hannay, J., Arisholm, E., Engvik, H., & Sjoberg, D. 2010. Effects of personality on pair programming. IEEE Transactions on Software Engineering, 36(1), 61.
Hanson, W. E., & Claiborn, C. D. 2006. Effects of test interpretation style and favorability in the counseling process. Journal of Counseling and Development : JCD, 84(3), 349.
Harris, M.E. & Greene, R.L. 1984. Student's Perception of Actual, Trivial and Inaccurate Personality Feedback. Journal of Personality Assessment. 48(2): 179-184.
Leung, K., Su, S., & Morris, M. W. 2001. When is criticism not constructive? the roles of fairness perceptions and dispositional attributions in employee acceptance of critical supervisory feedback. Human Relations, 54(9), 1155.
MacDonald, D.J. & Standing, L.G. 2002. Does Self-Serving Bias Cancel the Barnum Effect? Social Behavior and Personality, 30(6): 625-630.
Meehl, P. E. 1956. Wanted-A good cookbook. American Psychologist. 11. 262-272.
Michels, P.J. & Layne, C. 1980. Inventory Responding Models People's Acceptance of Feedback "Derived" From Tests and From Interviews. Journal of Personality Assessment, 48(2): 179-184.
Mundfrom, D. J., Young, S., Shaw, D. G., Thomas, A., & Moore, A. D. 2003. A common knowledge base in introductory graduate research courses. Research in the Schools, 10(2), pp. 71-78.
Wyman, A., & Vyse, S. 2008. Science versus the stars: A double-blind test of the validity of the NEO five-factor inventory and computer-generated astrological natal charts. The Journal of General Psychology, 135(3), 287.
Stephen C. Betts, William Paterson University
Table 1: Items on the 'Personality Test' Anchor 1 Anchor 2 Red Blue Cats Dogs Cotton Satin Meat Vegetables Night Day City Country Travel Staying Home Activity Relaxation Sun Clouds News Sports Radio TV Science Art Rock Classical Solid Stripes Drama Comedy Table 2: Correlation Results Given these pairs: This is positively related to these: Activity Relaxation Activity Cotton, Travel, Sun, Comedy Sun Clouds Sun Rock, Activity, Day Travel Stay Home Travel Day, Dog, Activity Rock Classical Rock Meat, Sun Comedy Drama Comedy Activity, Sports Science Art Science Art Dogs Cats Dogs Travel Meat Vegetables Meat Rock Cotton Satin Cotton Activity Sports News Sports Comedy Day Night Day Sun Table 3: Factor Analysis Results Analysis reveals two groups: Group 1 Sun Clouds Science Art Day Night Rock Classical Group 2 News Sports Drama Comedy Vegetables Meat