Impulsive Response Style Affects Computer-Administered Multiple-Choice Test Performance.

The current study was designed to test whether an impulsive response style affects performance on computer-administered multiple-choice tests and whether the consequences of such a response style are amenable to modification. The Matching Familiar Figures Test (MFFT) for adults was used to classify participants as either impulsive or reflective responders. Response time and accuracy measures were recorded as participants completed 2 multiple-choice tests. Half of the participants worked at their own pace and half were forced to delay their responses. Overall, the impulsive responders tended to answer questions more quickly and less accurately than the reflective responders. However, when the impulsive responders were forced to delay their response, accuracy improved to the level of their more reflective peers.

An individual's tendency to respond either impulsively or reflectively when presented with more than one response option, has been a subject of study for several decades. Kagan (1966) has used the term cognitive style to refer to the tendency to respond in either a reflective or impulsive manner. Cognitive style has been characterized as a cognitive trait that may, in part, account for individual differences in the quality of problem solving. Measurement of the trait has typically involved tasks requiring selection of a correct solution from among several response alternatives, with the latency and accuracy of responses recorded for each item. Most of the research on reflective-impulsive responding has focused on individuals who exhibit either a rapid-inaccurate (impulsive) response style or a slow-accurate (reflective) response style (e.g., Kagan, 1966; Egeland, 1974), although rapid-accurate and slow-inaccurate responders have also been identified (Kogan, 1983).

Yando and Kagan (1970) have suggested that reflective-impulsive cognitive styles are a stable trait and a better predictor of children's test-taking success than other indicators, such as intelligence. Messer and Brodzinsky (1981) found the trait to be meaningful and generally stable through early adolescence, and Kogan (1983) suggested that it might be applicable with adults as well, although research with adults has been limited. In one study, Phillips and Rabbitt (1995) determined that the tendency to respond impulsively is a stable characteristic apparent in certain adults' performance and that it exists independent of measured intelligence and extraversion.

Numerous studies have examined the impact of reflective-impulsive response styles on academic performance, especially with school-aged children. Those studies have indicated that reflective children, who take time to develop a solution hypothesis, commit fewer errors than impulsive children, who tend to focus on one solution without adequately considering the alternative solutions (e.g., Egeland, 1974). Kagan, Pearson, and Welch (1966) observed that impulsive children have a tendency to rapidly offer many answers without adequately considering the question. Kagan et. al claimed that an impulsive response style handicaps children in academic settings in which teachers tend to have a low tolerance for incorrect answers.

Students who have been classified as impulsive responders have also been found to have difficulty monitoring reading comprehension, particularly if they are asked to attend to detailed information. Walczyk and Hall (1989) found that, across reading levels, reflective children were better at detecting text inconsistencies and recalling text information than impulsive children. Consequently, Walczyk and Hall argued that reflective children experience more success with school-related tasks because they have developed an analytic style of reading that focuses on systematically assimilating and integrating detailed information.

Although cognitive response style has been characterized as a stable trait, researchers have attempted to alter response styles through a variety of strategies. Conducted mostly with children, these efforts have yielded mixed results. According to Egeland (1974), attempts to modify impulsive tempo have fallen into three categories: teaching the child simply to delay responding; modeling reflective techniques for the child; and directly teaching the child more effective analytic strategies. In his own study, Egeland (1974) found that instruction in scanning techniques and in simple response delay both produced longer latencies and fewer errors immediately after training. When Kagan, et. al (1966) trained children merely to delay their responses, without teaching them improved problem solving strategies, they found that response latencies lengthened; however, in contrast to Egeland's findings, error scores did not improve. Studies in which children have observed reflective models (e.g., Ridberg, Parke, & Hetherington, 1971) have resulted in longer response latencies but not improved error scores. Taken together, such findings suggest that impulsivity may be amenable to modification through training, but the mechanisms remain unclear for children and largely unknown for adults.

Despite evidence suggesting that an individual's cognitive style remains a stable characteristic into adulthood (e.g., Phillips & Rabbitt, 1995), virtually no research has been conducted to assess the impact of an impulsive response style among adults. The lack of empirical study in this area is unfortunate, particularly in light of the fact that adults, perhaps even more than children, are frequently faced with tasks involving a choice from among multiple response options, and their choices often have more far-reaching consequences. For example, multiple-choice tests, like the Scholastic Aptitude Test (SAT), are a significant factor in most college and university entrance requirements. If a person's cognitive response style has a significant impact on SAT performance, then it has a significant impact on where and, more importantly, whether a person goes to college. For impulsive students, whose ineffective strategies may cause them to perform poorly on multiple-choice tests, the results could be dire.

As society moves into a computer-based environment this trend necessarily becomes more evident in academic settings. Standardized tests such as the Scholastic Aptitude Test (SAT) and the Graduate Records Exam (GRE) have for several years now offered computer based versions where you answer questions at your own pace without the ability to go back and change answers. We are already seeing a trend towards similar types of testing in some academic classrooms. For example, in our department, testing in one of our Educational Psychology courses is entirely computer-based. Consequently, we need to have a good understanding of those factors, other than knowledge, that may affect test scores.

The current study was designed to determine whether an adult's cognitive response style (impulsive or reflective) affects performance on computer-administered multiple-choice tests. A pilot study using a split-test design demonstrated that forcing impulsive responders to delay their response would decrease the number of errors they made on a picture-matching task to the level of their more reflective counterparts. Consequently, a second goal of this study was to determine whether simply forcing impulsive responders to delay their response would improve their response accuracy.

Impulsive and reflective individuals were randomly assigned to either a self-paced or forced-delay condition and asked to complete a computer-administered, multiple-choice test. The delay was imposed by not allowing an answer to be entered until at least 25 seconds had elapsed. For exploratory purposes, two question formats (analogies and fill-in-the-blank) were used. Percentage of correct responses served as the primary outcome variable and response latencies were compared for the participants assigned to the self-paced condition. It was predicted that impulsive students, relative to their more reflective peers, would exhibit shorter response latencies and more errors under the self-paced condition, and that their performance would approximate that of their reflective peers when a delay was imposed.

Methods

Participants

Participants for this study were 48 general psychology students from Appalachian State University. All participants spoke English as their native language. Participants received credit toward a course requirement for their participation.

Design

Because of the inherent differences between the analogies and fill-in-the-blank question-types, this study utilized two, 2 x 2 between-groups factorial designs. For each question-type design, the relevant between-group variables were cognitive style (impulsive, reflective) and response type (self-paced, forced-delay). The dependent variables for each test were response latency and the percentage of correct responses.

Materials

The Matching Familiar Figures Test (MFFT) for adults (Kagan, 1966; Kagan, Rosman, Day, Albert, & Phillips, 1964) was used to identify participants as either reflective or impulsive. The MFFT is a visual discrimination task containing 12 items. On each trial a black line drawing of a familiar object is presented along with 6 similar pictures, only one of which is exactly like the original. The participant is required to select the picture that is identical to the original. For each item, latency to the first response is recorded using a stopwatch. If the first response is incorrect the participant is asked to try again and the total numbers of attempts until the correct match is identified is recorded.

The primary stimulus materials for this study were 32 multiple-choice questions chosen from the analogies and fill-in-the-blank sections of the Scholastic Aptitude Test (SAT). Sixteen items of each type were randomly chosen from three versions of the SAT administered in 1993. The questions were randomly arranged in two discrete tests, analogies and fill-in-the-blank. Instructions and a question example taken directly from the SAT preceded each test. The only difference between the instructions presented in this study and those from the SAT was the inclusion of instructions related to any response restrictions (self-paced or forced, 25-second delay).

The test items were presented using an IBM personal computer. Micro-Experimental Laboratories software was used to create the two computer programs for test presentation. In both program versions, each test began with test instructions, followed by 16 individually presented questions. Each question was presented with a question-stem and five response options labeled A, B, C, D, and E. In addition, each question was presented with an answer blank (ANSWER--) that signaled when the computer would accept a response. The computer recorded response latency and accuracy for each question. In the self-paced program the answer blank was presented at the same time as the question, thus allowing participants to answer questions at their own speed. In the forced-delay program the answer blank appeared 25 seconds after the presentation of the question and remained on the screen until the question was answered. The computer was programmed to emit a beep and not accept a response made before 25 seconds had passed. The order in which participants took the analogies and fill-in-the-blank tests was counterbalanced within each program version.

Procedures

Participants were told that we were interested in people's performance on multiple-choice tests. They were also told that they would be asked to take three different types of multiple-choice tests, one involving picture matching, one involving analogies, and one involving fill-in-the-blank items. Each participant was individually tested and randomly assigned to either the self-paced or forced-delay-test condition. For all participants, testing began with the MFFT followed by the computer-administered analogies and fill-in-the-blank tests. Results of the MFFT, which identified the participant as either reflective or impulsive, were not scored until all measures were completed. This was done to avoid the possibility of experimenter bias.

For administration of the analogies and fill-in-the-blank tests, participants were seated at a computer with the instructions for the first test displayed on the monitor. At the end of the instructions, participants were prompted to press the space bar to begin the test, at which point the first question appeared on the monitor. Responses to questions were made by pressing the appropriate key on the number pad of the computer keyboard. For this purpose the 4, 5, 6, 1, and 2 keys were relabeled A, B, C, D, and E, respectively. Once a response was entered for a question, the question would disappear leaving a blank screen for one-second before the next question was presented. Immediately after a participant answered the last question of the first test, the instructions for the second test appeared on the screen, and the same process was initiated. Testing sessions lasted approximately 45 minutes.

Results

Participants were divided into cognitive style categories based on a median split of the response latency and error scores. For this sample, the median first response latency was 32.0 seconds per item with a range of 6.0 to 94.0 seconds per item. The median number of errors was 0.92 per trial with a range of 0.17 to 2.0 errors per item. Participants who scored below the median response latency and above the median error rate (fast/inaccurate) were categorized as having an impulsive cognitive style. In contrast, participants who scored above the median response latency and below the median error rate (slow/accurate) were categorized as having a reflective cognitive style. From 60 initial participants, nine participants were excluded from the study because their performance fell either in the fast/accurate or slow/inaccurate range. An additional three participants were randomly excluded from groups to equalize cell means. Thus, the final analyses were conducted on data from 48 participants, 12 in each cell of the design.

The length of the delay (25 seconds) imposed on the forced-delay participants was derived from scores from individuals in a pilot study. To determine whether 25 seconds was an effective delay time for the current study, the response latencies for individuals assigned to the self-paced conditions were examined as a function of cognitive style and the test-item type. As can be seen in Table 1, the mean response latency was less than 25 seconds (19" and 25" for impulsive and reflective participants, respectively) for analogy items and greater than 25 seconds (27" and 35" for impulsive and reflective participants, respectively) for fill-in-the-blank items. Thus, 25 seconds was sufficient to delay impulsive responses on the analogy questions but not the fill-in-the-blank questions. This difference provides additional justification for the use of separate two-factor designs, rather than a single three-factor mixed design.

Table 1

Mean Proportion of Correct Responses (with Response Latency in seconds) for each Test-type broken down by Cognitive Style and Response Type Conditions.

                                        Cognitive Style
                                  Impulsive         Reflective

Test/Response Type
Analogies
  Self-Paced                     .461 (19.05)      .745 (24.87)
  Forced-Delay                   .708 (29.34)      .735 (29.24)

Fill in the Blank
  Self-Paced                     .609 (27.43)      .756 (35.47)
  Forced-Delay                   .634 (32.24)      .790 (31.53)

Note: response latency in the forced-delay conditions was calculated by adding 25 seconds (the delay interval) to the mean response latency for each participant.

Of critical interest was the proportion of correct answers for each participant in each condition and the mean response latency for each participant in the self-paced conditions. The mean response latencies for subjects in the forced-delay conditions were not included in the statistical analyses because 25 seconds of that response time was a preset condition of the study. The mean latency scores for the impulsive and reflective participants in the self-paced condition were analyzed using t-tests. The percentage correct scores were analyzed using 2 (cognitive style) x 2 (response type) between-group ANOVAs. The data for both the proportion of correct answers and mean response latencies are presented in Table 1. Please note that, although not included in statistical analyses, the response times for the forced-delay participants are included in Table 1 for consideration.

The t-test on the mean response latency data for the self-paced condition of the analogies questions confirmed that participants classified as impulsive responders tended to answer questions significantly faster then reflective responders (t (22) = 2.145, p [is less than] .05). The ANOVA conducted on the proportion of correct answers on the analogies test revealed that impulsive responders had significantly fewer correct answers than reflective responders (F (1,44) = 18.435, Mse = 0.344, p [is less than] .05) and that self-paced participants had significantly fewer correct answers than participants in the forced-delay condition (F (1,44) = 6.864, Mse = 0.128, p [is less than] .05). However, these main effects were qualified by the finding of a significant cognitive style by response type interaction (F (1, 44) = 8.383, Mse = 0.156, p [is less than] .05). Pairwise comparisons using Tukey's HSD revealed that the self-paced impulsive responders had significantly fewer correct answers then participants in all other conditions. No other differences were found.

The t-test on the mean response latency data for the self-paced condition of the fill-in-the-blank test confirmed that participants classified as impulsive responders tended to answer questions significantly faster then reflective responders (t (22) = 2.860, p [is less than] .05). The ANOVA conducted on the proportion of correct answers on the fill-in-the-blank test revealed only a significant main effect of cognitive style; impulsive responders had significantly fewer correct answers than reflective responders (F (1,44) = 5.734, Mse = 0.274, p [is less than] .05). However, it should be noted that the mean response latency for impulsive responders in the self-paced condition (27.43 seconds) was longer than the 25 seconds used in the forced-delay condition.

Discussion

The data from this study suggest that an adult's cognitive style, as measured using the MFFT, can impact performance on other types of multiple-choice tests. When allowed to work at their own speed, participants who were classified as impulsive (fast/inaccurate) tended to answer questions faster and less accurately than participants classified as reflective (slow/accurate). More importantly, the performance of the impulsive responders in the forced-delay condition of the analogies test clearly shows that the poor performance is not the result of a lack of knowledge but, rather, occurs because of a general failure to reflect long enough on each item. When the impulsive responders were forced to delay their responses, their performance improved to the level of the reflective responders. Unfortunately, this pattern was not noted in the fill-in-the-blank data. However, a cursory check of the fill-in-the-blank data reveals that the mean response latency of the self-paced impulsive responders was 27.4 seconds with a standard deviation of 6.9 seconds. Consequently, having 25 seconds pass before a response could be entered probably did not force any significant delay in participant responding on the fill-in-the-blank questions.

The most probable reason for the longer latency in responding to the fill-in-the-blank questions is that the fill-in-the-blank question stems simply take longer to read than the analogy question stems. However, it is of interest to note that within each cognitive style, the fill-in-the-blank self-paced and forced-delay groups performed at comparable levels (see Table 1). This finding makes it difficult to argue that simply making participants aware of the importance of delay is enough to change test-taking behavior. If simply suggesting a delay were sufficient to change performance, then we should have found improved performance for the impulsives in the forced-delay condition, regardless of the actual time taken to respond.

Because this appears to be one of the only studies that has compared adult test-taking performance based on cognitive response style, caution must be used in generalizing to other test-taking situations. In particular, the absence of test related consequences and the computer-presentation format might impact the generality of these results. In this study, poor test performance had no particular consequence for the participant. Thus, one alternative explanation for the current results could be that in this study the MFFT actually identified samples of individuals who either take their time in all testing situations (our reflectives) or vary their response strategy depending on the consequences related to test performance (our impulsives). From this perspective, it could be argued that adding a consequence to the testing situation might change the self-paced performance of the impulsive participants. However, even if the MFFT was measuring something other than a general tendency to respond impulsively or reflectively it does not dramatically change the importance of these results. The consequences of test performance vary across testing situations (i.e., a pop-quiz verses a final exam) and an individual's perception of consequences can be affected by a multitude of factors. Thus, it is unlikely that impulsive responders would adjust their test taking approach to perform as well as reflective responders in any particular testing situation.

The second issue is that care must be taken in generalizing these results across testing situations. The format used here is most similar to test situations like those presented by the SAT and the Graduate Record Exam (GRE). In particular, these are tests where the content can only be generally prepared for prior to testing. On most classroom tests of knowledge, participants are able to prepare for the specific content of the test and as a consequence, preparation would be a significant determining factor of performance. The current study cannot address how response style might interact with a variable amount of preparation. In our own classes, we have noted that many of the students who finish a test the quickest also do well on the test. However, these may also be the students who kick themselves because of "dumb" mistakes like "not reading carefully enough".

In spite of the cautionary notes presented above, there are several practical implications of these data. Regardless of whether they are considered measures of aptitude or achievement, tests like the SAT and GRE may underestimate the actual knowledge of some individuals taking the tests. We need to be aware that scores on these tests are not just a measure of knowledge, but are also affected by test taking style. In addition, the detrimental effects of an impulsive response style can also be expected to increase as the complexity of test scoring criteria increase. For example, the GRE specialty exams are designed to measure existing knowledge in particular topic domains and have much more complex scoring criteria than most other tests (-.25 for a wrong answer versus -0.0 for no answer). Such scoring criteria require not only that test takers use care in choosing a particular answer, but also requires that they use care and deliberation in deciding which questions to answer.

As with most research, the current results raise more questions than are answered. The data suggest that some individuals use a cognitive response strategy that may be detrimental to peak performance in certain test-taking situations. The current data also suggest that simply forcing impulsive responders to slow down may be sufficient to improve their performance. However, a great deal more work needs to be done to clarify test-taking factors that may interact with response style tendencies and the most efficient way of modifying such performance characteristics.

References

Cole, P. M., & Hartley, D. G. (1978). The effects of reinforcement and strategy training on impulsive responding. Child Development, 49, 381-384.

Edman, G., Schalling, D., and Levander, S.E. (1983). Impulsivity and speed and errors in a reaction time task: A contribution to the construct validity of the concept of impulsivity. Acta Psycholigica, 53, 1-8.

Egeland, B. (1974). Training impulsive children in the use of more efficient scanning techniques. Child Development, 45, 165-171.

Gerbing, D. W., Ahadi, S. A., & Patton, J. H. (1987). Multivariate Behavioral Research, 357-379.

Kagan, J. (1966). Reflection and impulsivity: The generality and dynamics of conceptual tempo. Journal of Abnormal Psychology, 71, 17-24.

Kagan, J., Pearson, L. & Welch, L. (1966). Modifiability of an impulsive tempo. Journal of Educational Psychology, 57, 359-365.

Kagan, J., Rosman, B. L., Day, D., Albert, J., & Phillips, W. (1964). Information processing in the child: Significance of analytic and reflective attitudes. Psychological Monographs, 78, (1, Whole No. 578).

Kogan, N. (1983). Stylistic variation in childhood and adolescence: Creativity, metaphor, and cognitive styles. In P. H. Mussen (Series Ed.), J. H. Flavell, & E. M. Markman (Vol. Eds.), Handbook of Child Psychology: Vol. 3. Cognitive Development (4th ed., pp. 630-705). New York: Wiley.

Messer, S. B., & Brodzinky, D.M. (1981). Three-year stability of reflection-impulsivity in young adolescents. Developmental Psychology, 17, 848-850.

Phillips, L. H., & Rabbitt, P. M. A. (1995). Impulsivity and speed-accuracy strategies in intelligence test performance. Intelligence, 21, 13-29.

Ridberg, E., Parke, R. D., & Hetherington, E. M. (1971). Modification of impulsive and reflective cognitive styles through observation of film-mediated models. Developmental Psychology, 5, 369-377.

Walczyk, J. J., & Hall, V. C. (1989). Is the failure to monitor comprehension an instance of cognitive impulsivity? Journal of Educational Psychology, 81, 294-298.

Yando, R. M., & Kagan, J. (1970). The effect of task complexity on reflection-impulsivity. Cognitive Psychology, 1, 192-200.

Douglas A. Waring, Department of Psychology, Cheryl Beck Farthing, Department of Psychology, Pamela Kidder-Ashley, Department of Psychology. We gratefully acknowledge Paul Fox for his helpful manuscript reviews.

Correspondence concerning this article should be addressed to Douglas A. Waring, Department of Psychology, Appalachian State University, Boone, North Carolina 28608. E-mail: [e-mail].