Printer Friendly

Effect of Dialectical Bootstrapping on Quantitative Estimation.


Estimation is assigning some value to attributes of an object or an event under the situation of incomplete information. Estimation plays an important role in human problem solving process, especially for real world problems. A typical real world problem requires us to make decision under the situation that we do not have enough information to calculate the outcome of each option that we choose as our course of action. Under the situation, it is unavoidable that we make a guess, or estimate on some attributes of some objects or events to calculate the outcome of each option.

As estimation is frequently a part of decision making process, the topic is intensively investigated by the research community of judgment and decision making under the topic of judgment. Researchers investigated the mechanism underlying human estimation process, and methods to improve human estimation, and they identified several cognitive biases that hamper sound and valid human estimation. Most methods for improving human estimation are based on debiasing human cognitive biases (Soll, Milkman, Payne, 2014).

A consistent finding in human estimation is that the average of multiple people's estimation is, in most cases, better than a single person's estimation, which was coined as the 'wisdom of audiences' or 'wisdom of crowd'. It is claimed that the mechanism underlying wisdom of crowd is that the errors made by individuals due to different knowledge bases partial out when they are aggregated. The logic is quite similar to classical measurement theory. The origin of research on wisdom of crown dates back to Galton(1907) who reported that the mean of 787 laymen estimation on the weight of an ox is very close to true weight of it. Later studies (Hourihan & Benjamin, 2010) also have supported the advantage of the average of multiple judgers' estimation over a single person's estimation. Based on the research results, we may be able to rely on others when we want to make better estimation. But a limit of the approach is that we cannot employ the wisdom of audience when others' estimation is not available.

Meanwhile, some researchers (Ariely, et al., 2000; Buehler, Griffin, Peetz, 2010; Herzog & Hertwig, 2009; Herzog & Hertwig, 2013; Herzog & Hertwig, 2014a, 2014b; Rauhut & Lorenz, 2011; Vul & Pashler; 2008; Winkler & Clemen, 2004) claimed or showed research results that the average of multiple estimation by an individual can be better than a single estimation by the individual as long as each estimation has independency to others. As independency of multiple estimations within individual is a critical factor to make multiple estimations better estimation than a single estimation, some techniques to increase the independency among multiple estimations within individual were suggested.

The first approach is to have time interval between each estimation by the individual. The longer the time interval between estimations, the higher the chance that estimator may make his estimation based on different knowledge base and dependency between the estimation may decrease. Herzog and Hertwig (2009) suggested another strategy that they named as 'dialectical bootstrapping'. The dialectical bootstrapping exploits multiple judgments from different knowledge sources or different perspective from a single person by asking to view the problem from different perspective when an estimator makes the second estimation. More specifically, they employed 'think the opposite' technique to increase the independency two estimation within an individual. Results of several studies (Herzog & Hertwig, 2009; 2013, Vul & Pashler, 2008) support dialectical bootstrapping technique for improving estimation.

But there is a logical pitfall in dialogical bootstrapping technique threatening its usefulness when only two estimations by an individual is employed for better estimation as Herzog & Hertwig's study (2009). When it is applied to estimation of quantitative properties of some object, and when estimators makes only two estimations, the second estimation should move toward the true value, not away from true value. But this is rather an optimistic expectation because the estimator cannot know to which direction from her first estimate the true value locates. Her second estimate has 50% of approaching toward the true value from her first estimate and the chance of running away from true value from her first estimate is also 50%. In this sense, it is not guaranteed that the average of the two estimates is better than the first estimate.

Kim (2015) failed to find the advantage of dialectical bootstrapping in his two experiments whose procedure quite was similar to Herzog & Hertwig's procedure. But he compared qualitative data gained from participants who showed much improvement by the second estimate and those gained from participants whose estimation got worsened by the second estimate. The main difference between the two groups was that the first group wrote down some reasons for his or her estimates while the second group did not wrote down any reason or wrote they relied on some instincts. Based on the results, Kim suggested that the reason why his study did not show the advantage of dialectical bootstrapping might be the procedure employed in his two experiments, that is the procedure in his study did not stimulate participants enough to make serious efforts to reason about their estimation. He suggested that the original idea of bootstrapping might be still valid as long as any procedure implements the idea that estimators make two estimations based on two different knowledge bases. But his study did not examine the whole data of participants in his study.

Because prior studies showed inconsistent results on the effect of dialectical bootstrapping on the improvement of estimation, and an explanation for the effect of the procedure and an explanation against it co-exists, it is necessary to examine effect of bootstrapping procedure on the improvement of quantitative estimation again. Thus, this study will replicate the bootstrapping procedure to examine its effect of quantitative estimation.

The specific research objective of this study is to examine if bootstrapping procedure improve the quantitative estimation as long as the estimators make some explicit reason for their estimation in their two estimations.



Sixty three university students from a medium sized university in South Korea participated in this study. Students aged from 18 to 25 years. Those students' high school average grade approximate ranged from top 13% to 50% among their peer group. All the participants except one were in their 2nd year in the university. Five participants were dropped in the final analysis because they did not complete the whole experimental procedure.

Procedure and material

The experiment was carried out a classroom where participants take a regular class. All the participants registered one of researcher's classes. The experiment began informing participants of the general research goal, experimental procedure, participants'right. They were also informed that 6 participants selected by lottery from estimator scoring as top 20 will be given a prize worth about 10 dollars. The prize was planned to stimulate participants' active participation in the estimation process. Five questions were presented to the participants for estimation. Two questions were about history, two questions were about physical estimation that participants actually could see while they estimate, and the rest one was a physical property that they cannot see while they estimate. All the questions (refer to Appendix) have only one true value and participants were requested to answer as a single integer. The questions were borrowed from Kim (2015), but one question was changed into rather an easier one. The original questions were developed by the researcher and a graduate student.

Questions printed in a A4 paper sheet were presented to the participants, the paper also has spaces for participants to write their knowledge base for each estimation. All the instructions were written in the paper and they were presented on the screen displayed in front of the classroom. As same as Herzog and Hertwig's (2009) procedure, after participant's first estimation for the all 5 questions, they were asked to make a second estimation for the same questions but with an assumption that their first estimations were incorrect, and think some other knowledge base for the estimation, which, if possible, has some base that is opposite to the first base. It was requested that participant write their knowledge bases for the estimation on the paper.

Ten minutes were spent for introducing the experiment, 15 minutes were spent for making the first estimations and writing their knowledge bases, and another 15 minutes were spent for making the second estimations and writing their bases.


The dependent variable of this study, the accuracy of estimation was measured as the absolute value of estimate's deviation(errors) from the true value. The accuracy of estimation from each participant's first estimation and second estimation was directly obtained by subtracting true value from their estimates. But the average accuracy of two estimations was calculated by dividing the added deviation value into 2. For example, if a participant's first estimation for the question "When was King Sejong born?"(true value is 1397) was 1350 and his second estimation was 1420 the average deviation is calculated as | ((1350-1397) + (1420-1397))/2|, which is 35.

As it is shown in <table 1>, the mean error of combined two estimation (M=259.63) decreased a little compared to the mean error of first estimation (M=263.14). But the result of paired samples t-test between the means of absolute error of first estimation and that of combined estimation did show statistically significant difference (t=.357, df=57, p. = .722). It means that dialectical bootstrapping procedure did not improve estimator's estimation compared to the estimator's first estimation.

The result of this study is not consistent with prior studies (Herzog & Hertwig, 2009; Hourihan & Benjamin, 2010; Muller-Trede, 2011; Rauhut & Lorenz, 2011; Stroop, 1932; Vul & Pashler, 2008; Winkler & Clemen, 2004) that reported the effect of audience within. In particular, this study employed quite similar procedure to that of Herzog and Hertwig(2009). There are some differences between the procedure in their study and that of this study such as the study used computer while this study paper for participants to respond the treatment, and the study used a date estimation task on historical events and this study used estimating date estimation, physical property estimation. But considering that there are studies which reported the effect of diverse domains (Fraundorf & Benjamin, 2014; Hourihan & Benjamin, 2010; Muller-Trede, 2011; Vul & Pashler, 2008), it is not very reasonable that domain difference mattered.

This study also employed rather an easy estimation task for the participants by selecting only questions that participants were rather familiar with so that participants actually could bring some of their knowledge bases while they estimate. To check if participants brought some knowledge base when they estimate, I reviewed their written bases for the estimation. Six of 58 participants did not write their knowledge bases for the estimation, which I assume could be ignored for the interpretation of this study. I further reviewed if participants employed different knowledge bases for their first estimation, and for their second estimation. Out of total cases of 290 cases of estimation (58 participants X 5 questions ), half of the cases(145) used same knowledge base for the first and second estimation. When they used same knowledge base, they just changed their second estimate regretting that they made incorrect or exaggerated first estimation. The results tells that all participants did not used the different knowledge bases for the two estimation, which the dialectic bootstrapping procedure aimed at.

Again, the result of this study also did not give a clear answer to the question if dialectic bootstrapping is a useful technique for the better estimation because it could be claimed that the procedure in this study failed to fully implement dialectical bootstrapping procedure. But it seems that just asking participants to making a second estimation thinking different knowledge base, hopefully that will bring opposite result, does not guarantee to materialize the dialectical bootstrapping procedure and improve the quality of our estimation.

Further studies that employ more thorough analysis of participants' estimation process will bring clearer answer to research question of this study.


This study examined if dialectical bootstrapping procedure improves quality of our quantitative estimation. Considering that only half of participants' second estimation cases were based on different knowledge base from that of participants' first estimation in this study, it is too early to say that dialectical bootstrapping does not improve our quantitative estimation. But just telling participants to make the second estimation with different knowledge base that may bring opposite result does not guarantee to materialize the idea of dialectical bootstrapping. Further studies with more refined research and replication studies will bring clearer answer for this research question.


Ariely, D., Tung W., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H., Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6(2), 130-147.

Buehler, R., Griffin, D., & Peetz, J. (2010). The planning fallacy: Cognitive, motivational, and social origins. In M. P. Zanna & J. M. Olson (Eds.), Advances in Experimental Social Psychology (Volume 43, pp. 1-62). San Diego: Academic Press.

Fraundorf, S. H., & Benjamin, A. S. (2014). Knowing the crowd within: Metacognitive limits on combining multiple judgments. Journal of memory and language, 71(1), 17-38.

Galton, F. (1907). Vox populi (the wisdom of crowds). Nature, 75, 450-451.

Herzog, S. M., & Hertwig, R. (2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20(2), 231-237.

Herzog, S. M., & Hertwig, R. (2013). The Crowd within and the benefits of dialectical bootstrapping A Reply to White and Antonakis (2013). Psychological science, 24(1), 117-119.

Herzog, S. M., & Hertwig, R. (2014a). Harnessing the wisdom of the inner crowd. Trends in cognitive sciences, 18(10), 504-506.

Herzog, S. M., & Hertwig, R. (2014b). Think twice and then: Combining or choosing in dialectical bootstrapping?. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(1), 218.

Hourihan, K. L., & Benjamin, A. S. (2010). Smaller is better (when sampling from the crowd within): Low memory-span individuals benefit more from multiple opportunities for estimation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(4), 1068-1074.

Kim, K. (2015). Effect of Dialectical Bootstrapping on Quantitative Estimation. The Korean Journal of Thinking Development. 11(4), 29-51.

Muller-Trede, J. (2011). Repeated judgment sampling: Boundaries. Judgment and Decision Making, 6(4), 283-294.

Rauhut H., & Lorenz, J. (2011). The wisdom of crowds in one mind: How individuals can simulate the knowledge of diverse societies to reach better decisions. Journal of mathematical Psychology, 55(2), 191-197.

Soll, J.B., Milkman, K. L. & Payne, J.W. (2014) A User's Guide to Debiasing. Chapter from Wiley-Blackwell Handbook of Judgment and Decision Making. (G. Keren and G. Wu, Eds).

Stroop, J. R. (1932). Is the judgment of the group better than that of the average member of the group? Journal of experimental Psychology, 15(5), 550-562.

Vul, E., & Pashler, H. (2008). Measuring the crowd within probabilistic representations within individuals. Psychological Science, 19(7), 645-647.

Winkler, R. L., & Clemen, R. T. (2004). Multiple experts vs. multiple methods: Combining correlation assessments. Decision Analysis, 1(3), 167-176.



(1) What year was the King Sejong bom?

(2) What year did President Kim Youngsam inaugurate?

(3) What is the height of the whiteboard attached on the front wall of this classroom?

(4) How much does the researcher weigh?

(5) How tall is the 63 building?

Kwangsoo Kim

Andong National University, Korea

Correspondence concerning this article should be addressed to Kwangsoo Kim, Andong National University, 1375, Gyeongdong-ro, Andong-si, Gyeongsangbuk-do, KOREA. E-mail:
Table 1
Descriptive Statistics of Absolute Errors of Estimation

                     N   M       SD

First estimation     58  263.14  198.24
Second estimation    58  278.53  238.91
Estimation combined  58  259.63  210.37
COPYRIGHT 2016 Korean Association for Thinking Development
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Kim, Kwangsoo
Publication:The International Journal of Creativity and Problem Solving
Date:Apr 1, 2016
Previous Article:Effects of a "Learn to Think" Intervention Program on Chinese Primary School Students' Learning Strategies.
Next Article:Creative Use of Visual Arts within Conflict Resolution.

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters