Think tank wars. (Letters To The Editor).

To the Editor:

We are somewhat underwhelmed by an "analysis" ("Think Tanks: Who's Hot and Who's Not," Fall 2002) by an Institute for International Economics scholar, Adam Posen, that purports to show IIE as the second most quoted economic think tank. At a minimum, the methodology is sloppy, with no explanation of its rationale. Why Foreign Affairs? Isn't including the International Herald Tribune with the New York Times and Washington Post double counting? Why Business Week but not Forbes or Fortune? Further, it's inexplicable that Cato's most quoted economist, Steve Moore, was excluded from the survey.

The International Economy is a fine magazine, so I'm surprised that you didn't consider the fact that the IIE came in 22nd out of 25 think tanks surveyed this year by FAIR in a far more extensive study. I believe it is odd that Brookings, Cato, and the American Enterprise Institute would be in the top four (along with the Heritage Foundation) in the FAIR study and that the only significant change was for IIE to move from 22nd to 2nd in its study. I would suggest you'd be on safer ground to limit these kinds of surveys to those undertaken by independent organizations.

EDWARD H. CRANE
President
Cato Institute
Washington, D.C.

Adam Posen responds:

We are glad to have the opportunity to respond to Mr. Crane's critical but unfounded assertions about the validity of our study of media citations of think tanks. This allows us both to clarify the methods we used in our research, and to correct those data errors and omissions which were brought to our attention since the data underlying our results were publicly posted (on the IIE website).

First, comparing the FAIR study of total media mentions of think tanks to our analysis misses the basic point--ours was a study of citations of economics think tanks and scholars of economic issues. Simply putting the word "Heritage" or "Brookings" without any specific scholars or subjects into a search engine, as the FAIR study does, lumps together every citation on every topic and does not serve the purpose that either we or The International Economy had in mind. We do not purport to study overall think tank mentions and explicitly acknowledge our limited focus in the article.

Second, any list of publications will in some sense be arbitrary. Why is it less arbitrary to take the Lexis/Nexis database's default list of "Major Newspapers and Transcripts" (which is what the FAIR study does) than to design a representative sample? Our sample of eleven publications consciously draws from a range of political views and selects those with wide readership and real economics coverage. Meanwhile, any search limited to Lexis/Nexis systematically undercounts certain publications, and double-counts others, which is why, as we explain in our methodology statement (also posted on the IIE website), we used both Lexis/Nexis and Dow Jones services for our searches. This is not a criticism of the FAIR methodology but underscores that search design is a choice along a tradeoff, wherein some conscious design may well be justified depending upon the study's goal.

Incidentally, Mr. Crane's misconception about the International Herald Tribune is another example of why we put such careful effort into our study design. Including citations in the International Herald Tribune and the Washington Post or the New York Times is not a double-count because they reach different subscriber bases and because there is no automatic reprint of Washington Post or New York Times articles in the International Herald Tribune. There are also economics articles that appear only in the International Herald Tribune written by their own staff.

Third, to test Mr. Crane's contention that our study misses a great deal by not including Forbes or Fortune, we performed an experiment. We searched for any mentions of the "Cato Institute," unconstrained by reference to specific economists, for the five-year period of our study. We found 48 citations. Were these added to Cato's total, and no credit given to any other think tank for any cites in Forbes and Fortune, Cato's total of 382 cites would still rank well behind AEI's third-place total of 662. This robustness of rankings suggests that we did use a representative and unbiased sample of publications in our study.

It is interesting to note that the main difference between our study's results and those of the FAIR study appears to be that IIE displaces Cato in the top ranks. That would seem to suggest that there is some benefit to Cato from the design of the FAIR study's sample of publications and/or its inclusion of non-economic issues that does not affect the relative positions of the other three institutions Mr. Crane mentions (Brookings, AEI, and Heritage), whose rank is essentially unchanged between the two studies when the focus narrows.

Fourth, the reason IIE is not in the top ranks in the FAIR study is self-evident--unlike Cato or Brookings or Heritage, we do not cover social, foreign policy, or other non-economic issues so of course we do worse in a general count of every press appearance by every think tank (including think tanks whose sole focus excludes economics), IIE's high ranking in our study is not the result of bias, but rather of the openly declared focus of the study.

Finally, I would like to explain the "inexplicable" exclusion of Stephen Moore from our study (which I had already explained directly to Mr. Moore in three long e-mails, the first of which pre-dated by three weeks Mr. Crane's letter to you). Our study included people with an obvious economic subject focus, who are resident at or have a primary affiliation at an included think tank, and who had noticeable economic research output listed on the relevant think tank's Web site (in a given period). The idea was to make inclusion a matter of rules insofar as possible, and to leave as little to judgment as possible.

On the first criterion, Mr. Moore is quoted quite extensively about fiscal issues. Upon search, we find that he has a total of 123 cites in our sample publications and period (when searching for his name and Cato affiliation). This would have made him the sixth most cited individual in our study (though adding his total to that of Cato would still not change that institute's rank overall). As I informed Mr. Moore in our earlier correspondence, we were going to note in our already planned update (for this issue of The International Economy) where he would have placed had he been included as a think tank economist.

As I also noted to him, however, Mr. Moore fails on the remaining criteria for inclusion in our study:

* Moore's primary affiliation since at least 1998 is as President of the Club for Growth, a political advocacy and fundraising organization, and that is his first designation both on that organization's and on Cato's Web site;

* At the time of our research, he was listed on Cato's Web site in the past tense, as "former head of fiscal studies";

* His own e-mail address was listed on the Cato Web site as "[e-mail]," not at @cato.org, seemingly confirming his non-resident status;

* He was listed on the Cato Web site with their "Senior Fellows" at Cato, the place where they list such notables as James Buchanan and PJ O'Rourke, hardly full-time or primary Cato affiliates, and he was not listed on the Web site with William Niskanen and the other resident working economists;

* There was a single research publication since July 1998 for Cato by Mr. Moore linked to his Cato Web site bio ("A Fiscal Report Card on America's Governors: 2002") and the other publications listed either pre-dated our period of study or were reprints of op-eds published elsewhere.

All this led to our assessment from the Cato Web site that Mr. Moore's primary affiliation was not as a research economist with Cato. On the last point, we have since discovered that Cato published his 2000 co-authored book, It's Getting Better All the Time. However, the Cato Web site itself does not feature this fact on Mr. Moore's page or anyplace else obvious (we only discovered this by going to the Club for Growth Web site, clicking on their link to Amazon.com, and then scrolling down that page to find the publisher). While we regret this oversight, it is understandable that such obscured information would fail to be uncovered in our research encompassing a couple of hundred economists. In any event, it does not remove the other aspects that would disquality Mr. Moore from being included.

There are a few individual economic scholars at think tanks whom we did omit or undercount by mistake, and we would like to take this opportunity to apologize to them and to publicize the relevant corrections. As we hoped, by publishing our raw data on the IIE Web site, people were able to check our work and bring the inevitable errors found in such a large dataset to our attention; we also had the opportunity to run some rechecks on own work. The updated and corrected data are now posted at www.iie.com/study/study.htm [and two summary tables with the updated rankings are given here]. While these corrections have not necessitated any alteration in the study's basic findings, particularly with respect to overall think tank performance, which was our primary concern, we wish through disclosure to be fair to those whose rankings were inadvertently underestimated:

* Nancy Birdsall, president of the new Center for Global Development, and formerly of the Carnegie Endowment for International Peace, was inadvertently omitted. She ranks 35th overall with 35 cites in our sample. Some of her colleagues at the CGD should be expected to place well in future studies, once they have been in operation long enough to be counted (especially if they keep up the pace of citation they have shown in the Center's first year of existence).

* The citation totals of three noted scholars, Robert Barro and Milton Friedman of the Hoover Institution, and Eugene Steuerle of the Urban Institute, were undercounted due to data entry errors. Of course, we are particularly embarrassed to have mistyped the scores of the Nobel Laureate, Professor Friedman, of all people. In the corrected standings, Barro is ranked #20 (57 cites), Friedman is #21 (56), and Steuerle is tied for #24 (49). This moves Hoover to fifth place in total cites, to 313 (behind Cato's 334, ahead of EPI's 269). It does not affect the Urban Institute's ranking.

* James Glassman of AEI was undercounted when we tried to check for double-counting of his regular column in the Washington Post. His actual cites total is 229, rather than 187, and his rank is tied for #3 rather than #4.

* William Gale of Brookings was misrepresented in the text of the article. He does have five citations in our International Press publications, but they were overlooked because they were attributed to "Bill Gale."

* This differential for "Bill" vs. "William" Gale led us to run a robustness check for all of our top fifty economists (actually, fifty-four counting ties, accounting for 70 percent of all cites) to see whether they might have been significantly undercounted due to nicknames. Full details of this alternative search are now posted on IIE's Web site. We found that four out of the fifty-four economists had significantly higher scores when common alternative first names were included:

** Daniel/Dan Mitchell (61 vs. 42 cites; rank #20 vs. #30);

** Robert/Bob/Rob Greenstein (84 vs. 71; #13 vs. #14);

** William/Bill/Will Gale (82 vs. 75; #14 vs. #13); and

** Edwin/Ed/Ted Feulner (56 vs. 49; #22 vs. #24).

We are grateful to the many economists and their associates who brought constructive suggestions and legitimate corrections to our attention, after reading our study and examining the methodology and underlying data posted. We are grateful as well to The international Economy for giving us the opportunity both to publicize the original study and to address the questions raised about it since then. We look forward to other researchers looking at other measurable aspects of think tank performance, and making their data and methods public in turn, so that all of us in the economic policymaking and research community may learn and benefit.

ADAM POSEN
Senior Fellow
Institute for International Economics

To the Editor:

If citations are the standard by which judgements are made, you might have mentioned Irwin Stelzer, who is cited more often than anyone at the Hudson Institute and much more often than scholars you do mention.

HERBERT I. LONDON
President
Hudson Institute

FAIR Study: Think Tank Media Visibility

                                     Political      Number of Media
           Think Tank                Orientation    Citations 2001

     Brookings Institution          centrist            3,753

                                   conservative/
         Cato Institute             libertarian         2,364

      Heritage Foundation          conservative         2,044

      National Bureau of            centrist            1,901
      Economic Research

American Enterprise Institute      conservative         1,804

Council on Foreign Relations        centrist            1,570

  Center for Strategic and
   International Studies           conservative         1,525

     RAND Corporation              center-right         1,124

  Family Research Council          conservative         1,121

    Carnegie Endowment              centrist            1,081

     Urban Institute               center-left          1,077

 Economic Policy Institute         progressive           829

     Hudson Institute              conservative          767

  Citizens for Tax Justice         progressive           504

Center for Defense Information     progressive           498

    Center on Budget and           progressive           490
     Policy Priorities

     Hoover Institution            conservative          471

    Manhattan Institute            conservative          431

Center for Public Integrity        progressive           380

Competitive Enterprise Institute   conservative          370

   Washington Institute for        center-right          369
      Near East Policy

       Institute for                centrist             368
   International Economics

       Freedom Forum                centrist             368

 Institute for Policy Studies      progressive           336

 Progressive Policy Institute       centrist             298

Data source: Nexis database search of major newspapers and radio and
TV transcripts. Political orientation is based on FAIR's evaluation.

Note: The Heritage Foundation's citations were adjusted to reflect the
incidence of "false positives." Approximately 22 percent of the time
the words "heritage foundation" appeared in Nexis without referring to
the Washington-based think tank.

Source: Fairness and Accuracy in Reporting, March/April 2002, Extra!.

Think Tank Rankings for Overall Period From 1997-2002 and Five-Year
Average Cites Per Economist (including and without most cited
economist), updated October, 2002

Total Cites                                           Total
 Five Year                                  Total     No. of
   Rank               Think Tank            Cites   Economists

     1           Brookings Institution      1252        37

     2        Institute for International    781        12
                       Economics

     3            American Enterprise        662        19
                       Institute

     4              Cato Institute           334        22

     5            Hoover Institution         313        51

     6         Economic Policy Institute     269        16

     7            Heritage Foundation        228        16

     8              Urban Institute          216        15

     9           Center on Budget and        182        13
                   Policy Priorities

    10             Economic Strategy         110         6
                       Institute

    11           National Center for         107        12
                    Policy Analysis

    12                Council on             105        10
                   Foreign Relations

    13            Progressive Policy         98          9
                       Institute

    14          Carnegie Endowment for       95          8
                  International Peace

    15         Center for Strategic and      79         20
                 International Studies

    16             Hudson Institute          31         10

Total Cites     Five Year                         Cites per Economist
 Five Year    Average Cites      Most Cited          Without Most
   Rank       Per Economist       Economist         Cited Economist

     1            33.8          Robert Litan             27.4

     2            65.1        C. Fred Bergsten           42.8

     3            34.8         James Glassman            24.1

     4            15.2         Michael Tanner            13.6

     5             6.1         Robert J. Barro            5.1

     6            16.8         Jared Bernstein           10.2

     7            14.3          Edwin Feulner            11.9

     8            14.4        Robert Reischauer           8.4

     9            14.0        Robert Greenstein           9.3

    10            18.3              Clyde                 3.6
                               Prestowitz. Jr.

    11             8.9         Bruce Bartlett             3.5

    12            10.5           Benn Steil               7.4

    13            10.9          Will Marshall             4.5

    14            11.9          Anders Aslund             5.7

    15             4.0         Edward Luttwak             3.1

    16             3.1          Dennis Avery              1.7

Total Cites    % of Total Cites
 Five Year    Without Most Cited
   Rank           Economist

     1              78.7%

     2              60.3%

     3              65.4%

     4              85.3%

     5              81.8%

     6              56.9%

     7              78.5%

     8              54.6%

     9              61.0%

    10              16.4%

    11              36.4%

    12              63.8%

    13              36.7%

    14              42.1%

    15              73.4%

    16              48.4%

Thirty Highest-Ranking Scholars (Five-Year Rank and Total Cites
per Fellow) Update/Correct October 25, 2002

Five-Year                                                Total Cites
  Rank              Scholar              Institution      by Fellow

    1           C. Fred Bergsten             IIE             310
    2           Robert E. Litan           Brookings          267
    3          James K. Glassman             AEI             229
    3          Robert Reischauer       Brookings/Urban       229
    5          Nicholas R. Lardy          Brookings          149
    6           Morris Goldstein             IIE             118
    7           Kevin A. Hassett             AEI             116
    7           Jared Bernstein              EPI             116
    9            John H. Makin               AEI             105
   10       Clyde V. Prestowitz, Jr.         ESI              92
   11            Henry J. Aaron           Brookings           89
   11            Gary Hufbauer               IIE              89
   13           William G. Gale           Brookings           75
   14          Robert Greenstein            CBPP              71
   15              Bruce Katz             Brookings           70
   16            Bruce Bartlett             NCPA              68
   16            Adam S. Posen               IIE              68
   18             Alice Rivlin            Brookings           66
   19            Will Marshall               PPI              62
   20           Robert J. Barro            Hoover             57
   21           Milton Friedman            Hoover             56
   22            Anders Aslund              CEIP              55
   23          Robert W. Crandall         Brookings           52
   24          Nicholas Eberstadt            AEI              49
   24          Michael D. Tanner            Cato              49
   24            Edwin Feulner            Heritage            49
   24          C. Eugene Steuerle           Urban             49
   28          Jeffrey J. Schott             IIE              45
   29            Gary Burtless            Brookings           43
   30          Daniel J. Mitchell         Heritage            42