Manuscript Characteristics Affecting Reviewers' Decisions for Rehabilitation Counseling Related Journals.

The growth and development of any profession depends in part on how well its members communicate innovative ideas, trends, practices, and research findings with colleagues and the public. Journal publication is one major vehicle for disseminating such information (La Forge & Coelho, 1998). Moreover, the quality of a profession's journals is a representation of the quality of the profession itself. Thus, vigilance by editorial boards will ensure that the quality of both is maintained.

The primary objective of scholarly journals and their editorial review boards is the dissemination of accurate and valid research. The peer review system is the heart of the publication process and a major pathway to publication for scholars (Beebe, 1993). Editorial boards are staffed by journal reviewers who are recognized for their expertise in specific topical areas. They are charged with upholding journal policy on the standards for publication and for reading and evaluating the scholarly merits of manuscripts the journal receives (Coelho & Saunders, 1997). Although peer reviewers play an important role in controlling the quality of scholarly work, little research has systematically examined the criteria employed in the manuscript selection process (Emden, 1996; King, McGuire, Longman, & Carroll-Johnson, 1997).

Reviewing manuscripts is an intellectual process with both objective and subjective elements (Kassirer & Campion, 1994). Reviewers must be objective in the sense of eliminating any personal bias toward an author's subject matter (Beebe, 1993). Also, reviewers must evaluate a manuscript as to one's own experience in and knowledge of the specialized area under study (Coelho & La Forge, 1996). Even when the journal editor provides a list of evaluation elements to be considered, peer review asks reviewers' opinion. Thus, reviewers bring an uncertain mix of preferences, skills, moods, and intellectual vagaries to the task (King et al., 1997). Reviewers are influenced by many factors in evaluating manuscripts. Some of these factors are subtle and beyond quantitative analysis (Rowney & Zenisek, 1980). Other factors are more evident and can be articulated by reviewers during review. The present study examines the basic criteria offered by reviewers of prominent rehabilitation counseling related journals for accepting or rejecting a particular manuscript submission.

Method

Participants

The 1998 listing of editorial reviewers were obtained from the Journal of Applied Rehabilitation Counseling, Journal of Rehabilitation, Rehabilitation Education, Rehabilitation Psychology and the Rehabilitation Counseling Bulletin. There were 176 reviewers on the editorial boards of the five journals, 21 of whom served on two and four of whom served on three of the editorial boards, leaving 151 individuals each of whom were mailed a survey. Each journal reviewer was mailed a four page, 63 item questionnaire, a stamped self-addressed envelope, and a cover letter. The cover letter explained the purpose of the study and assured confidentiality. A follow-up reminder to nonrespondents was sent four weeks after the initial mailing, resulting in 116 returned questionnaires, or a return rate of 77%. The five journals included in the survey, with numbers of questionnaires mailed and returned, are shown in Table 1.

Table 1 Number of Questionnaires Mailed and Returned by Journal

                                     Number    Number    Response
Journal                              Mailed   Returned   Rate (%)

Journal of Applied

Rehabilitation Counseling              43        38        88%
Journal of Rehabilitation              25        21        84%
Rehabilitation Education               37        27        73%
Rehabilitation Counseling Bulletin     31        24        77%
Rehabilitation Psychology              43        31        72%

  Total: (Unduplicated reviewers)     151       116        77%

Instrument

The questionnaire was designed to gather data along four dimensions.

1. Part I: Manuscript Characteristics. This section was adapted from Kerr, Tolliver and Petree (1977) who surveyed editors and editorial board members of management and social science journals. The 32 items dealt with such things as manuscript length, type of data, sample characteristics, generalizability, design and analysis, and statistically insignificant findings. Respondems were asked to adopt the following perspective:

   As a journal reviewer, assume that you have been asked to review a
   manuscript for publication in one of our rehabilitation counseling related
   journals. Assume further that you judge the piece to be COMPETENTLY DONE.
   However, it has the characteristic described in each item below. Please
   indicate for each item below, the importance to the decision you would make
   as a reviewer for the journal. Use the following response choices to
   indicate that the information:

   1. would add to the likelihood that you would recommend acceptance.

   2. would count neither for nor against the article.

   3. would count against the article, but not enough to cause you to
   recommend rejection.

   4. would count against the article and might cause you to recommend
   rejection.

   5. would probably cause you to recommend rejection.

   6. would almost surely cause you to recommend rejection.

2. Part II: The Review Process. Respondents were asked to use a five-point Likert-type scale (1=not important to 5=extremely important) to express how important various elements of the review process were to the respondent's personal evaluation style.

3. Part III: Editorial Review Experience. This section asked respondents to identify journals they are reviewers for, time required to evaluate a typical manuscript, experience as a reviewer, how they came to become a reviewer, and publication and conference presentation history over the prior five years.

4. Part IV: Demographics. Respondents were asked to provide information about their employment setting, educational background, professional memberships, credentials, and whether they carry an active client caseload.

One other section for additional comments appeared on the questionnaire. Space was provided for respondents to address any comments they might have about improving the journal peer review process.

Results

Demographics

One-hundred (89%) of the participants were European American, three (3%) African American, four (4%) Latino American, two (2%) Asian American, and three (3%) Other American. Participants were identified vocationally as follows: 59 (51%) rehabilitation counselor educators, 22 (19%) psychologists, nine (8%) researchers, eight (7%) rehabilitation counselors, eight (7%) administrators, and 10 (9%) other occupations (e.g., student, educator outside of rehabilitation counseling). Ten (9%) had master's degrees and 106 (91%) had doctoral degrees. Overall, 74 (64%) of the respondents were nationally certified as rehabilitation counselors, 27 (23%) were licensed as professional counselors, and 32 (28%) were licensed as psychologists. Thirty-nine (34%) carded client caseloads.

Individual memberships in professional associations included 62 (53%) in the National Rehabilitation Counseling Association, 59 (51%) in the American Rehabilitation Counseling Association, 52 (45%) in the American Psychological Association, 48 (41%) in the National Council of Rehabilitation Educators, 12 (10%) in the National Association of Rehabilitation Professionals in the Private Sector, and 47 (41%) were also members in other professional associations. The group averaged membership in three professional associations, with 61 (53%) maintaining membership in four or more professional associations. Six (5%) of the respondents were not members of any professional association.

Editorial Review Experience

Of the respondents, 77% (n=89) had been journal reviewers for five or more years. Further, 63% (n=73) had been reviewers for three to nine journals over their careers. The majority of reviewers (n=84, 72%) were from academia, with 51% (n=59) from rehabilitation counselor education programs. Doctorate degrees were held by 91% (n=106) of the reviewers. The respondents had considerable years of association with their profession; 90% (n=104) worked in the field for 10 years or more. Many respondents (n=66, 57%) review one to six manuscripts per year.

Many respondents became peer reviewers at the invitation of a journal editor (n=81, 70%) or were recommended by a colleague to serve (n=42, 36%). However, 38 respondents (33%) appealed directly to the editor offering their expertise, 14 (12%) participated in a journal reviewer training session, and 8 (7%) had been mentored before they began to review manuscripts.

Over the past five years, 48% (n=56) had published 10 or more articles in peer reviewed journals; 51% (n=59) had participated in nine or more conference presentations over the same period. Reviewers stated that they require from two to three hours to review an average manuscript (n=66, 57%). Most reviewers (n=82, 71%) indicated they had received no formal training in the editorial review process prior to starting as a reviewer.

Manuscript Characteristics

Greatest positive influence. The factor that reflected the greatest positive effect was that of a manuscript representing a new, original theory (33%). This was followed closely by an author analyzing interval data appropriately (29%). Reviewers were positively predisposed to accept manuscripts with contents of interest to the field, but differing in content from those traditionally published in the journal (23%). New statistical methods, including data collection techniques, enhanced the manuscript (20%). An author's reputation also enhanced reviewers' predisposition for manuscript acceptance (20%).

Greatest negative influence. Direct replications that added little to theory development received the most negative response (51%). Topics that present material well outside the mainstream of the field or articles that are too lengthy were subject to rejection (47% and 36%). Manuscripts that present pilot study research, with little evidence of generalizability, are likely to be rejected (43%). Experimental data with no control group and manuscripts containing only secondary analysis of data presented by others were subject to rejection (29% and 32%).

The next two items dealt with statistical criteria. Studies lacking statistical significance, be they based on either new or currently popular theories, were likely to be rejected (28%). Application of inappropriate analyses (parametric tests for ordinal data) produced a similar result (35%).

Neutral factors. The factors included here represent those that received a neutral response from at least 90% of the sample. They appeared inconsequential to reviewers in their decision making processes. These included the existence or nonexistence of references to earlier publications in the same journal. Previous presentation of the data at a conference yielded a similar result. An author being a member of the journal's editorial advisory board was also of little consequence. The reviewer's knowledge or suspicion that an author is known and writing on a topic for which he or she has no reputation, also had little consequence on the review process.

Personal Evaluation Style

Reviewers were asked to express the importance of various aspects of the review process in their personal evaluation style. Results showed that reviewers primarily consider the (a) accuracy of information in the manuscript; (b) reasonableness of the conclusions based upon appropriate data analysis; (c) contribution that the manuscript offers to the field; (d) readability, grammar, and style; and (e) organization in terms of logic (cogency) and standard format.

Reviewer Suggestions

Respondents were asked to offer suggestions for improving the current peer review system. Thirty-seven (32%) individuals provided comments grouped into three general theme areas based on their content. The first theme was reviewer training. Eighteen reviewers suggested that more ongoing training and mentorship opportunities be available for new and current reviewers. Training could also help to focus reviewers on the explicit guidelines of the journal and reduce sarcastic tones noted in some reviews. Providing regular reviewer training opportunities, through yearly meetings, teleconferences, or Internet listservs were suggested as options to help reviewers stay connected and updated. Some noted that they had attended a manuscript reviewer training session for the Journal of Applied Rehabilitation Counseling and that they found it useful. The second area that received support from 12 reviewers was receiving access to the comments of other reviewers from the manuscripts reviewed. This is a practice of the Rehabilitation Counseling Bulletin and is a most helpful feedback source for many reviewers. The third area concerned general journal operations. Seven reviewers suggested that more manuscripts be assigned to each reviewer per year and that the publication lag time be reduced.

Discussion

Overall, neutral factors predominated over positive and negative ones in making a decision on a manuscript. Although some consistency exists in the positive and negative factors related to manuscript review decisions, the highest concordance across reviewers was 33% for positive factors and 51% for negative factors. This finding seems to indicate more inconsistency than consistency in reactions to manuscripts during the review process. Reviewers bring a variety of opinions and reactions to various manuscript characteristics. It may be inherent to the peer review system or due, in part, to a lack of consistency in reviewer training within and across journals (i.e., 71% of reviewers had received no formal training in the editorial review process before beginning as a reviewer).

Manuscript Characteristics

One factor believed to be important for a manuscript to be accepted was presenting an original theory based upon personal work. However, 51% of the reviewers would reject material that simply replicates previous work without exploring new dimensions. It is important for authors to recognize that new material, correctly analyzed, plays a major role in a manuscript's selection for publication. Reviewers subscribe to the notion that a major function of journals is to advance the profession. Hence, reviewers believe that manuscripts that add little to the field should not be published.

Our findings also suggest that the quality of statistical work is important in the selection of a manuscript. Inappropriate use of statistics, inappropriate statistical analyses, lack of control groups, and poor generalizability are all key factors in rejecting a manuscript. There also may be a predisposition toward rejecting manuscripts reporting statistically insignificant findings. In other words, statistically significant findings are a sine qua non for selecting a manuscript that displays all those other significant criteria for manuscript selection.

Statistically insignificant findings could be just as meaningful as significant ones (e.g., proving or disproving some stated theory or hypothesis) depending upon the larger context of the results. Thus, there is no inherent reason to believe that a quality manuscript displaying all the other positive criteria for selection should be rejected for reporting an insignificant statistical difference. Authors, however, should know that reporting statistically significant differences may enhance the probability of a manuscript's acceptance and, conversely, reporting insignificant findings may diminish the probability of a manuscript's acceptance.

In rehabilitation counseling there is an interest in material outside the traditional scope of the journal's stated focus. Authors, however, would be wise not to stray too far outside the mainstream of the profession; topics too tangential tend to be negatively viewed. While reviewers have the responsibility of making sure that what gets published is relevant and appropriate, critical or valid information that may advance the field may be rejected because it seems too far removed from the professional mainstream.

While journals profess to using masked reviews (i.e., where the identity of the author of a manuscript is concealed from reviewers during the review process), the reality may be different. Reviewers' knowledge of the author's reputation and reviewers' belief on whether the manuscript's subject matter is within the author's field of expertise have no impact on the reviewers' decision to publish. Although 20% of the reviewers rate knowing the author and believing in the credibility of the author's work as a positive factor for selection, reviewers also report knowing the author does not have an impact upon acceptance or rejection. The questions here are: Do we in fact have masked reviews? Are reviewers attempting to guess the author's identity and base their selection decision accordingly? In that case, we are dealing with subjective bias which is explicitly against journal policy.

We recommend that when a reviewer receives a manuscript authored by a person with whom there is a personal or professional relationship, the existence of that relationship be disclosed to the editor. Further, a reviewer must terminate the review of a manuscript that the reviewer knows or suspects to be that of a friend (Coelho & Saunders, 1997). In other words, the reviewer should self disqualify from making publishing decisions because a masked review process is no longer possible.

This finding illustrates some of the difficulties in interpretation of the questionnaires' data. It is unclear if the respondents answered the question assuming knowledge of an author, when in fact they did not really know his or her identity. Is the masked review system more fiction than fact? Further investigation may be necessary to settle this question.

Personal Evaluation Style

Personal evaluation factors that affect acceptance most strongly were (a) accuracy of information in the manuscript and (b) presentation of reasonable conclusions. Referring to our earlier discussion, if the manuscript appears to offer genuine advancement of the profession, reviewers are willing to accept material that is beyond the traditional focus of a journal's topic area provided that (a) the information is accurate, (b) the study was conducted competently, and (c) the conclusions are reasonable based upon the data. On the other hand, manuscripts that are simply replications of previous studies with no new dimensions are more likely to be rejected, whether or not they meet the reviewer's personal evaluation factors.

The last two factors concerned style. Most reviewers believe manuscripts should be readable, should exhibit good writing style, and should be carefully organized. Although these stylistic features are important, the reviewers appear to regard these as remediable with revision. Conversely, inaccurate information or poorly conducted research are not regarded as remediable with revision.

Reviewer Suggestions

Reviewers offered several suggestions for improving the current peer review system. Reviewers would like more feedback regarding their reviews than they currently receive. Some reviewers would also like training in how to review manuscripts and would like to initiate some type of mentorship for both new and current reviewers. Finally, reviewers would also like to review comments from other reviewers of the manuscript in question.

Both the request for more training and the opportunity to review other reviewer evaluations show a need for more input, training, and feedback on the reviewing process itself. Reviewing is a lonely job. There is no discussion with anyone regarding a manuscript. Usually, reviews are returned to the editor with no feedback on the competency of the review process.

Clearly, reviewers take their jobs very seriously as suggested by the high return rate of the survey and the nature of their responses. Their comments reveal the seriousness which reviewers place upon the entire review process. Demonstrating a strong desire for continuous improvement to benefit the profession, reviewers provide guidance for our journals.

Limitations

The limitations of research in which respondent participation is voluntary are applicable to these data. Although there is no reason to believe that a biased sample has been attained, nothing is known about the nonrespondents. Also, the non-random nature of the sample limits the application of the findings to the population of journal reviewers. This study used self-report information. Thus, one cannot assume that positive predispositions necessarily relate to positive behavior.

Conclusion

Professional journals are responsible for attracting and evaluating contributions to knowledge. Peer review has a key role in determining which research becomes part of the accepted body of scientific knowledge. Moreover, the quality of our journals is to a large measure in the hands of the reviewer. Reviewers are gatekeepers that have both positive and negative influences. Therefore, understanding and maintaining the quality of the peer review process is essential to the advancement of our profession.

The findings from this study are consistent with similar studies of manuscript reviewers for journals of the Canadian Psychological Association (Rowney & Zenisek, 1980) and management and social science journals in the United States (Kerr et al., 1977). The variables identified may be viewed as representative of a set of common reasons why manuscripts are accepted or rejected. They may not, however, parallel actual reviewer behavior. Further research is needed to investigate the relationship between reviewer predispositions and actual reviewer behavior. Future research should explore the standardized methods of selecting reviewers, the ways of screening and training them, and the impact of training on the reliability of their evaluations. With more careful attention to the process of selecting, training, and mentoring reviewers; the field could further enrich the quality of its journals. Little is more critical to the advancement of a profession than the policies adopted and the practices used to determine which articles are published.

Table 2 Self Reported Factors Most Influencing Reviewers' Decisions

                                  N    %

Positive Influence Items

1. The theory tested is
  new and is the author's own.    38   33%

2. It contains interval data
  and is treated accordingly.     33   29%

3. It is on a topic of
  interest to the field, but
  differs in content from
  articles traditionally
  published in the Journal.       26   23%

4. It discusses a new
  statistical test or a new
  data collection technique
  and contains no new data.       23   20%

5. You know who the author is
  and believe that s/he has a
  justifiably strong reputation
  in the area s/he writes
  about.                          22   20%

Negative Influence Items

1. It is a direct replication
  of an original study recently
  published in the journal; it
  adds no new dimension to
  theory.                         59   51%

2. It is on a topic well
  outside the mainstream of
  the field.                      54   47%

3. It is a pilot study and
  gives no evidence of
  generalizability to other
  samples or situations.          50   43%

4. The manuscript is twice as
  long as those full-sized
  articles usually appearing
  in the journal and cannot be
  condensed or divided.           42   36%

5. The author uses parametric
  statistical tests, although
  his/her data are ordinal
  (the sample is fairly
  large).                         39   35%

6. It contains only a
  secondary analysis of data
  previously collected and
  analyzed by others.             37   32%

7. The study is a field
  experiment but contains no
  control group.                  34   29%

8. The study does not yield
  results which approach
  statistical significance.
  (The theory tested is new
  and is the author's own.)       32   28%

Note: The positive influence percentages indicate respondents per item endorsing category 1, "would add to the likelihood of acceptance." The negative influence percentages indicate respondents per item endorsing categories 4, 5 and 6, "would probably (almost surely) cause rejection."

References

Becker, H.S. (1986) Writing for social scientists. Chicago: The University of Chicago Press.

Beebe, L. (1993). Professional writing for the human services, Washington, D.C.: NASW Press.

Coelho, R. J., & La Forge, J. (1996). Preparing a journal manuscript: A guide for rehabilitation counselor practitioners. Journal of Applied Rehabilitation Counseling, 27(3), 37-41.

Coelho, R. J., & Saunders, J. L. (1997). Journal publication and peer review: Guidelines and standards for authors and reviewers. Journal of Applied Rehabilitation Counseling, 28(3), 18-22.

Emden, C. (1996). Manuscript reviewing: Too long a concealed form of scholarship? Nursing Inquiry, 3, 195-199.

Kassirer, J. & Campion, E. (1994). Crude and understudied, but indispensable. Journal of the American Medical Association, 270, 96-97.

Kerr, S., Tolliver, J., & Petree, D. (1977). Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20(1), 132-141.

King, C. R., McGuire, D.B., Longman, A. J., & Carroll-Johnson, R. M. (1997). Peer review, authorship, ethics, and conflict of interest. Image: Journal of Nursing Scholarship, 29(2), 163167.

La Forge, J., & Coelho, R. J. (1998). Writing for publication: Advice from authors. Journal of Applied Rehabilitation Counseling, 29(2), 35-39.

MacDonald, S. P. (1994). Professional academic writing in the humanities and social sciences. Carbondale, IL: Southern Illinois University Press.

Rowney, R. A., & Zenisek, T. J. (1980). Manuscript characteristics influencing reviewer decisions. Canadian Psychology, 21(1), 17-21.

Richard J. Coelho Clinton-Eaton-Ingham Community Mental Health Board

Jan La Forge Wright State University

Richard J. Coelho, Ph.D., CRC, Clinton-Eaton-Ingham Community Mental Board, 812 E. Jolly Road, Suite G10, Lansing, Michigan 48910.