Your activities as a reviewer impact the careers of authors, the editorial team who has limited journal space to publish articles, and thus ultimately the quality of research and the knowledge base in our field. Thus, your job as a reviewer is twofold: 1) evaluative — is the quality level sufficient to invite a revision and perhaps ultimately an acceptance of the paper in the Journal, and is the paper's topic neither overly narrow nor overly broad to warrant publication?; and 2) developmental — can the reviewer provide constructive comments for improving the quality of the paper? While a reviewer should identify a manuscript's deficiencies, a good reviewer should then go on to provide suggestions for how the deficiencies can be addressed. To be effective, both of these roles must also be performed in a timely manner. An author's tenure and promotion decision might rest on the timeliness of the reviewer's response. Further, not only does a paper not contribute to the discipline as it sits on the reviewer's desk, but the paper's data and even ideas may become increasingly less valuable over time.
What's in it for you? There are a number of benefits to being a reviewer. First, you remain current with emerging topics and research methods and can build your critical thinking skills, both of which are useful when writing your own papers. Second, the editors track the performance of reviewers in terms of both quality and timeliness. If a reviewer is performing multiple, high-quality reviews, then s/he may be asked to serve the Journal as a member of the Review Board (or, in the case of reviewers who are currently members of the Review Board, the Associate Editor). Service as an editorial reviewer or associate editor results in external professional recognition and is often recognized internally during the merit review process and tenure and promotion process. Conversely, poor performance, in terms of timeliness and/or quality of reviews, may result in a reviewer's removal from the Review Board. Finally, your high-quality and timely reviews contribute to the development of the Journal, and thus our field. The best manuscripts are likely to be submitted to journals that provide fast turn-around and high-quality reviews, and that are open to innovative ideas and methodologies. As the editors, we depend on you for each of these.
A high-quality review consists of at least five characteristics. First, a reviewer can create a positive rapport and demonstrate an understanding of the paper by providing a brief summary of the paper at the beginning of the review, followed by a list of the paper's strengths. This shows that the reviewer has read and understands the paper, and it acknowledges the often extensive effort of the authors. The summary should also include a description of the paper's problem areas.
Second, the reviewer should provide a list of specific comments regarding weaknesses and concerns about the manuscript. These comments should not simply highlight the problem areas of the paper, but should also provide recommendations for how they might be improved. In order to truly understand the full range of these potential improvements that can be made to a manuscript, the reviewer may need to read the paper more than once, as opposed to a single, "aggressive skim".
Third, the reviewer should convey a constructive attitude. This is a common courtesy, as most papers consist of what the authors considered to be good work. Further, authors can better build on positive comments (e.g., "A clearer argument is needed prior to the introduction of Hypothesis 2. The authors may want to integrate the work of Krause et al. (2006) and related work which has examined negotiations from the economic approach — see for example Nash 1950, 1953 and Tversky and Kahneman 1991") than on negative and/or vague remarks (e.g., "The discussion in the Hypothesis section makes no sense to me."). The reviewer should, however, be careful to avoid being overly kind and giving false hope to the authors. If the reviewer believes that a manuscript cannot viably be revised and ultimately published in the Journal, or if a submitted paper suffers from a fatal flaw that is not correctable or unlikely to be corrected by the authors (for example a defect with the data collection), the reviewer needs to state why the paper cannot make a contribution to the Journal. Even if a reviewer recommends that a manuscript be rejected, s/he can still provide suggestions for improvement as noted above, along with suggestions for potential, alternative outlets for the authors' work.
Fourth, the reviewer must ensure that an article which reports the collection and analysis of data (whether quantitative or qualitative,) must either be inductive — building grounded theory using accepted techniques — or deductive — using theory to develop hypotheses which are tested through an analysis of the data. Articles which simply collect quantitative data and then report summary, descriptive statistics, or which use inferential statistics without the testing of theoretically derived hypotheses, cannot be accepted for publication. In addition, for most empirical manuscripts, whether quantitative or qualitative, reviewers must determine whether the authors have adequately assessed validity, the "sin qua non" of empirical research. Have the authors addressed both reliability and the various dimensions of validity?
Fifth and finally, the reviewer should assess the level of theoretical development in the paper. Have the authors provided logical, well-integrated, and compelling arguments for the introduction of their propositions or hypotheses? Does the paper correctly integrate existing theories in building these arguments? In addition and better yet, does the article test theory for the first time or in unique and novel ways, challenge existing theory, or synthesize recent advances and concepts to generate new theory?