Bias in peer review of organic farming grant applications

Summary

An analysis of peer reviews of 84 organic farming grant applications, submitted to The Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning (FORMAS) in 2001 was carried out to investigate whether peer reviews were influenced by the peer reviewers’ affiliation to organic agriculture research and to what degree peer reviewers distinguish between scientific quality and societal relevance. Fifteen reviewers were grouped in three groups (1) scientists with affiliation to organic farming research, (2) scientists without affiliation to organic farming research and (3) users of the research. The scientist groups assessed societal relevance of the grant applications and scientific quality as a sum of three scientific criteria and the user group assessed solely societal relevance. The analysis showed poor agreements between reviewer ratings with respect to societal relevance and all scientific criteria expect applicant’s qualifications. The scientists’ affiliation to organic farming research created systematic biases in the peer review process. Scientists, who were experienced in organic farming research, were more in agreement with the users concerning the relevance of grant applications than reviewers without this experience. Regardless of affiliation to organic farming, both scientific reviewer groups did not clearly distinguish between societal relevance and scientific quality of the grant applications. The study challenges the idea of an objective science. In practice, the contextual values, which are associated with the traditional norms of good agriculture, were not clearly distinguished from the constitutive values of science, which are associated with the norms of good science. The reviewers’ engagement in organic farming research was important in the review process, and the best way to handle this “problem” is discussed in the paper.