This
article wrote as my aknowledgment for Fadhilaturrahmi, who asked me to join as
reviewer in her journal, BasicEdu. It nice to be active in peer review’s
process as a reviewer.
Since
it became an established part of the academic publishing process, peer review has
been employed to ensure research accuracy, originality, and significance. However,
like any human process, peer review is imperfect. There have been mounting concerns
expressed in recent years about quality in peer review, including biases against
null or negative results, the potential for research spin, and the ongoing replication
crisis.
As the
“gatekeepers” of research, journals and publishers are arguably on the frontlines
of quality in peer review and have the potential to lead the way in addressing many
of the challenges faced by the community. In this post, we look at three pillars
of peer review at academic journals that can help to build a stronger foundation
for quality research:
[1] Clear
peer review policies and standards
[2] Peer
review performance tracking
[3] Transparent
publishing and data sharing policies
Journals
and publishers should work to fortify these pillars by assessing their performance
in these three areas and looking for ways to improve based on past performance and
new developments in the publishing landscape. Below are specific steps that journals
and publishers can take to improve quality in peer review.
[1]
Clear peer review policies and standards
The
first pillar of quality peer review is establishing clear peer review policies and
standards. Journal policies and standards ensure that all parties involved in peer
review — editors, authors, and reviewers — know what is expected of them and that
all manuscripts are handled in the same way. To ensure quality in peer review, consistency
is key! Journals should have:
Established peer review policies: Peer review policies are a statement
on the peer review guidelines and processes the journal follows for all manuscript
submissions. Peer review policies should include an overview of the journal’s peer-review
process (e.g. blindness, reviewers per manuscript, rounds of review allotted) and
the anticipated peer review timeframe, as well as statements on publication ethics.
Some guidelines, such as the Plan S implementation criteria, also require journals
to make available annual statistics on their peer review performance to demonstrate
adherence to their policies.
Standardized submission guidelines: Authors should know what is expected
of them when preparing their manuscript for a journal, from layout requirements
to the citation and data reporting standards they’re expected to follow. All of
this information should be included in the author guidelines section of the journal
website. Journals should also require authors to provide statements of originality
and disclosures with their submission. We provide examples of journal statements
of originality and disclosures in this blog post.
Standardized reviewer feedback: Journals should establish clear guidelines
for peer reviewers to follow, including reminders of the duties of reviewers — to
be objective, diligent, and confidential in their reporting — as well as a standardized
reviewer feedback form. You can only expect reviewers to answer the questions that
you ask them, required feedback forms ensure greater consistency and quality in
referee reports.
And,
of course, all journal policies and standards must be actionable. For example, all
of the process steps explained in a journal’s peer review policies must be carried
out. So if it states all original research manuscripts will have two external reviewers,
this should always be the case. Additionally, journals must have plans in place
for enforcing all ethical policies and standards. If an article is found to have
a conflict of interest or if there is an allegation of misconduct post-publication,
the journal must have processes in place to address the situation. Additionally,
if the journal requires authors to follow certain reporting guidelines, it should
have a process in place to check for adherence.
It is
the duty of journal publishers to follow and enforce publishing best practices,
such as the Committee on Publication Ethics’ Core practices and
the Plan
S implementation guidelines, as well as disciplinary research reporting
guidelines like those compiled by equator, such
as the CONSORT checklist for randomised trials.
[2]
Peer review performance tracking
The
next pillar of quality peer review is performance tracking. Peer review quality
depends on editors following journal policies and processes and reviewers completing
reports in a timely and thorough manner. Journals should track peer review data
in the following areas:
Editorial team performance metrics: Journals should track their editorial
performance including the average manuscript acceptance rate at the editor- and
journal-level, average days to decision at the editor- and journal-level, and average
manuscripts per editor. These metrics can help journals ensure they are accepting
manuscripts at a reasonable rate and that editors are working within the peer review
timeline outlined in the journal’s policies without being overburdened. If you find
that some editors are managing more manuscripts than others or that some editors
are struggling to make manuscript decisions within the established peer-review timeframe
you know you have some areas to work on.
Reviewer performance metrics: Journals should track reviewer performance
metrics, including their average time to complete a review and number of reviews
completed. Journals can use these metrics to know which reviewers are most reliable
and spot if a reviewer may need a break from review requests.
Manuscript stages: Journals should also track data around
each manuscript’s stage in review, including reviewers per manuscript and rounds
of revisions to ensure each submission has enough reviewers and is moving through
peer review efficiently and effectively (when manuscripts get caught in manuscript
R&R cycles it can often have diminishing quality outcomes).
In addition
to tracking peer review data, journals should also have processes for assessing
the quality of the peer reviews they receive. One proposal for assessing peer-review
quality is the review quality instrument (RQI),
which “assesses the extent to which a reviewer has commented on five aspects of
a manuscript (importance of the research question, originality of the paper, strengths
and weaknesses of the method, presentation, interpretation of results).” Ultimately,
harkening back to the importance of standardized reviewer feedback forms, journals
should set expectations for referee reports that they can communicate to reviewers
in the same way in order to get more standardized feedback and, ultimately, to also
be able to review the quality of referee feedback (e.g. did each reviewer provide
a comprehensive analysis of the strengths and weaknesses of the research method
as asked).
[3]
Transparent publishing and data sharing policies
Finally,
the third pillar of quality peer review at academic journals — transparent publishing
and data sharing policies — addresses key concerns around research biases and reproducibility.
Journals can facilitate the reporting of null and negative results as well as research
reproducibility by enabling and encouraging authors to share their manuscripts and
data pre- and post-publication. This includes having preprint policies and data
sharing policies or requirements.
One
recent initiative to encourage data sharing and reproducibility is the Center for
Open Science’s Open Science Badges, which
are designed to incentivize researchers to share research data and materials and
to preregister study
and analysis plans. Journals can even opt to follow the “Registered
Reports“
framework, in which peer review occurs before study results are known. The Registered
Reports approach is designed to emphasize the quality of research methodology over
the findings of the final outcomes to eliminate peer review biases, particularly
around null and negative results.
Putting it all together:
assessment and ongoing improvement
The
peer-review process is, well, a process — meta, we know. As stated by Todd Carpenter
in the recent Scholarly Kitchen article, “Ask The Chefs: Peer Review
Quality,”
when assessing the quality of peer review, “the question shouldn’t be whether the
process is perfect. Rather, as with all scientific processes, does this process
yield a better result than might otherwise be expected without it.” At the journal
and publisher level, this means ensuring that peer review is improving research
quality through clear assessment standards, consistent processes, and policies to
promote research transparency. Journals should regularly audit their peer review
processes and performance and look for opportunities to improve based on lessons
learned and new developments in scholarly publishing. In all of this, it’s also
vital for diverse voices to be part of the discussion of quality in peer review,
to bring varied perspectives and solutions.