Review Quality Collector (RQC) is an initiative for improving the quality of scientific peer review. Its core is a mechanism that supplies a reviewer with a receipt for their work for each conference instance or journal year.
Reviewing is important for the scientific process, but so far does produce very little benefit for the reviewer in the currency that counts in research: reputation. This makes many reviewers invest less care into their reviewing than would be necessary for reviewing to function properly, which hurts the quality of the scientific process.
Please see the description of the general mechanism on the homepage and specifically of the grading system on the grading system page. Please see the description of usage procedures on the how-to page.
See the description of the history of RQC.
As a reviewer, you will always be able to opt out of RQC processes and even if you don't, you can keep some or all of your receipts private. As a conference chair or journal editor, you may have to resist some pressure from your reviewers (and then risk losing them).
The RQC logo symbolizes the crowded schedules of typical reviewers.
RQC is useful for conferences and workshops large or small. The higher quality you expect from the reviewing process, the more reason there is to use RQC. For a workshop that will accept almost everything that is on-topic, RQC might be an overkill.
Ideally, you will announce the use of RQC when you hire your reviewers; if you announce it later, that's OK, too, as reviewers can still opt out. The actual review grading is best done during the discussion phase (after reviewing, before notification); one week is a good timeframe.
Reviewers will have to read and understand each co-review. Beyond that, it will take less than 5 minutes per co-review for the actual grading.
Conference RQC Organizers will have the following effort elements:
See the how-to page for the procedure description.
Journal Editors: (TODO)
RQC is compatible with these modes. Please see the description of the RQC reviewing model.
RQC is and will remain free for reviewers, for conferences, and for research institutions.
Once it is established, it is intended to require a per-receipt fee from publishers of journals.
Yes, you can. See How to try out RQC with fake data.
The How-To page has overviews what you need to do in order to use RQC if you are a reviewer, conference PC chair, journal editor, publisher, or research institution, or if you simply want to try RQC out in a sandbox to get a better idea how it works.
So far, only there is functionality only for conference chairs (and their reviewers): review quality definition, grading, receipt creation; and only for conferences using the EasyChair or HotCRP manuscript handling systems (MHS).
The functionalty for journals and publishers will be coming next:
An API for journal MHSs to submit reviewing data and interactive
functionality for creating journals and managing the grading.
(No, I will not promise any particular arrival time.)
Functionality for research institutions will arrive in phase 3.
RQC is still in its early adopter phase.
Review quality definitions (RQdefs) are represented as RTF (Rich Text Format) files.
On Windows, the natural choice for editing such files would
be MS Word.
Note that wordpad, although nominally it can edit RTF,
can do so only in a very restricted manner. The resulting files
may look alright, but they use an impoverished style of encoding
and RQC will not be able to analyze them.
On Mac OS X, use MS Word or LibreOffice Writer (TextEdit is not going to work).
On Linux, LibreOffice Writer is the most suitable choice.
On Android and iOS, various software might be able to produce suitable files, but I recommend to avoid these platforms.
Finally, Google Docs can handle RTF as well without any local software installation, just by using one of the major web browsers. It also allows a team to collaboratively edit the document at the same time, which is nice to get a good discussion e.g. during a Skype meeting.
Whichever software you use, make sure you store the result of your editing efforts as RTF again, even if the software suggests some other format.
The software is not yet very robust against various kinds of external problems, but the implementation is well-covered by automated tests and ought to be quite reliable.
Yes. There is a list of changes on the RQC history & future page.
Without the following great software and services, RQC could not exist:
In alphabetical order:
Fri 2018-04-27 06:12 UTC
Review Quality Collector © 2012-2018