The history, status, and future of RQC

History of the RQC initiative

Story told by Lutz Prechelt.

1997-2010

1997-12: When I receive my first article reviews as a young researcher I learn that not every reviewer worked as thoroughly as I would like. I understand that anonymous peer review has a difficult incentive structure and in December 1997 decide to do something about it:
I propose to two dozen journals that they should introduce a "Reviewer of the Year" (RoY) award based on a concrete quality evaluation of all reviews.
The 1997 proposal features the same core dimensions of review quality as RQC does today: timeliness, helpfulness for editor, helpfulness for authors. The (few) editors I talk to sort of like the general idea, but say the bookkeeping for the process would be impractical. One journal tells me they were picking up the idea, but nothing happenes otherwise; I give up for the time being.
2006-06: I make another attempt with another journal. Again nothing.
2010-04: I make another attempt with another journal. Again nothing.
2010-07: I make another attempt with one of the 1997 journals. Again some interest and discussion but no tangible effect.

2012

2012-09: The German Science Ministry BMBF issues a request for proposals on the topic of "Performance Evaluation in the Scientific System". I submit a proposal (in 2012-11) aiming at implementing the RQC procedure on a worldwide scale.
A lot of institutions support this proposal by sending a letter of interest:
- 6 of the largest scientific publishers in the world (together representing more than 7300 journals),
- 9 research institutions as a whole,
- individual researchers of another 25 research institutions from 14 countries (Brazil, Canada, China, Finland, Germany, India, Italy, Netherlands, Norway, Spain, Sweden, Switzerland, UK, USA).
2012-12: One of my many contacts during the proposal phase, Fabio da Silva, asks me if I want to become Reviewer Awards Chair at next year's EASE conference. This prompts me to build a prototypical implementation of RQC for conferences, RQCconf.

2013

2013-01: RQCconf goes online on 2013-01-12, has a surprisingly successful run (with only one glitch), and sends out the 48 first-ever RQC-style reviewing receipts on 2013-04-16.
2013-06: Despite the interest, the BMBF proposal gets rejected.
2013-10: I start several attempts to arrange a community-based funding of RQC, but the required budget (of nearly EUR 500.000) is too large.

2014

2014-03: I find an EU call for proposals (for rather large projects) that has a component for which RQC fits very well. The call is due in two weeks. I frantically search for a suitable consortium and surprisingly even find it, but I am too late to get RQC in.
2014-05: I start working with Elsevier on their "Reviewer Recognition Program" (RRP).
This was originally aiming at incentivizing the quantity of reviewing only, but now they have become interested in picking up RQC ideas as well.
Elsevier is initially implementing the RRP in a pilot form with only a few journals and intends to then roll it out more broadly: first across the Elsevier universe and then across the whole publishing industry (as a free service).
My attempts at making RRP pick up the full RQC idea fail, however. Elsevier is going for a much simpler grading scheme.

2015

2015-03: I finally decide that RQC is likely not going to happen unless I build the software myself, so I decide to do just that, starting with the version for conferences (RQCconf).

2016

2016-03: My sabbatical starts and RQC software development picks up a lot of speed. However, my software development skills are not up-to-date and so my knowledge of modern web development technologies, tools, and practices is lacking everywhere and progress is a lot slower than I would have thought. But as I take a long-term perspective, I remain patient and the results are satisfactory: Once something works, it works solidly and does not deteriorate as work continues. Good.
2016-09-01: RQC goes online.

History of the RQC software implementation

Story told by Lutz Prechelt.

2012

2012-12-01: Work on a rough RQCconf prototype starts. I had already decided to build RQC in Python and now look which framework to use: TurboGears and Django appear to be the strongest full-stack Python web frameworks. In one of the darker moments of my engineering life, I decide to use TurboGears; a bad decision: The TurboGears framework is OK, but the documentation is a mess; Django is in fact about ten times stronger. A quick search for the number of Google hits would have shown this, but for some reason, I never do that. I will struggle a lot more than necessary, but it will be alright.

2013

2013-01-12: RQCconf goes online and is used for the EASE 2013 conference. (It stays online until 2016, but I never try to find other users, because this software was meant as a proof of concept only.)

2015

2015-05: The first commits happen on the new implementation. I had already decided in 2014-07 that the new implementation would use Django so I start with that now. I don't have much time, though, so it is going slooooowly: 3 commits in May, 5 in July, and not much faster after that...

2016

2016-06: First deployment on a test machine. Functionality for conferences only. Some features are still missing, too.
2016-08-17: Start of RQC alpha test (with members of the rqc-interest mailing list only).
2016-09-01: RQC goes online with complete basic functionality for conferences using https://easychair.org.
2016-10-07: Corrected for a change in EasyChair URL structure.
2016-10-18: Corrected some more for changes in EasyChair URLs.
2016-10-19: Sandboxes can now be deleted, as can normal conferences that do not yet have receipts.
2016-10-30: Adapted to EasyChair review tables format variability.
2016-10-30: Can now have a 2nd organizer for a conference-RQC.
2016-11-14: Can now cope with incomplete EasyChair subreviewer entries.
2016-11-15: Chair statusreport now sorted alphabetically.
2016-11-16: Clarified in email the handling of reviews submitted late.
2016-11-17: Explained how superchairs (of EasyChair multi-track conferences) can determine a URL that actually works.
2016-11-22: Non-donated review texts now removed 100 days after end.
2016-11-23: Sandbox conferences now removed after 100 days.
2016-11-30: Corrected confusing grading status email when a co-reviewer had opted out after you graded their reviews already.
2016-11-30: Emails now removed from database 100 days after end.
2016-12-01: Receipt percentiles now include opted-out reviewers.
2016-12-31: The server availability of RQC in 2016 (Q4 only) was 99.97%. There were only 2 interruptions of more than 6 minutes, on 2016-10-07 and 2016-11-18.

2017

2017-01-10: Adapted to the presence of an EasyChair navigation bar.
2017-04-04: Split RQC-creation into two steps so that the instructions become easier to understand.
2017-04-11: Improvements to the guidance provided by the conference RQC-creation form.
2017-07-18: My extension for the HotCRP conference manuscript handling system is accepted by Eddie Kohler. It adds a function to export all relevant reviewing data in a single step and forms the basis for future RQC HotCRP support.
2017-09-07: Adapted to an EasyChair table format change
2017-12-20: New role-based 'Start' menu.
2017-12-31: The server availability of RQC in 2017 (Q1 to Q4) was 99.59%. There were 6 interruptions of more than 6 minutes: one in January, one in August, and four in December.

2018

2018-01-16: RQC for conferences now also supports the HotCRP manuscript handling system.
2018-02-26: Better instructions for using RQC with Easychair multitrack conferences.
2018-03-05: fixed a failure for the case when heuristic firstname/lastname splitting gets it wrong for Easychair.
2018-04-03: made RQC robust against missing reviewer firstname or lastname in Easychair.
2018-04-04: fixed failure when receipt creation encountered single quotes.
2018-04-17: made RQC robust against additional columns on submission detail page and against missing author email in Easychair.
2018-08-06: made RQC robust against new subheadings on reviews overview page.
2018-08-06: adapted to extended HotCRP user list collaborators entry format.
2018-09-06: adapted to the fact that EasyChair no longer allows track chairs to impersonate co-trackchairs (aaaargh!)
2018-09-06: deployed the first few pieces of RQC for journals. There is no user-visible functionality yet, but the data model and workflow ideas have stabilized enough to make this step. Still a long way to go, though.
2018-09-17: fixed a bug in conference creation due to recent library updates.
2018-09-18: deployed the RQC web API code for RQC for journals (not yet activated).
2018-09-20: Improved the instructions for EasyChair multitrack conferences again.
2018-11-19: Created the MHS info page, updated the RQC API description.
2018-11-28: Fixed a regression where the wrong token would be stored for a conference (since a forms library version change).
2018-12-10: Introduced Terms and Privacy.
2018-12-31: The server availability of RQC in 2018 was 100.00%. There were zero interruptions of more than 6 minutes.

2019

2019-03-22: EasyChair no longer allows a user to determine the correct all-reviews page URL for RQC, so RQC now starts from landingpage and role and determines the URL itself.
2019-05-07: Adapted to four EasyChair URL changes (e.g. /account/signin.cgi became /account/signin)
2019-05-15: Added support for "discussants": Graders can skip grading empty reviews without reduction in their "has graded co-reviews" score.
2019-05-24: Adapted to changed EasyChair landing page URL structure.
2019-08-28: Adapted to a changed HotCRP JSON field name.

2020

2020-07-24: Adapted to changes in HotCRP API.
2020-07-28: Adapted to changes in EasyChair URL structure.

Future of the RQC initiative

RQC is probably going to spread only slowly for a while, because conferences have sufficiently high relevance for serious publishing only in computer science and a few other disciplines.
So the scope of the software implementation is the key limiter currently and extending the software is the main medium-term activity and goal. See below for the plans in that regard.
But once the software functionality for publishers and journals becomes avaiable, some serious advertizing may lead to fast spread of the idea.
In my experience, the RQC idea sharply splits reviewers into two factions: Many think it's a very good idea, the others hate it -- most often because they despise all forms of quantification of their work.

Future of the RQC software implementation

Many small changes and improvements will be made to the software frequently for a long time. The larger pending improvement steps are the following, but development capacity is small and irregular, so there are no scheduled dates for any of this.

1. RQC manuscript handling system API

The main reason why RQC starts with conferences is that it is simpler:

The workflow is simpler, not involving as many types of people and having a compact and well-defined time frame.
The availability of reviewing data is also simpler, because RQC fetches the data from a conference management system rather than relying on that system to send it.
This approach is not realistic for journals for various reasons.

For journals, the manuscript handling systems will need to tell RQC what happens wrt reviewing so RQC can handle those episodes.
For that, RQC needs to provide an API (application programming interface), a well-documented and publicly available set of operations that can be called over the network via HTTP.

To make RQC available to journals, the providers of the manuscript handling systems need to extend their software to call the RQC API. (Getting them to do that will likely be one of the most difficult aspects of achieving large-scale RQC adoption.)

Such an API needs to be very well thought-out, because it cannot be changed often -- the providers of the manuscript handling systems would not like that.
Therefore, we will need a playground for trying it out and making it sound and stable. This playground will be OJS, an Open Source manuscript handling system used by many Open Access journals.

2. API trial phase

Next are a number of trial uses with journals using OJS in order to validate and perfect the API.

3. RQC for journals and publishers

Only then does it make sense to actually roll out the API. Until the providers of the manuscript handling systems have implemented their use of the API, there is time to build the remaining parts of the journal and publisher functionality.

4. RQC for research institutions

Research institutions can register at RQC (with manual accreditation beforehands).
They can then create an arbitrary number of subunits and sub-subunits.
Reviewers can assign the receipts they have received each to one such research institution or subunit.
Research institutions can download data about the receipts assigned to them (essentially one record per receipt describing the quantitative aspects of that receipt).
Subunits can also download their parts of that data.

5. ...and then some

Later, there will be several extensions of the core functionality of RQC that will provide additional benefits.

Expect news about RQC for a long time.

Want updates?

If you would like to receive updates on the status of RQC (12 messages per year max.), subscribe to the mailing list.