The history, status, and future of RQC

Last updated 2021-09-01

Elsewhere:

• RQC overview

History of the RQC initiative

Story told by Lutz Prechelt.

1997-2010: Early impulse (but to no avail)

• 1997-12: When I receive my first article reviews as a young researcher I learn that not every reviewer worked as thoroughly as I would like. I understand that anonymous peer review has a difficult incentive structure and in December 1997 decide to do something about it:
  I propose to two dozen journals that they should introduce a "Reviewer of the Year" (RoY) award based on a concrete quality evaluation of all reviews.
  The 1997 proposal features the same core dimensions of review quality as RQC does today: timeliness, helpfulness for editor, helpfulness for authors. The (few) editors I talk to sort of like the general idea, but say the bookkeeping for the process would be impractical. One journal tells me they were picking up the idea, but nothing happens otherwise; I give up for the time being.
• 2006-06: I make another attempt with another journal. Again nothing.
• 2010-04: I make another attempt with another journal. Again nothing.
• 2010-07: I make another attempt with one of the 1997 journals. Again some interest and discussion but no tangible effect.

2012-2016: From idea to proof-of-concept

• 2012-09: The German Science Ministry BMBF issues a request for proposals on the topic of "Performance Evaluation in the Scientific System". I submit a proposal (in 2012-11) aiming at implementing the RQC procedure on a worldwide scale.
  A lot of institutions support this proposal by sending a letter of interest:
  • 6 of the largest scientific publishers in the world (together representing more than 7300 journals),
  • 9 research institutions as a whole,
  • individual researchers of another 25 research institutions from 14 countries (Brazil, Canada, China, Finland, Germany, India, Italy, Netherlands, Norway, Spain, Sweden, Switzerland, UK, USA).
• 2012-12: One of my many contacts during the proposal phase, Fabio da Silva, asks me if I want to become Reviewer Awards Chair at next year's EASE conference. This prompts me to build a prototypical implementation of RQC for conferences, RQCconf.
• 2013-01: RQCconf goes online on 2013-01-12, has a surprisingly successful run (with only one glitch), and sends out the 48 first-ever RQC-style reviewing receipts on 2013-04-16.
• 2013-06: Despite the interest, the BMBF proposal gets rejected.
• 2013-10: I start several attempts to arrange a community-based funding of RQC, but the required budget (of nearly EUR 500.000) is too large.
• 2014-03: I find an EU call for proposals (for rather large projects) that has a component for which RQC fits very well. The call is due in two weeks. I frantically search for a suitable consortium and surprisingly even find it, but I am too late to get RQC in.
• 2014-05: I start working with Elsevier on their "Reviewer Recognition Program" (RRP).
  This was originally aiming at incentivizing the quantity of reviewing only, but now they have become interested in picking up RQC ideas as well.
  Elsevier is initially implementing the RRP in a pilot form with only a few journals and intends to then roll it out more broadly: first across the Elsevier universe and then across the whole publishing industry (as a free service).
  My attempts at making RRP pick up the full RQC idea fail, however. Elsevier is going for a much simpler grading scheme.
• 2015-03: I finally decide that RQC is likely not going to happen unless I build the software myself, so I decide to do just that, starting with the version for conferences (RQCconf).
• 2016-03: My sabbatical starts and RQC software development picks up a lot of speed. However, my software development skills are not up-to-date and so my knowledge of modern web development technologies, tools, and practices is lacking everywhere and progress is a lot slower than I would have thought. But as I take a long-term perspective, I remain patient and the results are satisfactory: Once something works, it works solidly and does not deteriorate as work continues. Good.
• 2016-09-01: RQC for conferences goes online.

2017-2021: RQC for journals

• Some advertising of RQC at conferences. A few of them pick up RQC and use it. About half of the reviewers opt out. I take this as a proof of concept and do little to increase usage, because most conference instances create some support overhead as essentially every new conference organizer is an RQC beginner.
• Near-invisible work on RQC-for-journals, in particular a web API by which manuscript handling systems (MHS) can submit reviewing cases
• Work on an RQC adapter for the Open Journal Systems (OJS) open source MHS system.
• 2021-04: Another sabbatical starts and RQC software development picks up a lot of speed. And this time I am more capable.
• 2021-09: The demomode variant of RQC-for-journals goes online.

History of the RQC software implementation

Story told by Lutz Prechelt.

2012-2013: RQCconf prototype

• 2012-12-01: Work on a rough RQCconf prototype starts. I had already decided to build RQC in Python and now look which framework to use: TurboGears and Django appear to be the strongest full-stack Python web frameworks. In one of the darker moments of my engineering life, I decide to use TurboGears; a bad decision: The TurboGears framework is OK, but the documentation is a mess; Django is in fact about ten times stronger. A quick search for the number of Google hits would have shown this, but for some reason, I never do that. I will struggle a lot more than necessary, but it will be alright.
• 2013-01-12: RQCconf goes online and is used for the EASE 2013 conference. (It stays online until 2016, but I never try to find other users, because this software was meant as a proof of concept only.)

2015-2016: RQC-for-conferences

• 2015-05: The first commits happen on the new implementation. I had already decided in 2014-07 that the new implementation would use Django so I start with that now. I don't have much time, though, so it is going slooooowly: 3 commits in May, 5 in July, and not much faster after that...
• 2016-06: First deployment on a test machine. Functionality for conferences only. Some features are still missing, too.
• 2016-08-17: Start of RQC alpha test (with members of the rqc-interest mailing list only).
• 2016-09-01: RQC goes online with complete basic functionality for conferences using https://easychair.org.
• 2016-10-07: Corrected for a change in EasyChair URL structure.
• 2016-10-18: Corrected some more for changes in EasyChair URLs.
• 2016-10-19: Sandboxes can now be deleted, as can normal conferences that do not yet have receipts.
• 2016-10-30: Adapted to EasyChair review tables format variability.
• 2016-10-30: Can now have a 2nd organizer for a conference-RQC.
• 2016-11-14: Can now cope with incomplete EasyChair subreviewer entries.
• 2016-11-15: Chair statusreport now sorted alphabetically.
• 2016-11-16: Clarified in email the handling of reviews submitted late.
• 2016-11-17: Explained how superchairs (of EasyChair multi-track conferences) can determine a URL that actually works.
• 2016-11-22: Non-donated review texts now removed 100 days after end.
• 2016-11-23: Sandbox conferences now removed after 100 days.
• 2016-11-30: Corrected confusing grading status email when a co-reviewer had opted out after you graded their reviews already.
• 2016-11-30: Emails now removed from database 100 days after end.
• 2016-12-01: Receipt percentiles now include opted-out reviewers.
• 2016-12-31: The server availability of RQC in 2016 (Q4 only) was 99.97%. There were only 2 interruptions of more than 6 minutes, on 2016-10-07 and 2016-11-18.

2017-2021: RQC-for-journals

• 2017-01-10: Adapted to the presence of an EasyChair navigation bar.
• 2017-04-04: Split RQC-creation into two steps so that the instructions become easier to understand.
• 2017-04-11: Improvements to the guidance provided by the conference RQC-creation form.
• 2017-07-18: My extension for the HotCRP conference manuscript handling system is accepted by Eddie Kohler. It adds a function to export all relevant reviewing data in a single step and forms the basis for future RQC HotCRP support.
• 2017-09-07: Adapted to an EasyChair table format change
• 2017-12-20: New role-based 'Start' menu.
• 2017-12-31: The server availability of RQC in 2017 (Q1 to Q4) was 99.59%. There were 6 interruptions of more than 6 minutes: one in January, one in August, and four in December.
• 2018-01-16: RQC for conferences now also supports the HotCRP manuscript handling system.
• 2018-02-26: Better instructions for using RQC with Easychair multitrack conferences.
• 2018-03-05: fixed a failure for the case when heuristic firstname/lastname splitting gets it wrong for Easychair.
• 2018-04-03: made RQC robust against missing reviewer firstname or lastname in Easychair.
• 2018-04-04: fixed failure when receipt creation encountered single quotes.
• 2018-04-17: made RQC robust against additional columns on submission detail page and against missing author email in Easychair.
• 2018-08-06: made RQC robust against new subheadings on reviews overview page.
• 2018-08-06: adapted to extended HotCRP user list collaborators entry format.
• 2018-09-06: adapted to the fact that EasyChair no longer allows track chairs to impersonate co-trackchairs (aaaargh!)
• 2018-09-06: deployed the first few pieces of RQC for journals. There is no user-visible functionality yet, but the data model and workflow ideas have stabilized enough to make this step. Still a long way to go, though.
• 2018-09-17: fixed a bug in conference creation due to recent library updates.
• 2018-09-18: deployed the RQC web API code for RQC for journals (not yet activated).
• 2018-09-20: Improved the instructions for EasyChair multitrack conferences again.
• 2018-11-19: Created the MHS info page, updated the RQC API description.
• 2018-11-28: Fixed a regression where the wrong token would be stored for a conference (since a forms library version change).
• 2018-12-10: Introduced Terms and Privacy.
• 2018-12-31: The server availability of RQC in 2018 was 100.00%. There were zero interruptions of more than 6 minutes.
• 2019-03-22: EasyChair no longer allows a user to determine the correct all-reviews page URL for RQC, so RQC now starts from landingpage and role and determines the URL itself.
• 2019-05-07: Adapted to four EasyChair URL changes
• 2019-05-15: Added support for "discussants": Graders can skip grading empty reviews without reduction in their "has graded co-reviews" score.
• 2019-05-24: Adapted to changed EasyChair landing page URL structure.
• 2019-08-28: Adapted to a changed HotCRP JSON field name.
• 2019-12-31: The server availability of RQC in 2019 was 100.00%. There were zero interruptions of more than 6 minutes.
• 2020-07-24: Adapted to changes in HotCRP API.
• 2020-07-28: Adapted to changes in EasyChair URL structure.
• 2020-12-31: The server availability of RQC in 2020 was 100.00%. There were zero interruptions of more than 6 minutes.
• 2021-04-01: Sabattical starts. RQC is going to make a lot of progress in the next months. Most of it will initially not be visible to the public, but I'll anounce it here anyway.
• 2021-04-05 [not visible]: Demo Mode (demomode) introduced, which will become a virtual universe of publishers and journals for trying out RQC.
• 2021-04-06: Glossary begun.
• 2021-04-17 [not visible]: Can create Publishers and assign Publisherpersons.
• 2021-04-19 [not visible]: Can create Journals and assign RQguardians.
• 2021-04-28 [not visible]: Can create RQquestions.
• 2021-05-10 [not visible]: Can create Subjournals and RQdefs.
• 2021-05-21 [not visible]: Very preliminary form of API.
• 2021-06-02 [not visible]: Form for handcrafting pseudo-MHS-submissions to the API.
• 2021-06-14 [not visible]: Grading form for journal editors.
• 2021-06-15 [not visible]: Receipts in HTML format (for journals) besides PDF (for conferences).
• 2021-06-17 [not visible]: Overview added, many documentation pages updated.
• 2021-06-25 [not visible]: Lots of documentation improvements.
• 2021-07-08 [not visible]: First worklist: Statuspage for RQguardians introduced.
• 2021-07-20 [not visible]: Statuspage for Editors introduced.
• 2021-07-20 [not visible]: Reassign all objects of an anonymous user to a user account when an account is registered for that email address.
• 2021-07-23 [not visible]: Worklists for reviewers.
• 2021-08-13 [not visible]: Demomode email notification for reviewers.
• 2021-08-19 [not visible]: Demomode email notification for editors and RQguardians.
• 2021-08-23 [not visible]: Improved on-page menus and navigation throughout.
• 2021-08-25 [not visible]: Demomode manual receipt creation (and receipts browsing).
• 2021-09-01: Deployed all of the above 2021 changes. The functions are now visible and can be tried out in Demo Mode.

2022: Consolidation

• 2022-01-24: Adapted to changes in EasyChair default review form.
• 2022-07-25: Adapted to changes in HotCRP review form JSON format.
• 2022-07-28: added PDF rendering for journal receipts
• 2022-08-02: added HTML rendering for conference receipts

Current status of the RQC software implementation

To understand the current implementation status, you can either look at the past steps above and the next steps below and deduce the frontier between them. Or you look at the short summary on the homepage or at the summary in the FAQs

Future of the RQC initiative

• I hope to get some interest of early adopters to play with RQC in demomode
• Once the OJS RQC adapter is working, I will start advertising RQC to journals using OJS.
• I will also start advertising the whole RQC idea to universities.
  In my experience, the RQC idea sharply splits reviewers into two factions: Many think it's a very good idea, the others hate it -- most often because they despise all forms of quantification of their work.

Future of the RQC software implementation

Many small changes and improvements will be made to the software frequently for a long time. The larger pending improvement steps are the following, but development capacity is small and irregular, so there are no scheduled dates for any of this.

1a. OJS adapter and API trial phase

Once OJS becomes the first MHS to support RQC, an API trial phase will begin, where (possibly) changes to the API may be made.

1b. RQC for journals and publishers

The proper, non-demomode variant of RQC-for-journals may be finished and released before, during, or after the above.

2. RQC for research institutions

• Research institutions can register at RQC (with manual accreditation beforehands).
• They can then create an arbitrary number of subunits and sub-subunits.
• Reviewers can assign the receipts they have received each to one such research institution or subunit.
• Research institutions can download data about the receipts assigned to them (essentially one record per receipt describing the quantitative aspects of that receipt).
• Subunits can also download their parts of that data.

3. ...and then some

Later, there will be several extensions of the core functionality of RQC that will provide additional benefits.

Expect news about RQC for a long time.

Want updates?

If you would like to receive updates on the status of RQC (12 messages per year maximum.), subscribe to the mailing list.