The history, status, and future of RQC
Last updated 2021-09-01
History of the RQC initiative
Story told by
1997-2010: Early impulse (but to no avail)
- 1997-12: When I receive my first article reviews as a young researcher
I learn that not every reviewer worked as thoroughly as
I would like. I understand that anonymous peer review has
a difficult incentive structure and in December 1997 decide to
do something about it:
I propose to two dozen journals that they should introduce a
"Reviewer of the Year" (RoY) award based on a concrete quality evaluation
of all reviews.
The 1997 proposal
features the same core dimensions of review
quality as RQC does today: timeliness, helpfulness for editor,
helpfulness for authors.
The (few) editors I talk to sort of like the general idea,
but say the bookkeeping for the process would be impractical.
One journal tells me they were picking up the idea, but nothing
happens otherwise; I give up for the time being.
- 2006-06: I make another attempt with another journal. Again nothing.
- 2010-04: I make another attempt with another journal. Again nothing.
- 2010-07: I make another attempt with one of the 1997 journals.
Again some interest and discussion but no tangible effect.
2012-2016: From idea to proof-of-concept
- 2012-09: The German Science Ministry BMBF issues a request for
proposals on the topic of "Performance Evaluation in the Scientific System".
I submit a proposal (in 2012-11) aiming at implementing the RQC procedure
on a worldwide scale.
A lot of institutions support this proposal by sending a letter of interest:
- 6 of the largest scientific publishers in the world
(together representing more than 7300 journals),
- 9 research institutions as a whole,
- individual researchers
of another 25 research institutions from 14 countries (Brazil,
Canada, China, Finland, Germany, India, Italy, Netherlands, Norway,
Spain, Sweden, Switzerland, UK, USA).
- 2012-12: One of my many contacts during the proposal phase,
Fabio da Silva, asks me if I want to become Reviewer Awards Chair
at next year's EASE conference.
This prompts me to build a prototypical implementation of RQC for conferences,
- 2013-01: RQCconf goes online on 2013-01-12, has a surprisingly
successful run (with only one glitch),
and sends out the 48 first-ever RQC-style reviewing receipts on
- 2013-06: Despite the interest, the BMBF proposal gets rejected.
- 2013-10: I start several attempts to arrange a community-based funding
of RQC, but the required budget (of nearly EUR 500.000) is too large.
- 2014-03: I find an EU call for proposals (for rather large projects)
that has a component for which RQC fits very well. The call is due
in two weeks. I frantically search for a suitable consortium
and surprisingly even find it, but I am too late to get RQC in.
- 2014-05: I start working with Elsevier on their
"Reviewer Recognition Program" (RRP).
This was originally aiming at incentivizing the quantity of reviewing only,
but now they have become interested in picking up RQC ideas as well.
Elsevier is initially implementing the RRP in a pilot form with
only a few journals and intends to then roll it out more broadly:
first across the Elsevier universe and then across the whole publishing
industry (as a free service).
My attempts at making RRP pick up the full RQC idea fail, however.
Elsevier is going for a much simpler grading scheme.
- 2015-03: I finally decide that RQC is likely not going to happen
unless I build the software myself, so I decide to do just that,
starting with the version for conferences (RQCconf).
- 2016-03: My sabbatical starts and RQC software development picks
up a lot of speed.
However, my software development skills are not up-to-date and so
my knowledge of modern web development technologies, tools, and practices
is lacking everywhere and progress is a lot slower than
I would have thought.
But as I take a long-term perspective, I remain patient and the results
are satisfactory: Once something works, it works solidly and does
not deteriorate as work continues. Good.
- 2016-09-01: RQC for conferences goes online.
2017-2021: RQC for journals
- Some advertising of RQC at conferences. A few of them pick up RQC and
use it. About half of the reviewers opt out.
I take this as a proof of concept and do little to increase usage,
because most conference instances create some support overhead
as essentially every new conference organizer is an RQC beginner.
- Near-invisible work on RQC-for-journals, in particular a web API
by which manuscript handling systems (MHS) can submit reviewing cases
- Work on an RQC adapter for the Open Journal Systems (OJS) open source
- 2021-04: Another sabbatical starts and RQC software development picks
up a lot of speed. And this time I am more capable.
- 2021-09: The demomode variant of RQC-for-journals goes online.
History of the RQC software implementation
Story told by
2012-2013: RQCconf prototype
- 2012-12-01: Work on a rough RQCconf prototype starts.
I had already decided to build RQC in Python and now look which framework
to use: TurboGears and Django appear to be the strongest full-stack
Python web frameworks.
In one of the darker moments of my engineering life, I decide to use
TurboGears; a bad decision:
The TurboGears framework is OK, but the documentation is a mess;
Django is in fact about ten times stronger.
A quick search for the number of Google hits would have shown this,
but for some reason, I never do that.
I will struggle a lot more than necessary, but it will be alright.
- 2013-01-12: RQCconf goes online and is used for the EASE 2013 conference.
(It stays online until 2016, but I never try to find other users,
because this software was meant as a proof of concept only.)
- 2015-05: The first commits happen on the new implementation.
I had already decided in 2014-07 that the new implementation would
use Django so I start with that now.
I don't have much time, though, so it is going slooooowly:
3 commits in May, 5 in July, and not much faster after that...
- 2016-06: First deployment on a test machine.
Functionality for conferences only.
Some features are still missing, too.
- 2016-08-17: Start of RQC alpha test (with members of the
rqc-interest mailing list
- 2016-09-01: RQC goes online with complete basic functionality
for conferences using https://easychair.org.
- 2016-10-07: Corrected for a change in EasyChair URL structure.
- 2016-10-18: Corrected some more for changes in EasyChair URLs.
- 2016-10-19: Sandboxes can now be deleted, as can normal conferences
that do not yet have receipts.
- 2016-10-30: Adapted to EasyChair review tables format variability.
- 2016-10-30: Can now have a 2nd organizer for a conference-RQC.
- 2016-11-14: Can now cope with incomplete EasyChair subreviewer entries.
- 2016-11-15: Chair statusreport now sorted alphabetically.
- 2016-11-16: Clarified in email the handling of reviews submitted late.
- 2016-11-17: Explained how superchairs (of EasyChair multi-track conferences)
can determine a URL that actually works.
- 2016-11-22: Non-donated review texts now removed 100 days after end.
- 2016-11-23: Sandbox conferences now removed after 100 days.
- 2016-11-30: Corrected confusing grading status email when a co-reviewer
had opted out after you graded their reviews already.
- 2016-11-30: Emails now removed from database 100 days after end.
- 2016-12-01: Receipt percentiles now include opted-out reviewers.
- 2016-12-31: The server availability of RQC in 2016 (Q4 only) was 99.97%.
There were only 2 interruptions of more than 6 minutes,
on 2016-10-07 and 2016-11-18.
- 2017-01-10: Adapted to the presence of an EasyChair navigation bar.
- 2017-04-04: Split RQC-creation into two steps so that the instructions
become easier to understand.
- 2017-04-11: Improvements to the guidance provided by the
conference RQC-creation form.
- 2017-07-18: My extension for the HotCRP conference
manuscript handling system is accepted by Eddie Kohler.
It adds a function to export all relevant reviewing data in a single
step and forms the basis for future RQC HotCRP support.
- 2017-09-07: Adapted to an EasyChair table format change
- 2017-12-20: New role-based 'Start' menu.
- 2017-12-31: The server availability of RQC in 2017 (Q1 to Q4) was 99.59%.
There were 6 interruptions of more than 6 minutes: one in January,
one in August, and four in December.
- 2018-01-16: RQC for conferences now also supports the
manuscript handling system.
- 2018-02-26: Better instructions for using RQC with
Easychair multitrack conferences.
- 2018-03-05: fixed a failure for the case when heuristic
firstname/lastname splitting gets it wrong for Easychair.
- 2018-04-03: made RQC robust against missing reviewer
firstname or lastname in Easychair.
- 2018-04-04: fixed failure when receipt creation encountered
- 2018-04-17: made RQC robust against additional columns on
submission detail page and against missing author email in Easychair.
- 2018-08-06: made RQC robust against new subheadings on reviews overview page.
- 2018-08-06: adapted to extended HotCRP user list collaborators entry format.
- 2018-09-06: adapted to the fact that EasyChair no longer allows track chairs
to impersonate co-trackchairs (aaaargh!)
- 2018-09-06: deployed the first few pieces of RQC for journals.
There is no user-visible functionality yet, but the data model and workflow
ideas have stabilized enough to make this step.
Still a long way to go, though.
- 2018-09-17: fixed a bug in conference creation due to recent library updates.
- 2018-09-18: deployed the RQC web API code for RQC for journals
(not yet activated).
- 2018-09-20: Improved the instructions for EasyChair multitrack conferences
- 2018-11-19: Created the MHS info page,
updated the RQC API description.
- 2018-11-28: Fixed a regression where the wrong token would
be stored for a conference (since a forms library version change).
- 2018-12-10: Introduced Terms and Privacy.
- 2018-12-31: The server availability of RQC in 2018 was 100.00%.
There were zero interruptions of more than 6 minutes.
- 2019-03-22: EasyChair no longer allows a user to determine the correct
all-reviews page URL for RQC, so RQC now starts from landingpage and role
and determines the URL itself.
- 2019-05-07: Adapted to four EasyChair URL changes
- 2019-05-15: Added support for "discussants": Graders can skip
grading empty reviews without reduction in their
"has graded co-reviews" score.
- 2019-05-24: Adapted to changed EasyChair landing page URL structure.
- 2019-08-28: Adapted to a changed HotCRP JSON field name.
- 2019-12-31: The server availability of RQC in 2019 was 100.00%.
There were zero interruptions of more than 6 minutes.
- 2020-07-24: Adapted to changes in HotCRP API.
- 2020-07-28: Adapted to changes in EasyChair URL structure.
- 2020-12-31: The server availability of RQC in 2020 was 100.00%.
There were zero interruptions of more than 6 minutes.
- 2021-04-01: Sabattical starts. RQC is going to make a lot of progress
in the next months. Most of it will initially not be visible to the public,
but I'll anounce it here anyway.
- 2021-04-05 [not visible]: Demo Mode (demomode) introduced, which will become a virtual
universe of publishers and journals for trying out RQC.
- 2021-04-06: Glossary begun.
- 2021-04-17 [not visible]: Can create Publishers and assign Publisherpersons.
- 2021-04-19 [not visible]: Can create Journals and assign RQguardians.
- 2021-04-28 [not visible]: Can create RQquestions.
- 2021-05-10 [not visible]: Can create Subjournals and RQdefs.
- 2021-05-21 [not visible]: Very preliminary form of API.
- 2021-06-02 [not visible]: Form for handcrafting pseudo-MHS-submissions to the API.
- 2021-06-14 [not visible]: Grading form for journal editors.
- 2021-06-15 [not visible]: Receipts in HTML format (for journals) besides PDF (for conferences).
- 2021-06-17 [not visible]: Overview added, many documentation pages updated.
- 2021-06-25 [not visible]: Lots of documentation improvements.
- 2021-07-08 [not visible]: First worklist: Statuspage for RQguardians introduced.
- 2021-07-20 [not visible]: Statuspage for Editors introduced.
- 2021-07-20 [not visible]: Reassign all objects of an anonymous user to a user account
when an account is registered for that email address.
- 2021-07-23 [not visible]: Worklists for reviewers.
- 2021-08-13 [not visible]: Demomode email notification for reviewers.
- 2021-08-19 [not visible]: Demomode email notification for editors and RQguardians.
- 2021-08-23 [not visible]: Improved on-page menus and navigation throughout.
- 2021-08-25 [not visible]: Demomode manual receipt creation (and receipts browsing).
- 2021-09-01: Deployed all of the above 2021 changes. The functions are now visible
and can be tried out in Demo Mode.
- 2022-01-24: Adapted to changes in EasyChair default review form.
- 2022-07-25: Adapted to changes in HotCRP review form JSON format.
- 2022-07-28: added PDF rendering for journal receipts
- 2022-08-02: added HTML rendering for conference receipts
Current status of the RQC software implementation
To understand the current implementation status, you can either
look at the
past steps above and the
next steps below and deduce the frontier between them.
Or you look at the
short summary on the homepage
or at the
summary in the FAQs
Future of the RQC initiative
- I hope to get some interest of early adopters to play with RQC
- Once the OJS RQC adapter is working, I will start advertising RQC
to journals using OJS.
- I will also start advertising the whole RQC idea to universities.
In my experience, the RQC idea sharply splits reviewers into
two factions: Many think it's a very good idea, the others hate
it -- most often because they despise all forms of quantification
of their work.
Future of the RQC software implementation
Many small changes and improvements will be made to the software
frequently for a long time.
The larger pending improvement steps are the following, but
development capacity is small and irregular, so there are
no scheduled dates for any of this.
1a. OJS adapter and API trial phase
Once OJS becomes the first MHS to support RQC,
an API trial phase will begin, where (possibly) changes to the API
may be made.
1b. RQC for journals and publishers
The proper, non-demomode variant of RQC-for-journals may be finished
and released before, during, or after the above.
2. RQC for research institutions
- Research institutions can register at RQC (with manual accreditation
- They can then create an arbitrary number of subunits and sub-subunits.
- Reviewers can assign the receipts they have received each to
one such research institution or subunit.
- Research institutions can download data about the receipts assigned
to them (essentially one record per receipt describing the quantitative
aspects of that receipt).
- Subunits can also download their parts of that data.
3. ...and then some
Later, there will be several extensions of the core functionality of RQC
that will provide additional benefits.
Expect news about RQC for a long time.
If you would like to receive updates on the status of RQC
(12 messages per year maximum.),
subscribe to the mailing list.