The history, status, and future of RQC
History of the RQC initiative
Story told by
Lutz Prechelt.
1997-2010
- 1997-12: When I receive my first article reviews as a young researcher
I learn that not every reviewer worked as thoroughly as
I would like. I understand that anonymous peer review has
a difficult incentive structure and in December 1997 decide to
do something about it:
I propose to two dozen journals that they should introduce a
"Reviewer of the Year" (RoY) award based on a concrete quality evaluation
of all reviews.
The 1997 proposal
features the same core dimensions of review
quality as RQC does today: timeliness, helpfulness for editor,
helpfulness for authors.
The (few) editors I talk to sort of like the general idea,
but say the bookkeeping for the process would be impractical.
One journal tells me they were picking up the idea, but nothing
happenes otherwise; I give up for the time being.
- 2006-06: I make another attempt with another journal. Again nothing.
- 2010-04: I make another attempt with another journal. Again nothing.
- 2010-07: I make another attempt with one of the 1997 journals.
Again some interest and discussion but no tangible effect.
2012
- 2012-09: The German Science Ministry BMBF issues a request for
proposals on the topic of "Performance Evaluation in the Scientific System".
I submit a proposal (in 2012-11) aiming at implementing the RQC procedure
on a worldwide scale.
A lot of institutions support this proposal by sending a letter of interest:
- 6 of the largest scientific publishers in the world
(together representing more than 7300 journals),
- 9 research institutions as a whole,
- individual researchers
of another 25 research institutions from 14 countries (Brazil,
Canada, China, Finland, Germany, India, Italy, Netherlands, Norway,
Spain, Sweden, Switzerland, UK, USA).
- 2012-12: One of my many contacts during the proposal phase,
Fabio da Silva, asks me if I want to become Reviewer Awards Chair
at next year's EASE conference.
This prompts me to build a prototypical implementation of RQC for conferences,
RQCconf.
2013
- 2013-01: RQCconf goes online on 2013-01-12, has a surprisingly
successful run (with only one glitch),
and sends out the 48 first-ever RQC-style reviewing receipts on
2013-04-16.
- 2013-06: Despite the interest, the BMBF proposal gets rejected.
- 2013-10: I start several attempts to arrange a community-based funding
of RQC, but the required budget (of nearly EUR 500.000) is too large.
2014
- 2014-03: I find an EU call for proposals (for rather large projects)
that has a component for which RQC fits very well. The call is due
in two weeks. I frantically search for a suitable consortium
and surprisingly even find it, but I am too late to get RQC in.
- 2014-05: I start working with Elsevier on their
"Reviewer Recognition Program" (RRP).
This was originally aiming at incentivizing the quantity of reviewing only,
but now they have become interested in picking up RQC ideas as well.
Elsevier is initially implementing the RRP in a pilot form with
only a few journals and intends to then roll it out more broadly:
first across the Elsevier universe and then across the whole publishing
industry (as a free service).
My attempts at making RRP pick up the full RQC idea fail, however.
Elsevier is going for a much simpler grading scheme.
2015
- 2015-03: I finally decide that RQC is likely not going to happen
unless I build the software myself, so I decide to do just that,
starting with the version for conferences (RQCconf).
2016
- 2016-03: My sabbatical starts and RQC software development picks
up a lot of speed.
However, my software development skills are not up-to-date and so
my knowledge of modern web development technologies, tools, and practices
is lacking everywhere and progress is a lot slower than
I would have thought.
But as I take a long-term perspective, I remain patient and the results
are satisfactory: Once something works, it works solidly and does
not deteriorate as work continues. Good.
- 2016-09-01: RQC goes online.
History of the RQC software implementation
Story told by
Lutz Prechelt.
2012
- 2012-12-01: Work on a rough RQCconf prototype starts.
I had already decided to build RQC in Python and now look which framework
to use: TurboGears and Django appear to be the strongest full-stack
Python web frameworks.
In one of the darker moments of my engineering life, I decide to use
TurboGears; a bad decision:
The TurboGears framework is OK, but the documentation is a mess;
Django is in fact about ten times stronger.
A quick search for the number of Google hits would have shown this,
but for some reason, I never do that.
I will struggle a lot more than necessary, but it will be alright.
2013
- 2013-01-12: RQCconf goes online and is used for the EASE 2013 conference.
(It stays online until 2016, but I never try to find other users,
because this software was meant as a proof of concept only.)
2015
- 2015-05: The first commits happen on the new implementation.
I had already decided in 2014-07 that the new implementation would
use Django so I start with that now.
I don't have much time, though, so it is going slooooowly:
3 commits in May, 5 in July, and not much faster after that...
2016
- 2016-06: First deployment on a test machine.
Functionality for conferences only.
Some features are still missing, too.
- 2016-08-17: Start of RQC alpha test (with members of the
rqc-interest mailing list
only).
- 2016-09-01: RQC goes online with complete basic functionality
for conferences using https://easychair.org.
- 2016-10-07: Corrected for a change in EasyChair URL structure.
- 2016-10-18: Corrected some more for changes in EasyChair URLs.
- 2016-10-19: Sandboxes can now be deleted, as can normal conferences
that do not yet have receipts.
- 2016-10-30: Adapted to EasyChair review tables format variability.
- 2016-10-30: Can now have a 2nd organizer for a conference-RQC.
- 2016-11-14: Can now cope with incomplete EasyChair subreviewer entries.
- 2016-11-15: Chair statusreport now sorted alphabetically.
- 2016-11-16: Clarified in email the handling of reviews submitted late.
- 2016-11-17: Explained how superchairs (of EasyChair multi-track conferences)
can determine a URL that actually works.
- 2016-11-22: Non-donated review texts now removed 100 days after end.
- 2016-11-23: Sandbox conferences now removed after 100 days.
- 2016-11-30: Corrected confusing grading status email when a co-reviewer
had opted out after you graded their reviews already.
- 2016-11-30: Emails now removed from database 100 days after end.
- 2016-12-01: Receipt percentiles now include opted-out reviewers.
- 2016-12-31: The server availability of RQC in 2016 (Q4 only) was 99.97%.
There were only 2 interruptions of more than 6 minutes,
on 2016-10-07 and 2016-11-18.
2017
- 2017-01-10: Adapted to the presence of an EasyChair navigation bar.
- 2017-04-04: Split RQC-creation into two steps so that the instructions
become easier to understand.
- 2017-04-11: Improvements to the guidance provided by the
conference RQC-creation form.
- 2017-07-18: My extension for the HotCRP conference
manuscript handling system is accepted by Eddie Kohler.
It adds a function to export all relevant reviewing data in a single
step and forms the basis for future RQC HotCRP support.
- 2017-09-07: Adapted to an EasyChair table format change
- 2017-12-20: New role-based 'Start' menu.
- 2017-12-31: The server availability of RQC in 2017 (Q1 to Q4) was 99.59%.
There were 6 interruptions of more than 6 minutes: one in January,
one in August, and four in December.
2018
- 2018-01-16: RQC for conferences now also supports the
HotCRP
manuscript handling system.
- 2018-02-26: Better instructions for using RQC with
Easychair multitrack conferences.
- 2018-03-05: fixed a failure for the case when heuristic
firstname/lastname splitting gets it wrong for Easychair.
- 2018-04-03: made RQC robust against missing reviewer
firstname or lastname in Easychair.
- 2018-04-04: fixed failure when receipt creation encountered
single quotes.
- 2018-04-17: made RQC robust against additional columns on
submission detail page and against missing author email in Easychair.
- 2018-08-06: made RQC robust against new subheadings on reviews overview page.
- 2018-08-06: adapted to extended HotCRP user list collaborators entry format.
- 2018-09-06: adapted to the fact that EasyChair no longer allows track chairs
to impersonate co-trackchairs (aaaargh!)
- 2018-09-06: deployed the first few pieces of RQC for journals.
There is no user-visible functionality yet, but the data model and workflow
ideas have stabilized enough to make this step.
Still a long way to go, though.
- 2018-09-17: fixed a bug in conference creation due to recent library updates.
- 2018-09-18: deployed the RQC web API code for RQC for journals
(not yet activated).
- 2018-09-20: Improved the instructions for EasyChair multitrack conferences
again.
- 2018-11-19: Created the MHS info page,
updated the RQC API description.
- 2018-11-28: Fixed a regression where the wrong token would
be stored for a conference (since a forms library version change).
- 2018-12-10: Introduced Terms and Privacy.
- 2018-12-31: The server availability of RQC in 2018 was 100.00%.
There were zero interruptions of more than 6 minutes.
2019
- 2019-03-22: EasyChair no longer allows a user to determine the correct
all-reviews page URL for RQC, so RQC now starts from landingpage and role
and determines the URL itself.
- 2019-05-07: Adapted to four EasyChair URL changes
(e.g.
/account/signin.cgi
became /account/signin
)
- 2019-05-15: Added support for "discussants": Graders can skip
grading empty reviews without reduction in their
"has graded co-reviews" score.
- 2019-05-24: Adapted to changed EasyChair landing page URL structure.
- 2019-08-28: Adapted to a changed HotCRP JSON field name.
2020
- 2020-07-24: Adapted to changes in HotCRP API.
- 2020-07-28: Adapted to changes in EasyChair URL structure.
Future of the RQC initiative
- RQC is probably going to spread only slowly for a while,
because conferences have sufficiently high relevance for
serious publishing only in computer science and a few other
disciplines.
- So the scope of the software implementation is the key
limiter currently and extending the software is the main
medium-term activity and goal.
See below for the plans in that regard.
- But once the software functionality for publishers and journals
becomes avaiable, some serious advertizing may lead to fast
spread of the idea.
In my experience, the RQC idea sharply splits reviewers into
two factions: Many think it's a very good idea, the others hate
it -- most often because they despise all forms of quantification
of their work.
Future of the RQC software implementation
Many small changes and improvements will be made to the software
frequently for a long time.
The larger pending improvement steps are the following, but
development capacity is small and irregular, so there are
no scheduled dates for any of this.
1. RQC manuscript handling system API
The main reason why RQC starts with conferences is that it is simpler:
- The workflow is simpler, not involving as many types of people and having
a compact and well-defined time frame.
- The availability of reviewing data is also simpler, because
RQC fetches the data from a conference management system
rather than relying on that system to send it.
This approach is not realistic for journals for various reasons.
For journals, the manuscript handling systems will need to tell RQC
what happens wrt reviewing so RQC can handle those episodes.
For that, RQC needs to provide an API (application programming interface),
a well-documented and publicly available set of operations that can
be called over the network via HTTP.
To make RQC available to journals,
the providers of the manuscript handling systems need to extend their
software to call the RQC API.
(Getting them to do that will likely be one of the most difficult
aspects of achieving large-scale RQC adoption.)
Such an API needs to be very well thought-out, because it cannot be
changed often -- the providers of the manuscript handling systems
would not like that.
Therefore, we will need a playground for trying it out and making
it sound and stable. This playground will be OJS,
an Open Source manuscript handling system used by many Open Access journals.
2. API trial phase
Next are a number of trial uses with journals using OJS
in order to validate and perfect the API.
3. RQC for journals and publishers
Only then does it make sense to actually roll out the API.
Until the providers of the manuscript handling systems have
implemented their use of the API, there is time to
build the remaining parts of the journal and publisher
functionality.
4. RQC for research institutions
- Research institutions can register at RQC (with manual accreditation
beforehands).
- They can then create an arbitrary number of subunits and sub-subunits.
- Reviewers can assign the receipts they have received each to
one such research institution or subunit.
- Research institutions can download data about the receipts assigned
to them (essentially one record per receipt describing the quantitative
aspects of that receipt).
- Subunits can also download their parts of that data.
5. ...and then some
Later, there will be several extensions of the core functionality of RQC
that will provide additional benefits.
Expect news about RQC for a long time.
Want updates?
If you would like to receive updates on the status of RQC
(12 messages per year max.),
subscribe to the mailing list.