How To Use Review Quality Collector (RQC)

On this page:

How To Use RQC as a reviewer at a conference

  1. Convince the PC chairs of your conference to apply RQC
  2. Perhaps participate in the process of deciding the review quality definition or even use an RQdef suggestion to start step 1.
  3. Once the PC chairs set up RQC, follow the instructions in the emails you receive from RQC.

How To Use RQC as a reviewer at a journal

The RQC functionality for journals is not yet implemented!
Once that changes, the process will be:

  1. Convince the editors of your journal to apply RQC
  2. Perhaps participate in the process of deciding the review quality definition or even use an RQdef suggestion to start step 1.
  3. Once the editors set up RQC, follow the instructions in the emails you receive from RQC (only if they involve their reviewers in the review grading).
  4. Sit back and wait for your receipt.

How To Use RQC as a program committee chair at a conference

  1. Decide the review quality definition and decide whether to hide reviewer names in the manuscript handling system in order to blind the RQC grading process.
  2. Inform your program committee members (ideally when you hire them) that the conference will apply RQC.
    Here is a possible paragraph for doing that:
    We will use Review Quality Collector (RQC) to provide you with a reviewing quality receipt as a tangible return for your work. Read up about this service on https://reviewqualitycollector.org.
    If you don't want to participate, you can opt out of the process with three clicks when it starts.
  3. Register a personal account at RQC. (Reviewers need no account.)
  4. Create an RQC instance for your conference
    (or for a sandbox pseudo-conference, see below)
    and upload your review quality definition.
  5. Most of the process runs automatically then:
    • reviewers get invited (and can opt out)
    • reviewers grade reviews (or not)
    • reviewers get reminded
      • again
    • the grading period ends and you receive a report
  6. If any reviews have received no grading, you need to step in and provide one. You can also extend the grading period.
  7. Create the receipts and review the ranking table; print some receipts if needed and perhaps have an awards ceremony at the conference.
  8. Make RQC send out the receipt to each reviewer.

The last few steps are all activated from the conference overview page, which you find on your conference list page.

You can play around with this with no risk by using sandbox mode; see How To Try Out RQC with fake data. This way, you will get to know RQC from the organizer's and the reviewers' point of view.

How To Use RQC as a journal editor

The RQC functionality for journals is not yet implemented!
Once that changes, the process will very roughly be:

  1. Convince your publisher to register for RQC
  2. Ask your publisher to provide you with an RQC key
  3. Deposit the key in your manuscript handling system
  4. Have all relevant editors sign up for an RQC account
  5. Configure the RQC roles of those editors for your journal.
  6. Decide and upload a review quality definition
  7. Start using RQC
  8. Receipts will be created and sent out automatically.

How To Use RQC as a publisher

The RQC functionality for journals and publishers is not yet implemented!
Once that changes, the process will very roughly be:

  1. Write a signed letter to register for RQC.
  2. Make sure the manuscript handling system(s) used by your journals support RQC (they need to implement the RQC API).
  3. For each journal, generate an RQC usage key and hand it over to the respective editor, so the journal can use RQC and will sail under your flag there.
  4. You will be able to see RQC reviewing statistics for each journal.
    Your journals will be able to provide RQC reviewing receipts.
    Other than accrediting the journals, no work is needed from your side.

How To Use RQC as a researcher at a research institution

The respective RQC functionality is still being implemented.

How To Use RQC as a research institution

The respective RQC functionality is still being implemented.

How To Try Out RQC with fake data: sandbox mode

To understand the RQC ideas and process more thoroughly, you can play through a complete scenario in about an hour or so, as follows.

  1. Register for an RQC account (Login→sign up for an account).
    This is a one-time action, please provide only serious data here.
    • Look into your email and confirm you email address
  2. Log in
  3. Create an RQC instance for a conference (Start→Create a conference-RQC). Hints:
    1. Carefully read and follow the explanations on that page.
    2. Think up a funny conference acronym and name, use any random web page as the conference homepage.
    3. Use sandbox instead of a real conference MHS page URL. This will result in a fake conference with no Easychair access. It will have 4 submissions, 4 reviewers, and 1 to 4 reviews per submission with no real or even pseudo review text.
    4. Choose a reviewing deadline a few minutes in the past.
    5. Choose a grading start about 10 minutes in the future and a grading end about half an hour later.
    6. When uploading the RQdef, start from the example definition, skim through the instructions, modify entries as you like, but keep only about 1 facet for "Helpfulness for Authors" and about 2 facets for "Helpfulness for Decision" in order to make the grading process shorter.
  4. Wait for the grading start time to arrive. (RQC checks time every 5 minutes only.)
    • Look into your email and simply follow the instructions in the various emails you will get and on the web pages they lead to.
    • The process is highly email-driven.
  5. When grading, think of a certain profile for each reviewer and grade accordingly, so that the whole thing makes some sense.
    • For instance, Alice Thorough might be a super reviewer, Bobonu Ubuntu a decent reviewer, Carla Verace a problematic reviewer, and Dag Wönnegården might opt out.
    • Monitor your email throughout the process.
      Be aware that the emails you receive and the web pages you work with would normally be used by five different people, not just yourself.
    • Do not finish all of the grading for all reviewers in order to get to see what orphan grading is like.
  6. Once grading time has ended and grading is more or less complete, generate the receipts and look at some of them.
    • Think how you could use them for a "best reviewers" award ceremony at your conference.
  7. Now send out the receipts.
  8. Congrats! That's it!
    You are now a somewhat experienced RQC user (and thus a member of a tiny elite).

How To Create a review quality definition

  1. Read up how RQC grading works overall.
  2. Find out which software you are (or could be) using that can handle RTF files.
  3. Download the RQdef template.
  4. Study its introduction.
  5. Study the suggested facets and decide which ones to keep in the same or modified form. Perhaps add further facets. Arrange them all to your liking.
  6. Set the localweights of the facets within each aspect.
    Set the weights of the aspects.

The next section discusses content decisions.

A new RQdef is required for each instance of a conference. A new RQdef is possible for each reviewing year of a journal. In those cases, you can reuse the previous RQdef with or without modifications or start from the template again.

Considerations for good RQdefs



Tue 2017-08-22 03:29 UTC

Review Quality Collector  © 2012-2017

Imprint / Impressum