How To Use Review Quality Collector (RQC)

On this page:

Elsewhere:

How To Use RQC as a reviewer at a conference

  1. Convince the PC chairs of your conference to apply RQC
  2. Perhaps participate in the process of deciding the review quality definition or even use an RQdef suggestion to start step 1.
  3. Once the PC chairs set up RQC, follow the instructions in the emails you receive from RQC.

How To Use RQC as a reviewer at a journal

  1. Convince the editors of your journal to apply RQC
  2. Perhaps participate in the process of deciding the review quality definition or even use an RQdef suggestion to start step 1.
  3. Once the editors set up RQC, follow the instructions in the emails you receive from RQC (only if they involve their reviewers in the review grading).
  4. Sit back and wait for your receipt.

How To Use RQC as a program committee chair at a conference

  1. Decide the review quality definition and decide whether to hide reviewer names in the manuscript handling system in order to blind the RQC grading process.
  2. Inform your program committee members (ideally when you hire them) that the conference will apply RQC.
    Here is a possible paragraph for doing that:
    We will use Review Quality Collector (RQC) to provide you with a reviewing quality receipt as a tangible return for your work. Read up about this service on https://reviewqualitycollector.org.
    If you don't want to participate, you can opt out of the process with three clicks when it starts.
  3. Register a personal account at RQC. (Reviewers need no account.)
  4. Create an RQC instance for your conference
    (or for a sandbox pseudo-conference, see below), upload your review quality definition, and define the grading time interval.
  5. Most of the process runs automatically then:
    • reviewers get invited (and can opt out)
    • reviewers grade reviews (or not)
    • reviewers get reminded
      • again
    • the grading time interval ("grading period") ends and you receive a report
  6. If any reviews have received no grading, you need to step in and provide one. You can also extend the grading period.
  7. Create the receipts and review the ranking table; print some receipts if needed and perhaps have an awards ceremony at the conference.
  8. Make RQC send out the receipt to each reviewer.

See the workflow description for some more detail on steps 4 to 8. Steps 6 to 8 are all activated from the conference overview page, which you find on your conference list page.

You can play around with this with no risk by using sandbox mode; see How To Try Out RQC with fake data. This way, you will get to know RQC from the organizer's and the reviewers' point of view.

How To Use RQC as a journal editor

  1. Convince your publisher to register for RQC
  2. Ask your publisher to provide you with an RQC key
  3. Deposit the key in your manuscript handling system (which assumes your manuscript handling system has implemented RQC support already)
  4. Have all relevant editors sign up for an RQC account
  5. Configure the RQC roles of those editors for your journal.
  6. Decide and upload a review quality definition
  7. Start using RQC
  8. Receipts will be created and sent out automatically.

How To Use RQC as a publisher

  1. Write a signed letter to register for RQC.
  2. Make sure the manuscript handling system(s) used by your journals support RQC (they need to implement the RQC API).
  3. For each journal, generate an RQC usage key and hand it over to the respective editor, so the journal can use RQC and will sail under your flag there.
  4. You will be able to see RQC reviewing statistics for each journal.
    Your journals will be able to provide RQC reviewing receipts.
    Other than accrediting the journals, no work is needed from your side.

How To Use RQC as a researcher at a research institution

The respective RQC functionality is still being implemented. See below for a description.

How To Use RQC as a research institution

The respective RQC functionality is still being implemented. Once completed, RQC will support the following process: 1. A research institution can join RQC 2. It can then define sub-institutions and sub-sub-institutions for representing its colleges, schools, faculties, departments, institutes or whatever parts it has. 3. Reviewers can then assign their reviewing receipts to an institution or sub-institution 4. Institutions can download a tabular representation of all receipts assigned to it and its sub-institutions. Sub-institutions can likewise download a tabular representation of all receipts assigned to it and their sub-institutions.

How To Try Out RQC with fake data: Demo mode

Before you pick up RQC for real reviews, you can experience its use using fake data only. This is what demo mode is for.

The idea is that one or two people play all the roles involved, perform all the various steps, and receive all of the email notifications (no matter which email addresses occur in the data). Eventually, all the data produced is thrown away again.

Demo mode for journals

  1. Log in with checking the "demo mode" checkbox.
  2. Create a publisher
  3. Create a journal
  4. Make submit several pseudo-reviewing-records via the respective function
  5. Perform review grading on them
  6. Generate demo reviewing receipts

Demo mode for conferences: Sandbox mode

For historical reasons, conferences are not integrated into normal demo mode. Instead, they have their own, called sandbox mode; the idea is the same.

To understand the RQC ideas and process for conferences more thoroughly, you can play through a complete scenario in about an hour or so, as follows.

  1. Register for an RQC account (Login→sign up for an account).
    This is a one-time action, please provide only serious data here.
    • Look into your email and confirm you email address
  2. Log in
  3. Create an RQC instance for a conference (Start→Create a conference-RQC). Hints:
    1. Carefully read and follow the explanations on that page.
    2. Think up a funny conference acronym and name, use any random web page as the conference homepage.
    3. Use sandbox instead of a real conference MHS page URL. This will result in a fake conference with no access to any actual submission and reviewing data. It will have 4 submissions, 4 reviewers, and 1 to 4 reviews per submission with no real or even pseudo review text.
    4. Choose a reviewing deadline a few minutes in the past.
    5. Choose a grading start about 10 minutes in the future and a grading end about half an hour later.
    6. When uploading the RQdef, start from the example definition, skim through the instructions, modify entries as you like, but keep only about 1 facet for "Helpfulness for Authors" and about 2 facets for "Helpfulness for Decision" in order to make the grading process shorter.
  4. Wait for the grading start time to arrive. (RQC checks time every 5 minutes only.)
    • Look into your email and simply follow the instructions in the various emails you will get and on the web pages they lead to.
    • The process is highly email-driven.
  5. When grading, think of a certain profile for each reviewer and grade accordingly, so that the whole thing makes some sense.
    • For instance, Alice Thorough might be a super reviewer, Bobonu Ubuntu a decent reviewer, Carla Vålström a problematic reviewer, and Deshi Wong might opt out.
    • Monitor your email throughout the process.
      Be aware that the emails you receive and the web pages you work with would normally be used by five different people, not just yourself.
    • Do not finish all of the grading for all reviewers in order to get to see what orphan grading is like.
  6. Once grading time has ended and grading is more or less complete, generate the receipts and look at some of them.
    • Think how you could use them for a "best reviewers" award ceremony at your conference.
  7. Now send out the receipts.
  8. Congrats! That's it!
    You are now a somewhat experienced RQC user (and thus a member of a tiny elite).

Workflow overviews

Conference workflow (Easychair, hotcrp.com)

Roles:

Workflow:

  1. Organizer: Log in --> Start --> Create conference-RQC
  2. Organizer follows the instructions carefully:
    • Enter RQC in your EasyChair or HotCRP program committee, including the correct token.
    • If applicable, observe the extra notes for EasyChair multi-track conferences.
    • Fill in the RQC-creation form.
    • (You will now receive a welcome email.)
    • Upload your customized review quality definition (RQdef).
  3. Time proceeds until the configured review period start time:
    • RQC invites Reviewers per email to grade their co-reviews. The email contains a link to the personal grading page; an RQC account is not required.
    • Reviewers grade co-reviews or opt out of the RQC process.
    • RQC sends a thank-you email to each Reviewer after grading her last co-review.
    • RQC sends one or two reminders to Reviewers with incomplete grading.
    • RQC sends periodical status reports to Organizers.
  4. Time proceeds until the configured review period end time:
    • RQC informs (by email) Reviewers with incomplete grading that the standard grading period is now over but that they may still be able to deposit further gradings until the Organizer finishes the process.
    • RQC informs (by email) the Organizers that they should now
      • opt out Reviewers who did not do anything;
      • grade 'orphaned' reviews which did not get any grading yet.
  5. Organizers grade orphaned reviews.
  6. Organizers trigger "compute receipts".
    They review grading results, print receipts of best Reviewers, organize a "best reviewers award" ceremony at the conference.
  7. Organizers trigger "send receipts to Reviewers".
  8. RQC emails a reviewing receipt link to each Reviewer who did not opt out.

Conference workflow with a self-hosted HotCRP

This case is just like the one above, with a small difference at the beginning:

Before you can perform the actual RQC creation, you need to inform me about the address of your HotCRP instance so that I can do these three things:

Expect two workdays for this step. It is described in more detail on the RQC creation page.

How To Create a review quality definition

  1. Read up how RQC grading works overall.
  2. Find out which software you are (or could be) using that can handle RTF files.
  3. Download the RQdef template.
  4. Study its introduction.
  5. Study the suggested facets and decide which ones to keep in the same or modified form. Perhaps add further facets. Arrange them all to your liking.
  6. Set the localweights of the facets within each aspect.
    Set the weights of the aspects.

The next section discusses content decisions.

A new RQdef is required for each instance of a conference. A new RQdef is possible for each reviewing year of a journal. In those cases, you can reuse the previous RQdef with or without modifications or start from the template again.

Considerations for good RQdefs


Contact/Imprint/Impressum  ·  Terms of use  ·  Privacy  ·  Documentation hub