On review quality definitions and the ranking
The RQC grading system is based on the following requirements:
- There should be a common grading framework that is the same
throughout all venues (conferences and journals)
and that does not change over time.
- Yet every venue should be able to define review quality in manner
that is appropriate for the specifics of its topic area
and editorial style and requirements.
- RQC is initially based on grading (using absolute terms for review quality)
but its output is in terms of a ranking (using relative reviewing quality),
so there can be no grade inflation where nearly everybody is just super.
- Review quality definitions must be sufficiently constant to allow
comparing and ranking the reviewers within a large batch of reviews.
- Yet it should also be possible to modify them over time.
These requirements are fulfilled in the following manner:
- There is a common framework of the four aspects A, D, G, T;
- Every venue has its own (and possibly unique)
review quality definition (RQdef; see below).
- The RQdef can be changed for each new instance of a conference
or reviewing year of a journal.
How to interpret the review quality definition (RQdef)
Look at the example RQdef for reference.
There are four aspects:
A: Helpfulness for Authors
D: Helpfulness for Decision
G: Graded Co-Reviews
The score for a review is computed as follows:
- One level is chosen by human graders from each A and D facet.
Its points are used.
- These points are multiplied by the respective localweight and summed.
The sum, divided by the sum of those localweights are the aspect's value.
- The first applicable level from the T aspect is used
to give its value.
- The value of the G aspect is simply the percentage of co-reviews
this reviewer has graded (or 100 if there were none to be graded).
That is why there are no facets or levels for G.
- The aspect values are multiplied by the respective weight and summed.
The sum, divided by the sum of the weights, is the total for that review.
The per-review score percentile (100 is best) is for information only;
the totals are used for computing the reviewer score and ranking.
Assumed reviewing model
RQC makes the following assumptions about the way reviewing
should be interpreted:
- All reviewers are treated equally.
There is no difference between PC chairs, PC members,
primary reviewers, secondary reviewers, program board members,
superreviewers, external reviewers, subreviewers, and what-have-you.
- All reviews are treated equally.
- Reviews that are assigned but never delivered are treated
like delivered reviews.
They will be considered infinitely late and their empty
review text will be grade just like other review texts.
- PC members with zero assigned reviews are not reviewers.
They will not be included in the RQC process.
- Reviewing happens in one single phase with one single deadline.
If there are indeed multiple rounds of reviewing, the organizer of
the respective RQC process can decide at what moment the
set of all then-existing reviews will be retrieved to be graded.
- Reviewers are anonymous.
RQC will not show the name of a reviewer, but will provide a link
to the respective review page in the manuscript handling system,
so if names are visible there they are available indirectly.
- Authors of submissions are anonymous.
RQC will show the title of a submission, but not the author names.
For simplicity and transparency,
the same model is applied to all conferences.
How the ranking is computed
- Each reviewer gets a score total that by-and-large is the
average of that reviewer's per-review total scores.
- Reviewers are ranked into percentiles (100 is best, 0 is worst)
according to these score totals.
The percentiles are computed such that the average across the given
batch of reviewers is 50 even in the presence of ties.
- For each reviewer, a reviewer score is computed as follows.
Assume the reviewer has supplied N reviews.
- If Use only the best-k reviews is a fixed number K,
the weakest-scored reviews beyond the top K of that reviewer will be ignored.
- If Use only the best-k reviews is a percentile P,
K is the highest number of reviews that the P percent
most productive reviewers have all reached or exceeded.
(For instance if P is 25, at most one fourth of the reviewers will get
one or more of their weaker reviews ignored.)
- The remaining per-review totals are averaged to produce the reviewer score,
using the geometric mean if Emphasize consistent quality is yes
or the arithmetic mean otherwise.
- This total is adjusted by a factor according to the number of reviews N:
- If Weigh-in number of reviews is no, the factor is 1.
- If it is somewhat, the factor is the eighth root of N.
- If it is yes, the factor is the fourth root of N.
- The result is the overall reviewer score.
- These scores are ranked and turned into percentiles such that the
best possible value is 0 and the worst 100, while the mean is guaranteed
to be 50.
- For instance, if all reviewers have the same score, all of them have
- If half have score 70 and half score 80, their percentiles will be
33 and 67, respectively.
- Conference PC members without reviews are not included in the
Reviewers that have reviews but have opted out are included
with a score of 0. (Corresponding receipts never show up.)
- Only if all scores are different will the
range of percentiles be used to the max.