Review quality definition for ABCD 2030
Copyright: The content of this file is in the Public Domain (CC-0). Everybody is allowed to use it for any purpose in its original or any modified form.
This is an RQdef file (review quality definition file). See www.reviewqualitycollector.org for what that means.
-	The file will be analyzed paragraph-by-paragraph. 
-	Only some paragraphs are relevant: Those beginning with "Review quality definition", "Part", "Aspect", "Facet", "Helptext:", those ending with "(… points)", and the one-line settings paragraphs in part II.
-	Other paragraphs are irrelevant and will simply be ignored if they start with "comment:" or if they occur before the first part heading (like the present ones do). Other types of paragraphs are not allowed.
-	Relevant paragraphs have a specific format you need to keep intact.
-	What you can change: the numbers in weight, localweight, and points entries; the names given inside the quotes in Facet headings, the text of the items within a facet (within which, but not across which, you can also apply boldface and italics), the part behind the colon in the helptext entries and in the part II settings.
-	You can and should also insert or remove whole Facets or items within Facets.
-	", commentbox" after an Aspect weight means the grader will be asked for a textual comment on that aspect as well. You can remove that.
-	The helptext paragraphs are made available to the grader via a help icon.
-	You cannot change the set of Aspects (or their order) nor the set of settings (or their order).
-	In order to reduce the number of rating ties, all weights must be different and odd and all localweights within any one aspect must be different and odd.
-	RQC only supports English (and most but perhaps not all characters of other Latin alphabets).
That said, let's start the actual review quality definition:
Part I: Aspect definitions
Aspect "Helpfulness for Authors" (weight 19)
Helptext: This aspect is solely interested in how well the review aids the authors for improving their work and/or writing. Whether the review makes a good judgment regarding acceptance of the submission plays no role here whatsoever.
Facet "Is constructive" (localweight 77)
Helptext: Whether the review really provides help, as opposed to merely criticism.
•	The review offers suggestions or ideas for how to make the actual improvements for all or nearly all points of substantial criticism (100 points)
•	For much, but not nearly all criticism (50 points)
•	Only rarely or never (0 points)
Facet "Provides evidence" (localweight 57)
Helptext: Whether the review explains why its criticism holds.
•	The review provides justification or good explanation for all or nearly all points of substantial criticism in such a form that average authors are likely able to accept the criticism as valid (100 points)
•	For much, but not nearly all criticism (50 points)
•	Only rarely or never (0 points)
Facet "Uses a friendly tone" (localweight 55)
Helptext: Whether the review treats the authors gently, so they are treated like the human beings they are and so they are (more) willing to accept any criticism brought forth.
•	The review uses a friendly or polite tone throughout and phrases all its criticism gently. (100 points)
•	With some exceptions (50 points)
•	Very often not (0 points)
Aspect "Helpfulness for Decision" (weight 27, commentbox)
Helptext: This aspect should be evaluated regardless of how useful the review will be as feedback to the authors.
The only relevant criterion is how well the PC chairs and/or the co-reviewers of the same article will be aided in making a good, balanced acceptance/rejection decision.
Facet "Is thorough in breadth" (localweight 37)
•	The review discusses all aspects of the article worth discussing instead of only some of them (100 points)
•	It ignores one or two relevant aspects (66 points)
•	The review selectively discusses only a modest fraction of the relevant aspects (33 points)
•	The review discusses hardly anything of relevance (0 points)
Facet "Criticism is specific" (localweight 57)
Helptext: Whether the review explains where exactly a criticism applies.
•	For its criticism, the review is always specific what it applies to and where in the article this is to be found (100 points)
•	It is often or usually specific (50 points)
•	Only rarely or never (0 points)
Facet "Criticism is justified" (localweight 59)
Helptext: Whether the review explains where exactly a criticism applies and why it holds.
•	For its criticism, the review always provides reasoning or argumentation why a particular point is problematic and how: incomplete, illogical, inappropriate for the purpose, inconsistent with something else, etc. (100 points)
•	The justification is missing for at least one important point of criticism (50 points)
•	The justification is frequently missing (0 points)
Facet "Weighs its items" (localweight 51)
•	The review visibly allocates more weight to important mentioned points of criticism and praise than to unimportant points (100 points)
•	Does so for either criticism or praise, but not both (66 points)
•	Does so only vaguely (33 points)
•	Does not visibly weigh (0 points)
Facet "Provides criticism and praise" (localweight 27)
•	The review mentions several non-trivial points of criticism and of praise, not only one of these (100 points)
•	Does so for either criticism or praise, but not both (66 points)
•	Does so only vaguely (33 points)
•	Provides little substance at all (0 points)
Aspect "Graded Co-Reviews" (weight 49)
comment: This aspect has no facets. 
The points value is the percentage of co-reviews this reviewer has graded (or 100 if there were none to be graded, because all co-reviewers opted out or reviewers are not even asked to grade at this conference or journal).
Therefore, all well-behaving participants will always have 100 here; yet not grading co-reviews disturbs the process a lot, so this aspect should (somewhat counter-intuitively) have a large weight, probably the largest of all. 
The only exception would be if the organizers are fully willing to stand in for even many reviewers and treat grading by reviewers as non-mandatory even for reviewers that do not opt out.
Aspect "Timeliness" (weight 47)
comment: Just like before, any well-behaved reviewer will end up with the same score here: 100.
So just like before, use a rather large weight to reflect the damage any lower values represent for the reviewing process as a whole.
•	at least 2 days early (100 points)
•	at least 6 hours early (98 points)
•	at least 1 hour early (97 points)
•	at least 0 hours early (96 points)
•	at most 1 hour late (80 points)
•	at most 6 hours late (70 points)
•	at most 1 day late (50 points)
•	otherwise(0 point)

Part II: Settings
Show D before A: yes
comment: yes means Aspect D should be shown textually before A during grading. no means they are shown in the order given above.
Use only the best reviews: 75 percentile
comment: If a reviewer provided particularly many reviews, this option can be used to allow a few that are less good without the reviewer's ranking going down. This is done by taking into account only the K best of the reviews of each reviewer (or fewer if only fewer exist), called the relevant reviews. 
This option determines the value of K, either as a fixed number or preferably as a percentile of the total number of reviews per reviewer. Recommended values are 75 percentile (the default), 67 percentile, or perhaps 50 percentile. Fixed values can be useful in particular at conferences (as opposed to journals). For example, you might set 5 if most reviewers did 4 or 5 reviews, but a few did 6 or 7. 
To turn this mechanism off (and count in all reviews of each reviewer), set it to 100 percentile or to a large fixed value such as 999.
Emphasize consistent quality: no
comment: RQC will usually use the arithmetic mean of the total scores of the relevant reviews for the ranking. If you want to prefer reviewers that always provide good-quality reviews over reviewers that sway between very good and mediocre, you can set this to yes (rather than the default and recommended no) and RQC will use the geometric mean instead, which penalizes variability. Note that a single relevant review graded zero (in particular a missing review) will make the geometric mean zero. This is overly brutal, so RQC will adjust zero ratings to 1 if geometric mean is used. Example: For a reviewer with 6 relevant reviews with ratings 100, 90, 85, 80, 70, 0 the outcome will be 71 for 'no', but only 40 for 'yes', so beware!
Weigh-in number of reviews: somewhat
comment: If the numbers of reviews is very different between your reviewers, it appears reasonable to favor high-volume reviewers somewhat. The extreme position would be to simply add the ratings up instead of averaging them. This would favor volume far over quality, which is against the idea of RQC. The other extreme is just using the average rating as is. You can do that by setting this option to no. In between are two ways of increasing the rating average by a factor depending on the number N of reviews (limited to k if the only-the-best option is active): You can use the fourth root of N by setting this to yes or the eighth root of N by setting it to somewhat, the recommended value. Examples: If the number of reviews is 1, 2, 5, or 10, respectively, 'somewhat' will produce weighting factors of about 1, 1.1, 1.2, 1.3, respectively, and 'yes' will produce weighting factors of about 1, 1.2, 1.5, 1.8, respectively. ('no' will always result in a factor of 1, independent of N).
comment: end of review quality definition