The Unjournal · evaluation data · version 2 · 16 October 2026

What the ratings tell us: scales, agreement, intervals, and what to change

A reanalysis of the numbers our evaluators give: 116 evaluations of 61 papers, roughly 1,000 ratings (public data, dashboard, earlier posts). We ask what the numbers mean, what patterns they show (including signs that evaluators use the scales differently or systematically favor some kinds of work), how much evaluators agree, whether their stated uncertainty matches that disagreement, and which questions are worth asking.

The goal behind all of this: ratings that readers can rely on and compare across evaluators and papers, with honest uncertainty, at a cost in evaluator time that is worth paying. Many of the patterns here could reflect genuine biases or just different readings of the scales; without experimental variation (randomizing who evaluates what, or how questions are asked) it is hard to tell these apart.

Main findings human-reviewed 16 Oct

The numbers are computed from our data; the wording and interpretation were drafted by AI. Each finding has a fold with how we got it. Dotted underlines are terms with a short definition on hover or tap.

1. Two experts rating the same paper typically differ by about 10 points on our 0 to 100 scales, and often by much more. On the overall rating, the median gap is 9.5 points, one pair in five differs by more than 20, and the largest gap is 40. On "open science" the median gap is 19.
   How we got this

   For each paper with two or three evaluations, we took every pair of evaluations and the absolute difference of their midpoints on each criterion (54 pairs for overall, 382 pairs pooled over eight criteria). Full table in section 4.

2. A single evaluation carries roughly half signal, half noise on overall and methods; much less signal on the other criteria. "Signal" here means differences between papers that evaluators agree on; "noise" means disagreement between evaluators about the same paper, measured by the intraclass correlation and Krippendorff's alphaBoth compare the spread of ratings between papers with the spread between evaluators of the same paper. 0 means evaluators agree no better than chance; 1 means perfect agreement. Defined with formulas in section 4.. Averaging our usual two evaluations raises the share of signal in the paper-level number (from about 45% to about 60% for overall), but for "advancing knowledge" and "open science" even the average is mostly noise.
   How we got this

   We estimated how much ratings vary between papers versus within a paper (between evaluators). For overall, the spread of evaluators around a paper's mean (about 11 points) is about the same as the spread of papers' means around the grand mean (about 10 points). "Mostly noise even after averaging" uses the same two components: if papers barely differ on a criterion once evaluator disagreement is removed, then averaging two evaluators still leaves a number dominated by disagreement. The basis for comparison is always the spread between papers; no outside truth is needed. Krippendorff's alpha and the intraclass correlationThe share of total rating variance that is due to real differences between papers, rather than to evaluators disagreeing about the same paper. 0.5 means half and half. are both about 0.45 for overall and 0.50 for methods, against 0.23 for advancing knowledge and 0.08 for open science. Details in section 4.

3. Evaluators' 90% intervals are narrower than the disagreement between evaluators would suggest, by roughly half. A co-evaluator's best guess lands inside an evaluator's 90% interval about half the time. If each evaluator's interval honestly reflected their own error, and both were aiming at the same thing, we would expect about 70%. Nearly a quarter of evaluator pairs give intervals that don't overlap at all. This is not the test the intervals were designed for.Evaluators were asked for a 90% interval around the "true" percentile, not to predict what another evaluator would say. We use the co-evaluator because there is no observable true percentile. If the two are judging slightly different things, the intervals can be honest and still miss each other.
   How we got this, and an important caveat

   We can't check intervals against "the truth", because there is no ground-truth percentile. What we can check is whether the intervals are consistent with how much evaluators disagree. Under a simple model (both evaluators aim at the same target and make independent errors whose size matches their own stated interval) the expected hit rate is about 70% for our data. The observed rate is 51%. The caveat: if evaluators are answering slightly different questions (different reference groups, different ideas of what "open science" means) the hit rate would also be low. We can't fully separate "too confident" from "measuring different things", but either way the published intervals understate how much a second expert would differ. Details in section 5.

4. Ratings sit high on the scale: 90% of overall ratings are above 50. This is partly expected, because we select promising and relevant work, so it is not by itself proof of leniency. There may also be some leniency: anonymous evaluators rate about 5 points lower than named ones on the same paper. But evaluators choose whether to be anonymous, so this could be selection rather than the effect of being named; a randomized encouragement to anonymity would be the way to find out.
   Why this is hard to interpret

   The percentile scale asks where a paper sits among "all serious research in the same area" the evaluator has seen recently. If we evaluated a random sample of such research, ratings would be spread evenly from 0 to 100. We don't: papers are suggested and selected. So a well-calibrated evaluator could reasonably give mostly 70s and 80s. To tell selection from leniency we'd need anchor papers that everyone rates. Section 3.

5. The overall rating moves mostly with methods, logic and communication, and advancing knowledge. Together these three account for about 84% of the explained variation in overall. Relevance to global priorities accounts for a modest share (roughly 10 to 15%) that it mostly shares with advancing knowledge; open science accounts for almost none.
   What "accounts for" means here

   We regressed each evaluator's overall rating on their own six sub-ratings (96 evaluations with all six). The sub-ratings explain 74% of the variation in overall. We split that 74% among the sub-ratings using a Shapley decompositionAdds the predictors to the regression in every possible order and averages how much each one raises R². It shares overlapping explanatory power fairly between correlated predictors, instead of giving it all to whichever one the regression happens to pick.. Relevance's unique contribution, after knowing the other ratings, is near zero and statistically indistinguishable from zero, but relevance does correlate with overall (about 0.45 to 0.5). Section 7.

6. The "should" and "will" journal-tier answers are often identical, but this may partly be a data artifact. 61% of evaluations entered in 2024 give the same number to both, against 25% of those entered in 2025. Both questions remain on the form, and we think both should stay.
   More

   On average, "should" (merit) and "will" (prediction) are almost the same number. The 2024 entries were mostly older evaluations transferred from PubPub, so we're checking whether one value was sometimes copied into the other. Section 6.

7. There are some fixable data problems. The three most recent evaluation packages (May to September 2026) are missing from the ratings data. Twelve ratings have an impossible interval (midpoint outside its own interval). Since 2025 the "real-world relevance" field almost always equals the "global relevance" field, which looks like an import artifact. And the "within-rater" correlations in our 2025 post partly pooled different people.
   Why this matters

   None of these is large, but some quietly bias analyses. Treating two copies of one field as two questions inflates their correlation, and grouping by "Evaluator 1" across papers mixes different people. Section 2.

8. Frontier LLMs agree with a single human evaluator about as well as a second human does on overall, methods and journal tier. They are not on the same scale as humans, though. GPT-5 Pro rates about 6 points higher and compresses its ratings; Claude Opus 4.6 rates about 8 lower. All the models we tested disagree with humans on open science.
   How we compared

   For papers with two human evaluations, we compared how well a model's rating tracks one randomly chosen human, with how well the other human does (rank correlation, averaged over 200 random choices). This is fairer than comparing a model with the human average, which is easier to track. Section 9.

The data, and problems in it

We use the public ratings export: per-paper evaluator labels ("Evaluator 1", "Evaluator 2" within each paper), so no one is identifiable and no one can be followed across papers. A few aggregate checks on repeat evaluators used the private database, with names hashed and only counts reported.

About 58% of evaluations are anonymous. Most papers have two evaluations; a few have one or three.

Problems we found

These were worth fixing before anything more sophisticated. Most were fixed on 16 October 2026 and the numbers on this page use the corrected data.

1. Missing recent evaluations (now fixed). When this analysis was first run, the three evaluation packages published since February 2026 had no ratings in the database: evaluations submitted through the new form were stored on our server but never copied into the ratings table. That step now runs nightly, and the 2026 ratings are included in the numbers on this page.
2. Impossible intervals (mostly fixed). Twelve ratings had the midpoint outside the evaluator's own interval or the bounds reversed; seven were data-entry errors and have been corrected against the published evaluations, and five match the published packages, so the error is in the original evaluation. Examples: overall 83 with interval [93, 100]; methods 75 with [88, 33]; a journal tier of 4.0 with interval [25, 5] (probably 2.5). The new form already enforces lower ≤ midpoint ≤ upper, so these come from older entry routes. Fifteen intervals have zero width (five of them "open science 100 [100, 100]").
3. Real-world relevance copied from global relevance. Within an evaluation, the two are identical in 22 of 48 cases entered in 2024, but in 41 of 43 entered in 2025 and all 7 in 2026. The current form has no real-world question, so the import seems to fill it from global relevance. Any analysis that treats them as two questions overstates how similar they are.
4. Duplicates and split titles (fixed). 28 rows were exact duplicates from re-entry batches, and two papers appeared under two slightly different titles with one evaluation under each. Both are now corrected in the database; the export also runs automated checks for these problems.
5. "Within-rater" correlations in the 2025 post. The 2025 post computed correlations across papers "within each evaluator", grouping by the evaluator label. Of the seven labels with three or more papers, two were the generic "Evaluator 1" and "Evaluator 2", each covering 14 papers rated by different people. They supplied 56% of the paper observations behind the reported figures.
   What the corrected numbers look like

   Restricting to the five evaluators who were named (and so genuinely the same person across papers), the headline correlations fall: real-world with global relevance from 0.69 to 0.56, methods with overall from 0.55 to 0.46, logic with overall from 0.55 to 0.47. The same pairs still lead. But this rests on five people with three to seven papers each, so it is very imprecise. The right fix is to run evaluator-level analysis privately with a stable hashed evaluator key, and publish only aggregates.

How evaluators use the scales

Three things stand out.

1. High placement. As discussed above, this mixes selection (we pick promising work) with possible leniency. The relevance questions are highest, which fits selection: we choose papers partly because they're relevant to global priorities.
2. HeapingAnswers bunching on round numbers. If 77% of ratings are multiples of 5, the scale is effectively used in 5-point steps, so a difference of 2 or 3 points between papers means little. on round numbers. Three-quarters of ratings are multiples of 5. That's normal for this kind of question; it tells us the practical resolution of the scale is about 5 points. Summaries shouldn't report decimals.
3. Evaluators do differentiate between criteria. Only 2 of 109 evaluations give the same number on every criterion; the median spread within an evaluation is 25 points.

Interval shapes

The median 90% interval is 20 points wide (10 to 40 for the middle 80% of intervals). Intervals get narrower as the midpoint rises: median width 30 for midpoints at or below 50, 20 for 50 to 85, and 11 above 85. Some of that is the ceiling at 100. Two-thirds of intervals are symmetric around the midpoint; when they aren't, the longer tail is usually below.

How much do evaluators agree?

The usual summary is a reliability coefficient such as Krippendorff's alpha (definition)α = 1 − (observed disagreement ÷ disagreement expected by chance). It handles missing ratings and different numbers of raters per paper. We use the "interval" version, which treats a 10-point gap as twice as bad as a 5-point gap., with labels like "0.667 is acceptable". These labels hide more than they show. Here are three more concrete ways to read the same numbers.

Reading 1: how much of chance-level disagreement is left

Alpha is defined as one minus the ratio of observed disagreement to the disagreement you'd expect if ratings were shuffled at random across papers. So 1 − α is the share of chance-level disagreement that remains. That makes differences between alphas easier to compare.

Reading 2: the gap in percentile points

This is the most concrete reading. It answers "if I asked a second expert, how far off would they typically be?"

The median gap is about 10 points almost everywhere; what differs between criteria is the tail. On advancing knowledge, one pair in five is more than 30 points apart (the largest gap is 65), which suggests evaluators sometimes read the question quite differently. One likely reason: the guidance packs novelty, practical value, relevance and usefulness to other researchers into one slider.

Reading 3: signal and noise, and what averaging buys

If we think of each rating as the paper's true standing plus this evaluator's error, a reliability of r means true differences between papers account for a share r of the variation and evaluator error for 1 − r. At 0.5 they're equal. This is a model, not a mechanical translation of alpha, but for our data the ICCIntraclass correlation: the share of rating variance due to differences between papers. Estimated here from a one-way random-effects model; numerically close to alpha for our data. and alpha come out almost identical.

The useful consequence: the number we publish is usually the average of two evaluations, and averaging cancels some independent error. The formula is r_k = k·r / (1 + (k − 1)·r).

How this compares with journal and grant peer review

Our agreement on the overall rating looks similar to or somewhat higher than what's typically found for journal referees and grant reviewers. Our weaker criteria are about typical. These comparisons are rough, because studies use different statistics and different things are being rated.

Do the 90% intervals match the disagreement between evaluators?

We ask evaluators for a 90% credible intervalA range the evaluator believes has a 90% chance of containing the "true" value. Here, the guidelines define the true value as the percentile they'd assign "with complete evidence, knowledge, and wisdom". on each rating. There's no observable true percentile to check these against. So comparing with another evaluator's midpoint is not literally a calibration test. We'll call it cross-evaluator coverage: how often does evaluator B's best guess fall inside evaluator A's 90% interval? Evaluators were not asked to predict each other, so a low number here is not a failure on their part. It tells us whether the published intervals convey how much a second expert might differ, which is what a reader would want them for.

On its own, a coverage rate doesn't tell you much, because even perfectly calibrated evaluators wouldn't cover each other 90% of the time. So we compare it with what we'd expect if the intervals matched the evaluators' actual error.

Wider intervals do cover more: 67% for intervals wider than 30 points against 37% for 10 points or less. So the widths carry some information. But an evaluator's own interval width barely predicts how far they end up from their co-evaluator (rank correlation 0.13). There's a little self-knowledge in the intervals, not much.

This is what the research on overconfidence would predict. In many studies, people's stated 90% intervals contain the true answer only 30 to 60% of the time, and experts do only slightly better than novices (McKenzie, Liersch & Yaniv 2008; Moore & Healy 2008). Intervals are often less than half as wide as they should be (Soll & Klayman 2004). Our test is different (a co-evaluator's guess, not a known answer), but "about half as wide as warranted" fits that pattern. Our evaluators look about as overconfident as experts usually are, not unusually so.

The two journal-tier questions

We ask where a paper should be published on merit (as if journals judged fairly on the stated criteria) and where it will be published. Both are on the form now and we think both should stay. The contrast is the point: it lets evaluators say "this deserves better than it will get" or the reverse, and "will" is the one rating that eventually resolves.

In the data, the two are close. Means are 3.8 (should) and 3.9 (will), and the correlation is 0.73. 44% of evaluations give exactly the same midpoint, and 22% the same midpoint and interval.

The drop from 61% to 25% identical is large enough to suspect the transfer, not the evaluators. We're checking the 2024 rows against the published evaluations. If evaluators really do give different answers more often than the 2024 data suggest, the distinction is doing more work than it appears.

Do the "will" predictions come true?

Early evidence says somewhat. For 13 evaluated papers now published in journals that we could place on the 0–5 tier scale, the average predicted tier has a rank correlation of 0.71 with the actual tier. Predictions were slightly optimistic on average (+0.2 tiers; mean absolute error 0.5). 11 of the 18 evaluator intervals (61%) contained the actual tier. Papers already published in top journals had been predicted at 4.1 on average, against 3.7 for those still unpublished.

What the overall rating is built from

The question. When an evaluator gives a paper its overall rating, how closely does it track their own ratings on the other criteria?

The method. For the 96 evaluations that rated overall and all six older sub-criteria, we regressed overall on the six sub-ratings. Together they explain 74% of the variation in overall (R² = 0.74). To split that 74% among the six, we used a Shapley decomposition, which shares the explanatory power of correlated predictors fairly instead of letting the regression arbitrarily favor one.

What this means.

• Overall moves most with methods, then logic and advancing knowledge. Those three carry about 84% of what the sub-ratings explain.
• The two relevance criteria do correlate with overall (about 0.45 to 0.5) and carry 10 to 15% of the explained share. But once you know a paper's methods, logic and advancing-knowledge ratings, its relevance ratings tell you little more about its overall rating. Relevance overlaps strongly with advancing knowledge, which already asks about global priorities.
• The opposite signs for the two relevance coefficients don't mean anything; they're nearly the same variable (see the copying issue above), and the regression can't separate them. Combining them into one relevance index gives a unique effect of 0.03 [−0.14, 0.19].
• Open science barely relates to overall at all.

Is that what we want? Yes. The overall rating is meant to be a judgment of research quality and contribution. Importance for global priorities is mainly something The Unjournal filters for when choosing which papers to evaluate; it isn't the main thing we ask evaluators to judge. So the pattern above is roughly what we'd hope for. Readers who want the relevance judgment can see it as a separate rating.

Which questions are worth asking

One useful frame is to sort each question by two properties: is it measured reliably, and does it tell us something the other questions don't? The table below is the AI's reading of where each question falls, with the evidence; the suggestions in the last column are AI-generated and not yet decisions.

A caveat on all of this: "adds something distinct" here means "isn't predicted by the other ratings". The real test of usefulness is whether a question helps predict something we care about (later publication, citations, replication, policy use, or whether readers find it useful). We don't yet have enough outcomes to run that test except for journal placement.

How LLM ratings compare

Our separate LLM evaluation project asked several models to rate the same papers with the same rubric. We re-used those ratings for 47 matched papers.

The fair comparison: for papers with two human evaluations, how well does the model track one randomly chosen human, compared with how well the other human tracks that same person?

The two frontier models track a single human at least as well as a second human does on overall, methods, advancing knowledge and the "will" tier. The cheaper models don't. The samples are small (about 30 to 40 papers), so read these as "comparable", not "better".

Two cautions:

• Different scale use. GPT-5 Pro rates about 6 points above humans on overall and spreads its ratings less (standard deviation 10 against about 14 for humans). Claude Opus 4.6 rates about 8 below with a human-like spread. Mixing raw LLM numbers with human ones would distort averages. Adding GPT-5 Pro as a third rater actually lowers alpha because of this offset.
• Open science. No model's ranking of papers matches the humans', and the two frontier models rate it 17 to 20 points lower. Models reading a PDF can't easily check data and code availability, and the humans don't agree with each other either.

One promising use: show the cases where both humans agree and the model strongly disagrees, or where the humans disagree and the model sides with one. Those cases may teach more about the rubric than another correlation.

Possible changes (AI-generated suggestions)

These suggestions were drafted by AI (GPT-5 Pro and Claude) from the analysis above. We're listing them because they seem reasonable, not because we've decided on them; we'd like more human views before treating any as a priority. Two items marked "in progress" were simple data and form fixes we've already started. The rest are ordered roughly by the AI's estimate of value for effort. Tags: easy fix form change test first.

Data pipeline

1. easy fix Copy approved form submissions into the ratings table (the three 2026 packages are missing), standardize criterion names at import, and stop filling real-world relevance from global relevance. In progress.
2. easy fix Automated checks on every data refresh: impossible intervals, duplicates, near-duplicate titles, fields copied into other fields, heaping, interval widths of zero, and a rubric-version field so structural missingness isn't mistaken for nonresponse.
3. easy fix Keep a stable hashed evaluator key in the private data so evaluator-level questions (are some evaluators consistently harsher?) can be answered internally. Publish only aggregates.

Evaluation form

4. form change Next to each slider, show how much experts typically disagree on that criterion ("evaluators of the same paper usually differ by about 10 points, and one pair in five by more than 20"). Warn when an interval is narrower than 10 points or has zero width. Block impossible intervals and tier bounds outside 0–5. A first version of these hints and warnings is now on the evaluation form (October 2026); it still needs human review.
5. form change Replace the open-science percentile with a short checklist of verifiable items.
6. form change Sharpen the "should vs will" wording: for example, ask "will" first, then "if journals judged only on merit, where should it go?"
7. form change Tighten the claims/logic boundary; consider splitting advancing knowledge.
8. test first Anchor papers for calibration practice: a small set rated by many evaluators, with the reference distribution shown. The current calibration tool compares with a uniform distribution, which is only right for a random sample of papers.
9. test first Whether intervals on every criterion are worth the effort. One option: intervals only for overall, methods and the "will" tier, with a simple low/medium/high confidence elsewhere. Compare completion time, missing answers and reliability.
10. test first As one option for the "will" tier: probabilities over tiers 0 to 5 instead of a midpoint and interval. That would let us score forecasts properly once papers are published. It's more demanding to answer, so it's worth trying on a subset first.
11. test first Ask the sub-criteria before overall, so overall is a synthesis rather than an anchor. Easy to randomize.

Reporting

12. easy fix Report reliability in percentile points ("experts typically differ by about 10 points") next to any coefficient, and drop the "acceptable/good" labels.
13. easy fix Show overall and global-priorities relevance side by side rather than as one summary number.
14. easy fix Round published averages to the nearest 5, and indicate that a two-evaluator average on overall has a standard error of roughly 8 points from evaluator noise alone.
15. test first A standardized LLM rating as a separate, clearly labeled signal, adjusted for the model's offset, never averaged in raw.
16. test first A short "why do the evaluators disagree?" note from the evaluation manager when a pair is far apart (different reference group, different reading of the criterion, different evidence, different priorities). This turns disagreement into information about the form.

Open questions

We'd especially welcome comments on these.

1. Our current view is that "overall" should stay a judgment of research quality and contribution, since we filter for global-priorities importance when selecting papers. Does the guidance to evaluators make that clear enough, and should the global-relevance rating be framed more explicitly as a separate, secondary question?
2. How much should we ask of evaluators? Every extra question and interval costs time; which ones are worth it?
3. Can we get a few evaluators to rate shared anchor papers, so we can separate selection from leniency?
4. Is "advancing knowledge" one question or two?
5. Are named and anonymous evaluators' ratings comparable? On the same paper, anonymous evaluators rate about 5 points lower. That rests on about 15 papers with one of each, so it needs more data, but if it holds it matters for averages.
6. Which outcomes should we use to judge which questions are useful: journal placement, citations, replications, policy citations, reader feedback?

Sources and how this was made