VIEWS255
Desmond Elliott@delliott
@megamor2 Can you say more about the issues you are seeing in the LLM-generated reviews?
23hViews 255Likes 1
Tel Aviv University's Mor Geva calls for banning LLM-generated peer reviews, arguing they produce disorganized and unusable feedback
Critics say automated reviews suffer from fluff and hallucinations.
1210310610.3K
Original post unavailable.
Sentiment
Users criticize LLM-generated academic peer reviews for being fluffy with little real information plus hallucinations and overstated flaws, preferring to train human PhD students instead.
Pos
0.0%
Neg
100.0%
2 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
LIKES4
Mor Geva@megamor2
Reviews can seem very detailed but in practice there's little information there. The summary is often full of fluff so it's really just hard to understand what the paper is about. They often provide long lists of issues, and it's very hard to understand if the concerns are major or minor. They often make non realistic or nonsensical suggestions. All this without mentioning mistakes and hallucinations, which are often conveyed with such confidence that can bias the whole evaluation. The final scores are often borderline so no information there as well.
As author, it creates lots of work which is either stupid or not feasible. As AC it's a nightmare, I want to get a concrete evaluation to work with and I get this noisy not informative text. In rebuttals things become a joke.. the person who generated the review has no idea how to judge what's going on and so typically you get a short statement like "I read the response and decided to keep my score".
21hViews 107Likes 4
RETWEETS1
Sarah Wiegreffe@sarahwiegreffe
@ipeirotis @TuhinChakr @megamor2 But ACs typically can see who reviewers are and thus know when a reviewer is junior and factor that in (I'd like to think junior reviewers also generally give themselves lower confidence scores). With undisclosed LLM use, that's not the case.
6hViews 135Likes 2
REPLIES1
Tuhin Chakrabarty@TuhinChakr
@ipeirotis @megamor2 Lol we should train them instead of thinking AI is the solution ;) I am doing my part training my first PhD students
15hViews 54
Panos Ipeirotis@ipeirotis
@TuhinChakr @megamor2 We have the same problem with reviews that come from junior faculty and PhD students 😉.
15hViews 230
Taha 🫡@taha_moji
@megamor2 What is the most insightful public discussion notes or conversations in this topic?
21hViews 72
Panos Ipeirotis@ipeirotis
In general, junior people tend to be much stricter and return reviews that are more “nitpicky”.
Editors can handle such “nitpick objections relatively easily, especially for journals. For sure much easier to deal with a picky LLM who will not be hurt by having their objections overruled.
13hViews 37
Rishu Kumar@rishdotuk
I recently reviewed for a workshop, and the other two reviews on the paper I was assigned seem kinda oblivious to some of the shortfalls that popped out to me. I don't know if they used LLMs, because both of the reviews read quite similar-ish.
Admittedly, I am not an experienced reviewer, and it's very likely that one of the things I pointed out was obvious, but it still felt kinda funky.
14hViews 37
Tuhin Chakrabarty@TuhinChakr
@megamor2 @delliott @ipeirotis
17hViews 10