COLM area chair Yonatan Belinkov agreed, preferring human-written reviews.
as an AC: LLM assisted reviews are terrible. LLM assisted author responses are terrible. Even if the authors and reviewers do take the effort to carefully guide the LLMs so they reflect their points precisely (and not all do) the result is a wall of text that is just painful.
Users criticize LLM-assisted peer reviews for producing lengthy, polished walls of text that worsen signal-to-noise and overlook real academic problems.
really, the signal to noise ratio just became worse, even things are now phrased "nicely" (and at great length...) and you have tons of nicely crafted markdown tables with colorful ascii-art diagrams and bars. this is just exhausting and not helpful.
as an AC: LLM assisted reviews are terrible. LLM assisted author responses are terrible. Even if the authors and reviewers do take the effort to carefully guide the LLMs so they reflect their points precisely (and not all do) the result is a wall of text that is just painful.
maybe my mistake is that i try to read them without using an LLM. but then what is the point of all of this.
really, the signal to noise ratio just became worse, even things are now phrased "nicely" (and at great length...) and you have tons of nicely crafted markdown tables with colorful ascii-art diagrams and bars. this is just exhausting and not helpful.
The most useful reviews I got as an AC at COLM were clearly not written by an LLM. I just worry LLM reviewing will hill climb on this fuzzy usefulness notion at some point.
as an AC: LLM assisted reviews are terrible. LLM assisted author responses are terrible. Even if the authors and reviewers do take the effort to carefully guide the LLMs so they reflect their points precisely (and not all do) the result is a wall of text that is just painful.
Are you implying humans are better? That just proves how disconnected you are from what's actually happening in academia. Professors already reject genuine human work from students. That's exactly why students turned to LLMs in the first place. If human-assisted reviews and human-assisted responses were actually better, students wouldn't have needed LLMs at all.