Tuhin Chakrabarty proposes hybrid tools to standardize evaluations.
@auyonomous They should be integrated to address weaknesses in human review like non-uniformity. But some roles that review has traditionally played, like eliciting judgments about what people in a field find most meaningful or training the next generation are not so obviously substitutable.
Users express optimism about LLMs assisting peer review, expecting current limitations like mode collapse and lack of diversity to prove temporary while liking the idea of AI as regularization.
No Digg Deeper questions have been answered for this story yet.
@1a3orn @lu_sichu Notably, there's so much public data out there for peer review, rebuttals, meta-review, paper revisions, etc.; which is a big reason (together with the relatively short task horizon) why I expect peer review to get automated pretty early.
@BogdanIonutCir2 @lu_sichu Of course LLMs are more diligent and have a more comprehensive view of background literature, etc.
Maybe you don't want them in a role analogous to peer review -- either a step before PR, or just being used by peers, or something else, I'm not sure.
@1a3orn @lu_sichu I expect this is probably temporary too, and might have something to do with RL-like mode collapse. I think serious efforts to automate peer review have barely started, so my expectation is that peer-review-specialized-LLMs will get much beter as the required schlep gets done.
@1a3orn @lu_sichu My highest uncertainty is about robustness to silly mistakes a la hallucinations; it might take a long time for LLMs to be as robust as the best human reviewers, without losing on diversity.
@JessicaHullman I do like the idea of AI-reviewer-as-regularization
