Authors of seemingly low-quality TMLR submissions reportedly did not understand their own papers

@JessicaHullmanSad but not surprising. It’s pretty obvious that at this point in time, heavily AI-generated papers are less likely to have had a competent author at the helm. This is why I have little patience for anything Pangram says is mostly AI generated.

@danish037Sad but also unsurprising for people paying attention to deluge of AI slop in research submissions. I've heard people sharing that no one would stake their reputation on work they don't understand or can't verify, but for many people their reputations don't exist (just yet). Also, worth noting that disincentives/penalties for such actions are nowhere commensurate with the inappropriateness of the act.

@zetalyraeWho uses AI to write papers? This type of people:

@Dr_AtoosaPeer-review venues, education institutes, and universities face a flood of AI-assisted/AI-generated submissions lacking sufficient human oversight, making author names an unreliable signal of expertise. 3/11

@Dr_AtoosagreCAPTCHA aims to measure a construct of "the capacity to verify": we define it as the domain knowledge and reasoning skills required to critically evaluate and defend one's own contributions. 5/11

@sethlazarRT @zetalyrae: Who uses AI to write papers? This type of people:

@eigenhectorIronically the ML paper "Attention is all you need" gave rose to AI that sucks up all available attention

@tallinzen@anmarasovic Yes, I did this once for a neurips submission that ruined my life for a day and decided never again. There’s a 10% chance AI will kill us by 2029, gotta allocate your effort judiciously!

@tallinzenThere are also intermediate cases where I do try to justify why I'm not going to dig into the details. Here's something I wrote for a recent review: "the statistical analysis is nonstandard and apparently quite complex. This makes it hard to evaluate, and is quite unfortunate because the field has converged on a relatively small number of analyses and I don't see the motivation for a bespoke analysis here. There is a long appendix that describes this analysis but I am not required or inclined to read it. I have certainly had that experience myself in projects where my collaborators have asked a coding agent to analyze a dataset. I'm afraid this doesn't really inspire confidence in the results."

@PMinervini@tetraduzione yeah but it was over zoom, you can just have an LLM transcribe the Qs and generate the answers based on the paper in real-time -- in-person it's trickier

@PMinervini@tetraduzione yeah because they didn't use LLMs to generate the answers, but they could have done that if they wanted (some organisations perfected the technique and do that at scale, e.g. https://en.wikipedia.org/wiki/North_Korean_remote_worker_scheme)

@sethlazarThis is precisely right. As a track chair at Neurips, I am seeing cases where the papers were written by LLMs, reports likewise, rebuttals too, even in some cases Area Chairs joining the flood. It becomes a potemkin village where nobody is actually accountable for anything that they are saying, and neither individual understanding nor our collective epistemic projects make *any* progress at all.

@tetraduzione@PMinervini that would have made sense if the answers were good. the answers were poor right? even if they used LLMs with those answers clearly implies they don't know what they are talking about

@PMinervini@tetraduzione yeah but if they used LLMs they have would been harder/impossible to spot

@Dr_Atoosa@sethlazar We have cooked an evaluator tool, Seth. Would love to chat to test it for larger experiments: note that Nihar who ran the TMLR experiment is a co-author of this work:

@joelbot3000Poetic but sad: The thing we create (AI) contributes to perverting the spirit of what created it (the science of AI). I'm not a fatalist -- we'll find ways to adapt science; but I do think there's an allegory here we don't absorb fully enough, which is the general need to invest early in better immune systems. In particular, we gotta recognize when the tail (a tool serving an important purpose) starts to wag the dog (the purpose itself), which is when things distort in bizarre ways. This rhymes w/ MacIntyre's practices/institutions, McGilchrist's Master & Emissary, Goodhart's law, or Courtwright's Limbic Capitalism. One of my fav books I've read on this is "A Time To Build" by Yuval Levin -- that the purposes of good institutions is to help form us and for us to then improve that institution. But the general trend is for us instead to view institutions as platforms for our success, which leads to a race to the bottom. Or put in the words of Gus from The Wire: "The pond is shrinking, the fish are nervous. Get some profile, win a prize. Maybe find a bigger pond somewhere." Which to me captures the tragedy of the commons in ML research driving the Potemkin village described below.

Authors of seemingly low-quality TMLR submissions reportedly did not understand their own papers

TLDR

Combined views

Authors of seemingly low-quality TMLR submissions reportedly did not understand their own papers

16 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

16 Sources