Detectors like Pangram struggle to identify AI-assisted fluency edits.
Tuhin Chakrabarty#1053
@TuhinChakr I don't disagree. I can imagine a process in which the LLM writes every sentence but I am still fully in control of the content. However, this seems extremely inefficient compared to me doing much of it myself, given how sensitive I am to word choice when expressing a position.
Users criticize AI detectors for academic papers as unreliable because they flag AI-edited writing as 100% AI-generated.
This idea, that the source of ideas matters less than their substance, is not a good argument for conferences permitting the use of AI to write papers.
First, if the source doesn't matter, then credit shouldn't matter either, so why should the person prompting the AI be seeking the credit associated with peer reviewed publication? Just get the "ideas" out, by all means. If you want more attention, club together with the other folks who like prompting AI generated papers, and make journals, conferences, blogs, whatever you like. But don't submit it to a conference track that explicitly says not to use LLMs to write the final paper.
Second, if the manner of presentation doesn't matter—who cares whether it's AI or human?—then why are they using AI to write their paper in the first place? Just write it the way you wrote it six months ago, when LLMs weren't capable enough to substitute for your labour. You're not going to lose points for style; if you can't write clearly then that's a problem you should fix not cover up.
Third, according to these folks, every prompt is a special snowflake and deserves respect. Just think of all these AI-generated gems lying among the detritus rejected by the track chairs... But the reality is that by far the biggest reason for people using AI to write/finish their papers is to save them time, and because *they're* saving time they're submitting crap that reviewers then waste time identifying as such.
Using AI to write the final paper is just laziness that imposes costs on the rest of the community. This doesn't rule out all the ways in which one can use AI agents to do better research—there are many; by far the least interesting and least valuable is making yourself into a ventriloquist's puppet and claiming ownership over words put in your mouth by someone else.
We shouldn't reject papers for AI-assistance alone. But we SHOULD desk-reject A LOT MORE papers.
Many reviewers make >$200/hr consulting. Our failure to respect their time means it's difficult to get reviewers to engage.
@ipeirotis @TuhinChakr Using AI to make your writing more concise or fluent doesn't necessarily mean Pangram will detect it as AI-generated.
Ultimately I think we'll need more AI in review, but at present problem seems to be not enough people to judge the content of the papers getting submitted.
It does classify my AI-edited writing as 100% AI.
Or you can ask an LLM to generate real slop, add a couple of edits by deleting a word here and there and Pangrams detects it as 100% human.
It is a very thin layer of detection of shallow stylistic choices by LLMs.
Academics should absolutely refuse to use that dumpster fire of a tool.
Yes spray & pray authors are now using AI, but the worst will learn to get around Pangram, and the core problem will go unsolved.
