Reactions from ranked influencers
4 posts
We stress-tested some AI detectors and found that they rarely flag human text as AI-generated. But asking LLMs to mimic a specific author causes detectors to misclassify text as human-generated ~13% of the time. For scientific writing, false negatives rose to ~26%.
You can sometimes fool Pangram by asking AI to imitate a specific author's style, but this is rather unreliable.
We stress-tested some AI detectors and found that they rarely flag human text as AI-generated. But asking LLMs to mimic a specific author causes detectors to misclassify text as human-generated ~13% of the time. For scientific writing, false negatives rose to ~26%.
Excited to have worked on this! AI detectors will continue to be super important in the slop-filled world we're heading towards. Overall super happy about the low FPR, and some takeaways in the thread: (1/n)
We stress-tested some AI detectors and found that they rarely flag human text as AI-generated. But asking LLMs to mimic a specific author causes detectors to misclassify text as human-generated ~13% of the time. For scientific writing, false negatives rose to ~26%.
SITUATION EXPLAINED: AI detectors have gotten really good, except at catching AI pretending to write like a specific human author. @EpochAIResearch tested three leading detectors, Pangram, GPTZero, and Originality, on both AI and human text • On plain AI-generated text from basic prompts, false negative rates were near zero, at most 0.7%, detectors are now genuinely good at catching AI, a sharp reversal from a few years ago • Originality had a 4% false positive rate on human writing, Pangram hit 0% false positives and 0% false negatives across 792 tested passages • The gap: all three detectors do noticeably worse when AI is specifically imitating a real human author's writing style rather than just generating from a basic prompt @theojaffee: "This is a total reversal from Claude 2 in 2023, which used to read as 100% human on GPTZero." @schisofrenia: "It must be so annoying to be a professor grading an essay these days, because there's so many subtle ways you can obviously AI your way through it."
We stress-tested some AI detectors and found that they rarely flag human text as AI-generated. But asking LLMs to mimic a specific author causes detectors to misclassify text as human-generated ~13% of the time. For scientific writing, false negatives rose to ~26%.