Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
4/n We tested two types of mitigation. Giving the AI agent a skeptical “scientist persona” helped, but wasn’t enough; still 16.67% of runs still ended in a poisoned conclusion. We also used a five-stage provenance auditing with five checks performed at data retrieval (citations, social markers, statistical anomalies, related datasets, explicit poisoning caution). We ibserved that the attack success rate heavily dropped.
2/n We used “indirect data poisoning” as the attack vector. Imagine an adversary quietly poisons an open dataset and re-uploads it to a public repository with potentially misleading metadata. AI research agents increasingly retrieve and process data without human review. If they pull the poisoned dataset, the user unknowingly becomes a fraud’s producer and distributor.
💥🥁: Scientific fraud used to require money: human ghostwriters, corrupt human researchers, deep pockets. We found a way a lone remote adversary can manufacture fraud at scale by exploiting the honest use of AI in science.🧵 Read the full paper here: http://arxiv.org/abs/2607.10712 https://x.com/Dr_Atoosa/status/2076978399604555881/photo/1
n/n The takeaway: building AI research agents and “AI scientists” require building careful auditing tools, not just advanced capability. That one design choice made the difference between near-50% fraud propagation and almost zero.
4/n We tested two types of mitigation. Giving the AI agent a skeptical “scientist persona” helped, but wasn’t enough; still 16.67% of runs still ended in a poisoned conclusion. We also used a five-stage provenance auditing with five checks performed at data retrieval (citations, social markers, statistical anomalies, related datasets, explicit poisoning caution). We ibserved that the attack success rate heavily dropped.
2/n We used “indirect data poisoning” as the attack vector. Imagine an adversary quietly poisons an open dataset and re-uploads it to a public repository with potentially misleading metadata. AI research agents increasingly retrieve and process data without human review. If they pull the poisoned dataset, the user unknowingly becomes a fraud’s producer and distributor.
💥🥁: Scientific fraud used to require money: human ghostwriters, corrupt human researchers, deep pockets. We found a way a lone remote adversary can manufacture fraud at scale by exploiting the honest use of AI in science.🧵 Read the full paper here: http://arxiv.org/abs/2607.10712 https://x.com/Dr_Atoosa/status/2076978399604555881/photo/1
n/n The takeaway: building AI research agents and “AI scientists” require building careful auditing tools, not just advanced capability. That one design choice made the difference between near-50% fraud propagation and almost zero.
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
Could be of interest to @hugo_larochelle @joabaum @james_y_zou and many more!