Researchers Warn Against Unchecked AI Agent Coding In Science

@vsbuffalo This is one of the reasons I tell people being a prof is great training to work with agents (and vice versa). Being good at either requires getting good at learning how to ask the right questions to test/verify results confidently presented to you by someone else.

@vsbuffalo Case in point. This interaction reminds me so much of the kind of push back I'd need to make with a first year grad student who hadn't learned to clearly communicate the difference between "an answer that sounds plausible" and "an answer I've tested" yet.

@vsbuffalo I've been spending the past 3 months trying to solve that problem and my conclusion is basically that you have to treat them like CPUs and write the "code" to write code (bit like scripts that write script files to run). Otherwise, they always shit the bed, they're SWE tools.

@vsbuffalo Every paper should have a at least one red team coauthor who tries to destroy the results prior to a publication

@vsbuffalo I've been experimenting building reproducible papers using Snakemake, code and templates

https://github.com/SACGF/vibepaper

@vsbuffalo Just ask Claude Opus 4.8 on ultracode to look over your work before you show it to anyone.

Something like this would get caught easily.

@vsbuffalo @cremieuxrecueil you get a sense for each model's limits, push beyond those and it starts taking short cuts

gotta break the problem down into manageable pieces, and/or switch to an agent with a fresh context window

@vsbuffalo i will vibe code my rendering pipeline into absolute RUIN

@arjunrajlab What’s your workflow with Mycelium to catch these sort of issues?

@vsbuffalo AI explains "actually I did eat the cookies I wasn't supposed ro" like a three year old but with the vocabulary of CS Lewis.

@vsbuffalo Also, in science you can't have them run stuff you don't understand. In SWE stuff you sometimes can since they can figure it out cuz code crashes, but in science the "code crashes" is you realizing things are wrong, if you don't know, there's no epistemic gate.