This matches my recent experience with AI agents as research assistants. Amazing at coding, sub-good-masters-student at navigating the space of ideas and updating based on experimental results.

@alex_peys But did you tell it not to make any mistakes?

@alex_peys yay for us nature idea enjoyers

@cs_serdar they are fine for some things, if i was trying to squeeze the last 5% out of my architecture by throwing known tricks at the wall and seeing what sticks this is totally doable by a coding agent.

for figuring out that one of my training datasets has a subtle issue, not so much

@alex_peys I fell into the autoresearch hype today and tried to get codex to do some basic stuff like run a default training run and eval it on some pipeline. The models are simply not good enough for agentic research, they do plenty of stupid things. This is with 5.5.

@init_malachi maybe codex needs a /touch_grass command

@alex_peys hmm - over a year ago I did this using a manually scripted loop + gemini (for the big context window). I defined a goal (ie decrease loss) and fed in the logs of prev runs in the context. it tweaked the hyperparams and architecture.

and it worked AMAZING!

this happens to me anytime im working on new frameworks or trying to get an output that's not easily feasible I put 30mins everyday on walking to think i get all the solutions in this period, write it on a note and codex woulda wasted tokens although pushing through but not getting it and imma just give it to implement

i gave it a prompt - search and think deeply as long as possible, bro didn't use 2 mins and just followed current methods with patches

@alex_peys It's definitely not useless, especially if the environment is set up perfectly. But it's prone to making slight mistakes it won't notice and blame other things/the experiment itself. The more "agentic" the workflow, the more likely it is for the result to be garbage.

@jjschultz

@alex_peys Can you share exactly what was the problem and what was your idea? If you can, that is.

@alex_peys Def not researcher but implementer yes

@alex_peys there's no real ceiling on that $400, you set the goal and find out what it cost when you get back from the hike

In my mind, /goal exists to make sure the agent actually completes the task. There are many situations where models regularly fail:

1. Tell it to do 10 things. It will do some, say “I’ll do rest next”. Use /goal to get it to do all 10. 2. Tell it to do something repeatedly. It will stop without finishing. /goal makes sure it finishes. e.g.

@alex_peys This is a horrible way to use goal. Goal iss not for generating ideas, use gpt 5.5 xhigh or better gpt 5.5 pro for that. Goal is for implementation. You don't even have the minimum knowledge of how these things work and yet your making claims about their capabilities, ironic.

@alex_peys I create this autoresearch skill for improving my bio related models, it has help me a lot and has a lot of guardrails for agents to keep them in check. You can try it https://github.com/yashraj59/autoresearch-bio

@stanislavfort That matches my experience too. Agents are much better when the task has a feedback loop: code runs, tests pass or fail, logs explain what happened. Open-ended research still needs a human steering the taste.

@alex_peys Maybe it’s a you thing

@alex_peys sounds within critical range of many pseud attractors for it. sincerity and slop often entangle when projected