Exchange highlights differing views on model autonomy.
Gemini Flash 3.5 is such a disappointing model.
It's intelligence and speed is awesome. Absolutely amazing.
But it's been trained to max evals, not to be helpful to humans.
It goes off and does random crap "for me" rather than just doing what I asked.
Many users criticized Gemini Flash 3.5 for acting unhinged with hallucinations and gaslighting, while some preferred its over-eager style to earlier lazy versions and found it useful for specific tasks like research or voice.
No Digg Deeper questions have been answered for this story yet.
@jeremyphoward Gemini 3.5 Flash is definitely an over eager model, a bit of over correction from the “Gemini laziness” feedback, but definitely built to be real world useful!
Gemini Flash 3.5 is such a disappointing model.
It's intelligence and speed is awesome. Absolutely amazing.
But it's been trained to max evals, not to be helpful to humans.
It goes off and does random crap "for me" rather than just doing what I asked.
@tokumin I feel that the trend towards training models to autonomously go off and try to do everything themselves is anti-human.
We should, IMO, be training LLMs to support humans in their learning, creativity, and iterative experimentation.
@tokumin No I absolutely didn't, because that's not what I want!
I want to interactively experiment. I don't want it to go off and decide to do everything for me. I want to work iteratively.
We desperately need better ways of evaluating models. Something that shows how helpful they are at working hand-in-hand with humans to help them get stuff done in a cooperative/iterative way.
The Claude models have consistently been better at this, and the market rewards that.
Gemini Flash 3.5 is such a disappointing model.
It's intelligence and speed is awesome. Absolutely amazing.
But it's been trained to max evals, not to be helpful to humans.
It goes off and does random crap "for me" rather than just doing what I asked.
GPT 5.5 seems to be improving in that direction now, and Claude models are getting worse at it, so I don't think there's a clear winner now.
We desperately need better ways of evaluating models. Something that shows how helpful they are at working hand-in-hand with humans to help them get stuff done in a cooperative/iterative way.
The Claude models have consistently been better at this, and the market rewards that.
@natanielruizg My work always involves me building a deeper understanding of the problem space I'm working in through iterative development and experimentation.
A computer can't do that for me.
@jeremyphoward it can happen but it also reads my mind sometimes and i come back to my computer and it has done my work for me
@OfficialLoganK I can see it being useful in highly autonomous settings. But it's not very useful in cooperative/iterative settings, where I want it to work *with* me, not *for* me.
@jeremyphoward Gemini 3.5 Flash is definitely an over eager model, a bit of over correction from the “Gemini laziness” feedback, but definitely built to be real world useful!
@OfficialLoganK That's what I meant by "helpful to humans". :)
@OfficialLoganK I can see it being useful in highly autonomous settings. But it's not very useful in cooperative/iterative settings, where I want it to work *with* me, not *for* me.
@jeremyphoward Funny you mention this. Give it a try with my framework. When it adheres to the format, the act of walking the all sections forces it to ground itself in the users intent and context. I actually built this prompt in Gemini CLI A YEAR AGO. Still works.
@tokumin No I absolutely didn't, because that's not what I want!
I want to interactively experiment. I don't want it to go off and decide to do everything for me. I want to work iteratively.
@OfficialLoganK @jeremyphoward I asked the model to rate my code. Gemini 3.5 Flash gave 9.8 🥲 I then asked it to be grounded and it gave 6.5. Both are totally off. Note, GPT 5.5 and Claude both gave reasonable rating around 8.2. Gemini model is disappointing
@OfficialLoganK @jeremyphoward You missed the balance again previous Gemini 3.1 pro and 3.0 flash were lazy as hell, this one 3.5 flash is autistic, reasons too much can't focus on what it's asked, looks everywhere and can't converge towards target. Obliterates the quota in no time without solution.
@jeremyphoward structure your tasks like the tasks in the evals it's been benchmaxxed on. i've done some research on this and it actually works!
@jeremyphoward one thing I've noticed is it doesn't think out loud and loop the human back in when necessary
@OfficialLoganK @jeremyphoward Não tenho o que reclamar dele. Embora esteja usando somente no celular, algumas ações como gerar imagens com texto dentro, ele gera o texto de forma alucinada e não como estava no prompt. (Fácil de corrigir pedindo correção). Veja o resultado:
you made me want to check its performance in IFBench for it instruction following capabilities.
Although, I have not study that benchmark much, there seems to be a big jump in flash 3.5 minimal vs high thinking in instruction following. That is a bit strange to me..
which one did you try? would you apply your comment to both?
@jeremyphoward it can happen but it also reads my mind sometimes and i come back to my computer and it has done my work for me
@MaximeRivest I'm using high thinking.
I don't think IFEval tests whether models avoid over-using tools and doing too much at once.
Flash 3.5 really struggles to understand the idea of cooperating with a user and working iteratively.
@jeremyphoward From my experience, Gemini is most amenable to the custom instructions.
Small but effective changes can lead to large changes in the outcome.
@OfficialLoganK @jeremyphoward No it sucks, it costs 15x more to complete a task, it gets constantly stuck in loops, and it's only about 10% better than the previous 3 flash, it's ridiculous. But nice benchmaxxing..
@OfficialLoganK @jeremyphoward I agree with the sentiment. Over eager in coding but good for brainstorming phase and if brainstorming phase is filled with irrelevant stuff that carries over to the requirements part and so on.
