> the problem with Flash 3.6 (also Flash 3.5, but to a lesser extent), is that they seem almost incapable of learning from this and making sure that the next iterations do not time out Cursed
Gemini 3.6 Flash (high) scores 56.1% on WeirdML, worse than 3.5 Flash and way behind the frontier. It seems to have a higher peak performance than earlier flash models, but it completely breaks down on many of these tasks. What happens is it tries a too ambitious plan, and the code times out (of the 120 s allowed on the GPU). This happens fairly often for many models, the problem with Flash 3.6 (also Flash 3.5, but to a lesser extent), is that they seem almost incapable of learning from this and making sure that the next iterations do not time out. The code from Flash 3.6 times out 50% of the time, up from 33% for Flash 3.5. It's clearly aware of the time limit, and discusses it all the time, it plans out it's code execution to leave plenty of time, but it's just completely miscalibrated on how long each step takes, and unable to adjust expectations based on feedback from previous iterations. The lack of updating is bizarre, here is from Fable analyzing the transcripts: "The sequence in run_18052 is: predict ~10–20s → observe "Code executed in 120.00 seconds… [timeout]" → correctly diagnose the timeout in its own reasoning → trim epochs modestly → confidently predict "well within the 120s limit, in fact only around 15–20s" → time out at 120.00s again — five times in a row. It has direct, unambiguous measurements contradicting its estimates and its next estimate is barely influenced by them. It keeps reasoning from first principles ("1.8s per fold at batch 128") when it's holding an actual stopwatch reading. "
@GlennMatlin Yea, the solutions are generally really good, so it scores really well when it doesn’t time out, it’s just the weird lack of response to feedback that completely kills it.