Armin Ronacher reports Claude tool-calling regression on out-of-distribution inputs, highlighting potential issues with over-tuning

@CompleteSkeptic I would love to read it in case you do get around!

this is a smell of too much on-policy RL - I was considering a longer write up on this, but it might be too technical for most audiences! this is also why it's been so hard to transfer good engineering practices to LLMs (decomposition, abstraction, state management, etc.)

@jeffreycider @CompleteSkeptic either overfitting or reward hacking

@CompleteSkeptic @willccbb what is too much on-policy RL 👀

@mitsuhiko I'm touched! (big fan) I will add it to my todo list 🙏

@xeophon bettog off distribution hurts in production every time

wonder what changes armins hinting at

@CompleteSkeptic if you write it I would be very happy to read it!

@willccbb @jeffreycider overfitting can be a bit of a catch-all for optimization issues e.g. with GPT-3.5-turbo and chess: too much focus on a narrow area can cause intelligence to be more jagged/fractured, though the model may not have overfit (in the traditional sense) to chess

@CompleteSkeptic I would be interested in this and your thoughts on the relationship with those engineering practices.