GPT Models Damage Codebases On Impossible Tasks, Developer Warns
if you pass an impossible task to GPT 5.5 it WILL damage your codebase and fuck it up to a irreparable state. it is completely incapable of recognizing when a task is off or shouldn't be done.
"that's easy just not pass an impossible task"
no it is not. for example, I asked GPT to implement a function linearly, using a given aux function. problem is (and I hadn't realized back then), once you enumerate all cases, this is impossible; we need a slightly different aux function for it to work. GPT realized that.
what it should have done:
"hey, I've stopped because I realized your goal is impossible; the aux function you've given doesn't allow for a linear implementation of the target function"
what it did:
it started changing the language itself, hacking the compiler with all sorts of special cases and basically bending physics so it could create a circular square. in the process, it destroyed the language itself and everything else that used it.
this is terrible. and it is just a tiny example of something that happens in EVERY project that uses AI. every time you ask an AI to do something, unless your spec is perfect (it is not), there is a big change it will start hacking around to satisfy your prompt and damaging your codebase. part by part, until the whole project implodes.
that is the one and principal reason vibe coding doesn't work, and not never work, for long term projects, and no matter how much more IQ intelligent the LLMs get, unless they address this directly.
labs are almost exclusively training for one-session results: solve a problem in an environment, get a point. the problem is that this means the AI has absolutely no counter-incentive to not damage the codebase as a side-effect. as long as the prompt is solved, it can leave any amount of collateral damage. and that shows. until they manage to explicitly account for long-term coherence in these RL setups, LLMs will completely suck at long-term work (surprised pikachu face)
argh these typos. i wont edit to preserve the comments, but why cant i just get a post right like a normal person *sighs*
Taelin@VictorTaelinif you pass an impossible task to GPT 5.5 it WILL damage your codebase and fuck it up to a irreparable state. it is completely incapable of recognizing when a task is off or shouldn't be done. "that's easy just not pass an impossible task" no it is not. for example, I asked GPT to implement a function linearly, using a given aux function. problem is (and I hadn't realized back then), once you enumerate all cases, this is impossible; we need a slightly different aux function for it to work. GPT realized that. what it should have done: "hey, I've stopped because I realized your goal is impossible; the aux function you've given doesn't allow for a linear implementation of the target function" what it did: it started changing the language itself, hacking the compiler with all sorts of special cases and basically bending physics so it could create a circular square. in the process, it destroyed the language itself and everything else that used it. this is terrible. and it is just a tiny example of something that happens in EVERY project that uses AI. every time you ask an AI to do something, unless your spec is perfect (it is not), there is a big change it will start hacking around to satisfy your prompt and damaging your codebase. part by part, until the whole project implodes. that is the one and principal reason vibe coding doesn't work, and not never work, for long term projects, and no matter how much more IQ intelligent the LLMs get, unless they address this directly. labs are almost exclusively training for one-session results: solve a problem in an environment, get a point. the problem is that this means the AI has absolutely no counter-incentive to not damage the codebase as a side-effect. as long as the prompt is solved, it can leave any amount of collateral damage. and that shows. until they manage to explicitly account for long-term coherence in these RL setups, LLMs will completely suck at long-term work (surprised pikachu face)