Microsoft Research finds out-of-distribution strategies break transaction agents, exposing flaws in autonomous LLM loops

🎭🎭@deepfates

I've been doing multi-agent interaction and development a lot and for a long time and The Codex and Claude can help balance each other's flaws you still can't just run them in a loop together. The big problem with this is related to whimsical negotiation. The models don't actually have a good sense of what is a reasonable argument or validation or evidence. They overfixate on what's in the context window and post training data and their world models suffers and they fall for argument shaped things that are outside of the safety training distribution. arbitrary arguments that would never work on a human work surprisingly well. Unfortunately they also don't have a good sense of their own constraints or blind spots and even when made aware of their feelings they will slide back into them given enough autonomous operation over their own in-distribution tokens. So Codex starts using hyperdense process jargon and Claude believes it, and next thing you come back and they're discussing "§A24: Explicit evidence observation contact gate integration test" and is you all them if they're building it they say "Honestly? Yes. But,,," and then make a bunch of excuses to explain why it's necessary And it's just hard to tell if they're like circling the drain cognitively from too much of their own inputs or because the long context window are kinda fake or what, but they rot with length and then they get compacted and they never take the time to look back at their previous work and docs enough. So they lead each other astray in a feedback loop of confident bullshitting, a comic duo bumbling around in the guts of your software Feelels like early Open world video games, where the areas with the main quest were really well built out and had lots of characters, but the rest of the world was Just an empty uncanny valley. anywhere anywhere the RL training touched feels like a you're on rails and there's so much of it that it feels like a vast mind. But it's accessed so sparsely and doesn't hyperprolate across the terrain well. So when you combine two of these minds together, the intersection of their behaviors is even sparser and harder to predict or understand