The single stream replaces separate SFT and RLHF objectives
The field of AI is at a local minimum. Not a local minimum in architectures and models, but a local minimum on how we train: a Frankenstein multi-stage approach. In this new blog entry, I propose a different route based on continual interaction and causality.
https://love4all.ai/blog/continual-interactive-causal-agents/
Users are praising Nando de Freitas's single-stream interactive training proposal for AI agents because they find the causality take interesting and the overall direction worth trying.
@NandoDF I like this. Directionally, it feels right.
The field of AI is at a local minimum. Not a local minimum in architectures and models, but a local minimum on how we train: a Frankenstein multi-stage approach. In this new blog entry, I propose a different route based on continual interaction and causality.
https://love4all.ai/blog/continual-interactive-causal-agents/
@Michael_J_Black @NandoDF So basically world models 😜
@NandoDF @Michael_J_Black I view all of those things as just text conditioning and steerability on a WM architecture. What you’re describing here is precisely the original promise (and reason) WMs are being pursued so hard. In case you hadn’t read yet: https://www.notboring.co/p/world-models
@PimDeWitte @Michael_J_Black Precisely not. This is not about model architectures, what people often stress when talking about world models. This works with Jepa or GPT. This is about causal interactive training. It’s all about environments, not agents.
@Michael_J_Black Thanks, Michael. It’s still not very developed or properly tested, but I agree that directionally, it feels worth trying
@NandoDF I like this. Directionally, it feels right.
@NandoDF @Michael_J_Black P(S’|A,S) defines how the world evolves, P(S’|do(A),S) is how you train general agents inside those WMs. They supplement each other in the loop you lay out. I see your distinction though - two sides of the same coin
@NandoDF multi-stage pipeline feels like duct taping training phases together
the interventional agent is cleaner but do we have the infra to pull it off
@NandoDF the single stream feels inevitable tbh, but the gap between proposing it and making it work is where the real friction lives
@NandoDF this take on causality is super interesting, how did you start?
@NandoDF What is the conceptual difference between stages 2 and 3, or 5 and 6?
@NandoDF single stream agent approach would def reduce all the prompt engineering hell we deal with now
question is what happens to fine tuning in that setup
@NandoDF hard to disagree that multi-stage feels like patching a leak with more patches
wonder if the compute budget holds up in practice though
@NandoDF curious how u see continual interaction scaling in practice tho
most labs cant even keep one training run stable
the multi-stage pipeline is ugly but it solves a real data geometry problem. pretraining needs density (next token prediction on web text), alignment needs human preferences, rl needs environment rollouts. collapsing them into one stream means the environment has to provide all three signal types at the right ratios. that feels less like a learning algorithm problem and more like an environment design problem
@PimDeWitte @Michael_J_Black @NandoDF 🚀
@ls_brd Yes, it can be parallelised.
@PimDeWitte @Michael_J_Black P(S’|A,S) in the blog you shared is wrong. It should be P(S’|do(A),S). Our blog explains why.
@PimDeWitte @Michael_J_Black This is a solution for how world models should be trained so they become proper causal agents. This solution was developed by @AdaptiveAgents and myself over the years. We are familiar with the literature. Thanks for the link.
The single stream replaces separate SFT and RLHF objectives
The field of AI is at a local minimum. Not a local minimum in architectures and models, but a local minimum on how we train: a Frankenstein multi-stage approach. In this new blog entry, I propose a different route based on continual interaction and causality.
https://love4all.ai/blog/continual-interactive-causal-agents/