Tsinghua's You Jiacheng argues PufferLib counters a decade of reinforcement learning stagnation by running simulations up to 10,000 times faster

kache@yacineMTB

@jsuarez yes

A little perspective: RL as a field spent 10 years making algorithms slower and slower. If you look at the original ALE, it actually can sim a few thousand frames per second per core. If you look at some of the last big env releases before a ton of people moved over to LLMs, you'll find several at dozens to hundreds of steps per second with such bad engineering that they don't even scale with vectorization.

The field did this exactly because they presumed they would have to train directly in the real world. In reality, what we got out of this is a bunch of brittle off-pol and model-based algorithms that burn a ton of compute and don't work outside of the benchmarks shown in the original pubs. There's a clear gap between on-pol and other methods. You don't simply switch and scale up compute to save data. You have to spend a TON more compute to match the perf of on-pol, and then you spend even more compute to gain in sample efficiency.

Our whole core realization with PufferLib is that we can write good sims for a lot of problems 10000x faster. Good doesn't even mean accurate. It means accurate enough with domain randomization and other tricks that our agents can implicitly sysid their current setting and act robustly. So far, this has worked across several different industries. I'd love to give examples here, but this is unfortunately where exact client details get confidential. We need to be better about negotiating publicity, and we're starting to do that as Puffer gets bigger.

Another major flaw with slower and slower algorithms is that the core research loop also gets slower and slower. We sim mazes and 2048 at 10+m steps per second. Big deal right, those are easy. Wrong: algorithmic improvements on those envs have consistently predicted performance improvement on every single env in our test suite. Without this, we wouldn't have been able to release so many core breakthroughs in the last 2 years with a grand total of ~20 GPUs. We ran 20,000 experiments on ~12 of them in the 3 weeks leading up to Puffer 4 launch. At traditional speeds, it would have taken Google scale compute and an infra team.

So no, we're not going to step the real world at 20m sps, but assuming that matters (or at least that it is the only thing that matters) is where the field went wrong. /rant.

6h99712

BOOKMARKS5LIKES11REPLIES2

Joseph Suarez 🐡@jsuarez

We're not primarily robotics. PufferLib has been used professionally in finance, commerce, gaming, defense, and animation. Has worked great. The world model stuff straight up doesn't work. Some surrounding lit is borderline scientific fraud. I believe it works better where you have a good amount of static data to train it like in robotics. But in our case, we simply won't need to scale up because we can hit the throughputs of the largest RL projects from OAI/DM on a single node, and it seems like the models can be much smaller than previously expected

2d732115

RETWEETS29

Joseph Suarez 🐡@jsuarez

A little perspective: RL as a field spent 10 years making algorithms slower and slower. If you look at the original ALE, it actually can sim a few thousand frames per second per core. If you look at some of the last big env releases before a ton of people moved over to LLMs, you'll find several at dozens to hundreds of steps per second with such bad engineering that they don't even scale with vectorization.

The field did this exactly because they presumed they would have to train directly in the real world. In reality, what we got out of this is a bunch of brittle off-pol and model-based algorithms that burn a ton of compute and don't work outside of the benchmarks shown in the original pubs. There's a clear gap between on-pol and other methods. You don't simply switch and scale up compute to save data. You have to spend a TON more compute to match the perf of on-pol, and then you spend even more compute to gain in sample efficiency.

Our whole core realization with PufferLib is that we can write good sims for a lot of problems 10000x faster. Good doesn't even mean accurate. It means accurate enough with domain randomization and other tricks that our agents can implicitly sysid their current setting and act robustly. So far, this has worked across several different industries. I'd love to give examples here, but this is unfortunately where exact client details get confidential. We need to be better about negotiating publicity, and we're starting to do that as Puffer gets bigger.

Another major flaw with slower and slower algorithms is that the core research loop also gets slower and slower. We sim mazes and 2048 at 10+m steps per second. Big deal right, those are easy. Wrong: algorithmic improvements on those envs have consistently predicted performance improvement on every single env in our test suite. Without this, we wouldn't have been able to release so many core breakthroughs in the last 2 years with a grand total of ~20 GPUs. We ran 20,000 experiments on ~12 of them in the 3 weeks leading up to Puffer 4 launch. At traditional speeds, it would have taken Google scale compute and an infra team.

So no, we're not going to step the real world at 20m sps, but assuming that matters (or at least that it is the only thing that matters) is where the field went wrong. /rant.

2d47.5K421266

An Eevee@rw_eevee

Appreciate your perspective and yes, this is a largely accurate history of the field. We’ve been all-in on fast sim + DR + RNN-style networks for the past ~2 years at least. It works great and we’ve gotten some spectacular results.

But sadly some problems have refused to yield to sim as easily as locomotion. That’s why every company suddenly declared world models the new hype. It’s supposed to be a totally general AI-based sim. You can at least put it on thousands of GPUs to get 20 million sps if you want.

Will it work? idk maybe. VLAs kinda remain a joke. Hand-crafted sims aren’t accurate or diverse enough yet. Offline RL has never worked once in history. But you have to put your eggs in some basket 🤷‍♂️

2d84561

Joseph Suarez 🐡@jsuarez

@rw_eevee If you have your own fast rnn pipeline, try Muon and retune if you're still on Adam. Was a major boost. Our other things are harder to integrate. PufferNet rocks but needs kernels

2d29561

An Eevee@rw_eevee

@jsuarez Love it. You’re doing great work man 🫡

2d2872

jaisel@jaiselsingh

im very curious/ looking into more work that combines a model based planner w/ a policy (insp in part by guided policy search from levine). im curious as to why that's not as big of a research direction these days.

i think seeing @yacineMTB has nerdsniped me into wanting to run mujoco exp w/ pufferlib lol

15h2781

Zengineering@Samhanknr

@jsuarez I would love to understand in very very broad strokes what industrial applications this has. I’m trying to figure out how and why to learn puffer lib but it’s not easy to figure how it’s used in the real world.

1d199

Sciumo@SciumoInc

@jsuarez “In reality, what we got out of this is a bunch of brittle off-pol and model-based algorithms that burn a ton of compute and don't work”

A bit verbose for a tshirt, but still worthy

2d2921

gfodor.id@gfodor

@jsuarez @rw_eevee as a total RL outsider, but someone who just started dabbling with pufferlib - what's the best technique for leveraging the existing VR content ecosystem for this? i have been working on OpenXR for a while now and can't help but think there's a huge overhang from this.

2d901

Guilherme O'Tina@guilhermeotina

@jsuarez the same pattern is showing up in LLM agent eval now. everyone building multi-step benchmarks that take an hour to run. you end up optimizing for the benchmarks ontology instead of the actual task. RL already learned cheap fast eval loops beat elaborate ones

2d277

Joseph Suarez 🐡@jsuarez

@jaiselsingh @yacineMTB model based is the 2nd least promising direction for my style of work after off-pol. Terrible trade off with compute

7h271

Zengineering@Samhanknr

@jsuarez Like what’s an example of a task in a typical tech company it can do

1d54

Kekko D’Amato@kekkodamato_

The '20k experiments on 12 GPUs' number is the real proof. Fast-enough sims with DR don't just lower the hardware bar — they close the research loop so you can actually iterate. The field optimized for benchmark scores in expensive envs when the bottleneck was always iteration speed.

1d33

EB1A Experts@eb1aexperts

@jsuarez Interesting perspective on RL's evolution. A fast feedback loop is often an underrated advantage in research. The ability to run thousands of experiments can sometimes matter more than squeezing out every bit of efficiency from a single run.

1d32

Dalton@Daltonagray

@jsuarez @rw_eevee Which type of problems to WMs not work on? Comma say they're using WMs now. (Big fan of puffer btw.)

1d31

RARE@gunahneyd

@jsuarez Have y'all noticed that? Automatic delivery

2d10