Trained with PufferLib! One GPU is all you need when training runs 10-20m steps/second

real machine learning for robotics hasn't been tried. no one has thought carefully about what the simulator does. what the distribution of real life is. where the bottlenecks are and where the shortcuts are

puffer is going to make RL faster and faster. The only limit is the env!

So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode

Just a dumb trick I found experimentally to make these little rnns behave better

I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal

I have some time now (i'm looking for a job, that's actually how I closed the mechanize sponsorship deal 🤪). So I'm going to spend the rest of the week standing up existing robotics simulators w/ fast RL for others

That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.

I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance

The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy

You learn by experimenting. Shaping reward, helping it along to have the right behaviour, figuring out what it can and can't learn. These models have surprised me, being trained in RL. If you just hold them right.. you can make them do remarkable things

18m steps per second is ridiculously fast compared to what is in the literature. I saw 90k sps mentioned as fast today. That's so slow...

People are doing VLA shaped dead ends for robotics because they just don't have the software infra for RL

I ran 3.6k experiments for this!

greatness cannot be planned

@jsuarez more robotics stuff soon

@RichardSSutton @yoavgo hello professor. check out my bandwagon. it has 6 pendulums and it is very bitter

@Laz4rz you don't need to autoresearch the squared env. It trains in about a second with any reasonable hypers. PROTEIN >> llm guessing for hparam sweeps as well

@jsuarez also pufferlib btw

right but that problem is so easy that you can solve it with anything. So is cartpole. The minimum problem that has some signal is probably breakout. It trains in around 4 seconds. See if you can get it to solve in 2 w/ changes to the arch/alg, then see if those changes transfer to other envs!

@CarMarinkovic I've gotten drones to do backflips just waiting to be able to post it

@yacineMTB Will you expound on what happened to the indoor drone swarm? Even just a superficial update? I've been rooting for you, as many have, and I'm curious.

@yacineMTB lmao both of these tricks are ones I suggested (longer episodes + randomize episode lengths/stagger resets)

@yacineMTB no pufferlib, -1000000000000000 points

@CyborgLavery i have no idea lmfao