behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!

To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page

Thank you to mechanize for sponsoring!

Oh no .... I better solve 6 pendulums fast before it's too late and someone else does

This took me since last Thursday to solve. I had it solved by this morning. I'm only posting it this late in the evening because i had to learn blender.

http://mechanize.work/apply

Rest of the thread is how I solved it (it was the dumbest way possible)

i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy

@yacineMTB You famous because I make you be famous. Never forget who the mother for your famous be.

real machine learning for robotics hasn't been tried. no one has thought carefully about what the simulator does. what the distribution of real life is. where the bottlenecks are and where the shortcuts are

puffer is going to make RL faster and faster. The only limit is the env!

So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode

Just a dumb trick I found experimentally to make these little rnns behave better

I solved this by blasting the task in RL. Each dot here is an individual experiment with its own set of hyperparameters, trained in pufferPPO. Pufferlib is the fastest, by wallclock, RL training loop I've found. X axis is wallclock, Y axis is "score"

I have some time now (i'm looking for a job, that's actually how I closed the mechanize sponsorship deal 🤪). So I'm going to spend the rest of the week standing up existing robotics simulators w/ fast RL for others

i mean as far as i can tell i was

That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.

I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance

I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal

@yacineMTB I swear you just went from 180 to 300 in like a month

You learn by experimenting. Shaping reward, helping it along to have the right behaviour, figuring out what it can and can't learn. These models have surprised me, being trained in RL. If you just hold them right.. you can make them do remarkable things

@yacineMTB At this point it’s like taking credit for your kid’s accomplishments. The computer figures the stuff out now

The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy

18m steps per second is ridiculously fast compared to what is in the literature. I saw 90k sps mentioned as fast today. That's so slow...

People are doing VLA shaped dead ends for robotics because they just don't have the software infra for RL

I ran 3.6k experiments for this!

@yacineMTB what does solving pendulums mean?

@yacineMTB the real challenge is getting back to 600

bet you cant do it

@yacineMTB congrats nice job

5 pendulums is impressive