The project required building a custom simulation environment.
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
Many users expressed excitement and pride over the builder's first six-pendulum cartpole solve using AI, while a few dismissed the achievement by questioning its novelty or crediting the computer instead.
This took me since last Thursday to solve. I had it solved by this morning. I'm only posting it this late in the evening because i had to learn blender.
http://mechanize.work/apply
Rest of the thread is how I solved it (it was the dumbest way possible)
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
I solved this by blasting the task in RL. Each dot here is an individual experiment with its own set of hyperparameters, trained in pufferPPO. Pufferlib is the fastest, by wallclock, RL training loop I've found. X axis is wallclock, Y axis is "score"
This took me since last Thursday to solve. I had it solved by this morning. I'm only posting it this late in the evening because i had to learn blender.
http://mechanize.work/apply
Rest of the thread is how I solved it (it was the dumbest way possible)
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode
Just a dumb trick I found experimentally to make these little rnns behave better
The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy
real machine learning for robotics hasn't been tried. no one has thought carefully about what the simulator does. what the distribution of real life is. where the bottlenecks are and where the shortcuts are
puffer is going to make RL faster and faster. The only limit is the env!
I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal
I solved this by blasting the task in RL. Each dot here is an individual experiment with its own set of hyperparameters, trained in pufferPPO. Pufferlib is the fastest, by wallclock, RL training loop I've found. X axis is wallclock, Y axis is "score"
That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.
I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance
So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode
Just a dumb trick I found experimentally to make these little rnns behave better
You learn by experimenting. Shaping reward, helping it along to have the right behaviour, figuring out what it can and can't learn. These models have surprised me, being trained in RL. If you just hold them right.. you can make them do remarkable things
That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.
I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance
I have some time now (i'm looking for a job, that's actually how I closed the mechanize sponsorship deal 🤪). So I'm going to spend the rest of the week standing up existing robotics simulators w/ fast RL for others
The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy
I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal
i mean as far as i can tell i was
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
@yacineMTB At this point it’s like taking credit for your kid’s accomplishments. The computer figures the stuff out now
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
18m steps per second is ridiculously fast compared to what is in the literature. I saw 90k sps mentioned as fast today. That's so slow...
People are doing VLA shaped dead ends for robotics because they just don't have the software infra for RL
I ran 3.6k experiments for this!
@yacineMTB Congrats At this point you should just be a streamer, work yourself up to giving away autographed GPUs
@yacineMTB holy shit
@yacineMTB sim2real time
@yacineMTB no pufferlib, -1000000000000000 points
@yacineMTB I was so right
@yacineMTB Holy shit, yacine shipped
@yacineMTB When I grow up I wanna be like u
The project required building a custom simulation environment.
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
