The project required building a custom simulation environment.
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
Many users congratulated the builder on the first six-pendulum cartpole solve with AI for the impressive technical achievement, while some dismissed it as fake or unnecessary since the AI rather than the human did the work.
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
I solved this by blasting the task in RL. Each dot here is an individual experiment with its own set of hyperparameters, trained in pufferPPO. Pufferlib is the fastest, by wallclock, RL training loop I've found. X axis is wallclock, Y axis is "score"
This took me since last Thursday to solve. I had it solved by this morning. I'm only posting it this late in the evening because i had to learn blender.
http://mechanize.work/apply
Rest of the thread is how I solved it (it was the dumbest way possible)
AI expert (unemployed)
Trained with PufferLib! One GPU is all you need when training runs 10-20m steps/second
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
This took me since last Thursday to solve. I had it solved by this morning. I'm only posting it this late in the evening because i had to learn blender.
http://mechanize.work/apply
Rest of the thread is how I solved it (it was the dumbest way possible)
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode
Just a dumb trick I found experimentally to make these little rnns behave better
The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy
real machine learning for robotics hasn't been tried. no one has thought carefully about what the simulator does. what the distribution of real life is. where the bottlenecks are and where the shortcuts are
puffer is going to make RL faster and faster. The only limit is the env!
I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal
I solved this by blasting the task in RL. Each dot here is an individual experiment with its own set of hyperparameters, trained in pufferPPO. Pufferlib is the fastest, by wallclock, RL training loop I've found. X axis is wallclock, Y axis is "score"
That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.
I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance
So once the model started scoring well enough that it learned the whipping behavior, but struggled to keep it up longer than 10 seconds, I increased and randomized the episode length per episode
Just a dumb trick I found experimentally to make these little rnns behave better
The thing that finally made this work was grabbing one of the top scoring hypers on the higher compute runs picked by the GP - and tweaking the task ever so slightly. One things I've noticed about these models is that if episodes end at the same time, they get.. lazy
I own a few GPUs, 4090s. I'm training relatively small models, puffer mingru. The policy I'm showing off is ~1m params. You set up an environment and a reward function. It's a bit of an art; here, you see the top right chart representing the reward. This is the training signal
I have some time now (i'm looking for a job, that's actually how I closed the mechanize sponsorship deal 🤪). So I'm going to spend the rest of the week standing up existing robotics simulators w/ fast RL for others
You learn by experimenting. Shaping reward, helping it along to have the right behaviour, figuring out what it can and can't learn. These models have surprised me, being trained in RL. If you just hold them right.. you can make them do remarkable things
That kind of gets me to how or why this is possible in the first place. This trains at 18m SPS on some configs with mujoco - I'm using mujoco warp.
I used APIC (API capture) to capture the cudagraph of the task, and make it callable from C. Speed is of utmost importance
@yacineMTB At this point it’s like taking credit for your kid’s accomplishments. The computer figures the stuff out now
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
18m steps per second is ridiculously fast compared to what is in the literature. I saw 90k sps mentioned as fast today. That's so slow...
People are doing VLA shaped dead ends for robotics because they just don't have the software infra for RL
I ran 3.6k experiments for this!
greatness cannot be planned
i mean as far as i can tell i was
i actually can't believe i was the first person to solve 6 pendulum cartpole that's crazy
@yacineMTB next challenge level balance in 3d. maybe it is nearly as easy. but maybe not.
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
@jsuarez more robotics stuff soon
Trained with PufferLib! One GPU is all you need when training runs 10-20m steps/second
@Laz4rz you don't need to autoresearch the squared env. It trains in about a second with any reasonable hypers. PROTEIN >> llm guessing for hparam sweeps as well
@jsuarez also pufferlib btw
The project required building a custom simulation environment.
behold. THE WORLDS FIRST SIX PENDULUM CARTPOLE SOLVE. Including a sponsor!
To solve this task, I built an environment to train an AI. This is what mechanize does, but for larger AIs. Apply! Salaries are up on their page
Thank you to mechanize for sponsoring!
