Developer perumaal_s open-sources a PufferAI-compatible reinforcement learning environment for building and training 2D games

Publishing my RL env to build 2D games and train them using @puffer_ai . (link below)

I had a lot of fun working on this RL game env + pufferlib; this concludes my quest into RL + eng/research over the last year.

All of this is mostly experimental/prototype RL env (read: crappy) code that I used

(a) to make pufferlib faster with lots of low-level optimizations (b) to build a realistic env to understand and play with RL as it's an awesome branch of science (and something that is not already LLM'ified) (c) to try out advanced concepts like self-play/curriculum learning etc (caveat: not an actual researcher, so I don't fully understanding them).

Here is my code: https://github.com/rlplays/rlplays (MIT licensed).

If you find this useful, drop me a note (caveat: I am no longer working on this, so can't support/help!) + support/donate to the awesome OSS projects above.

This RL env includes an editor to build individual blocks in C++, have them show up in an editor to build levels/maps, and be instantly RL trainable with no specialization.

The code for a single block looks like this in C++. The block then shows up in the editor; you can move/size it around, twiddle properties, build levels with it. The RL pipeline does not need additional code/obs to know about this "brick" block (it's automatic). Similar ethos to Spelunky 2, where blocks are self-contained God blocks.

Fun thing is you can use an LLM to create blocks. Prompt: "Create an enemy that moves left and right and if it sees the player within a radius, it jumps towards and attacks them. The player can also jump and squash this character."

(Built with pufferlib @jsuarez raylib @raysan5 imgui @ocornut and @KenneyNL CC0 assets. Please support these OSS, they are all awesome.)

This is what it looks like during the RL training itself

(I did not use much LLMs for the actual code though, mostly for the blocks/characters/enemies) Here is another fun one that attacks the player when they are under this triggering block:

The RL env supports some 'advanced' features such as curriculum learning (ramp up difficulty with many levels in a syllabus starting from simple to more complex) so you would see a hill climbing like this

Or "create a hovering enemy that bobs around and attacks the player when they are nearby". I think LLMs can even reason about the level and create them but haven't tried them yet.

Putting this together, here is a complete level that is RL trained with an agent you can play against yourself:

It's pretty easy to start: just pull the docker image with the code in it and build/run the editor; mess with blocks/levels; then train with RL and play with it.. takes around ~2 hours with pufferlib (3.0) on a 2080 RTX with a dozen simple+complex levels.

@jsuarez always cool seeing someone take an implementation detail that deep and turn it into real protocol design