End-to-end neural networks racing drones in Abu Dhabi! 🚁

Check out the drone racing team from Delft University of Technology!

A completely end-to-end neural network solution, from pixels to direct motor commands.

There are no Kalman filters, and no computer vision feature detectors.

As they nicely put it in their article: "Just neurons flying the drone."

The challenge is extreme. These drones fly at high speeds and need split-second decisions with minimal onboard resources: a single rolling-shutter camera and an IMU.

Their approach is called SkyDreamer, based on the Dreamer-v3 reinforcement learning algorithm.

First, a world model is trained in simulation. Then, the neural network learns how to fly in its dreams through reinforcement learning. The network's internal state can be read out to see where it thinks it is on the track or how fast it's going.

Even better, the drone estimates some of its own body characteristics during flight, like the camera angle relative to the body, eliminating time-consuming manual calibration.

The system uses only a single camera and the gyros from the IMU, ignoring the accelerometers, just like human FPV pilots do.

Read more here & video source: https://mavlab.tudelft.nl/dronerace-defeating-champions/

~~

♻️ Join the weekly robotics newsletter, and never miss any news → http://ziegler.substack.com