It's interesting to see @MicrosoftAI uses ray actors not just for controller and rollout workers but problem workers for the posting training of the MAI-Thinking-1 model. Instead of introducing third party dependency like @modal for sandboxing, Ray actors could provide finer granularity and control for heterogeneous compute which could translate to better utilization of the unused CPU resources in the GPU cluster and easier communication of the agent execution results. Also the part of work I did was to support @sgl_project with @raydistributed backend to better support RL infra especially in weight syncing.