Announcement
PTRR balancing reportedly roughly halves ring-attention layer time
Luma Labs says its FlashAttention-4 kernel keeps softmax state in fp32 across ring steps, matching single-GPU outputs and gradients.
TLDR
Luma Labs says ring attention waits for its slowest rank at every step. Applying PTRR from PyTorch’s context parallelism to block-sparse costs brings the busiest rank within 0.1% of the rest and roughly halves layer time, it says. Luma Labs also says its online FlashAttention-4 kernel keeps softmax state in fp32 across ring steps instead of rounding partial outputs to bf16 each step, so outputs and gradients match a single GPU.
Combined views
—
2 Sources, first seen ago
