ReLoRA Enables Full-Rank Fine-Tuning Via Repeated Low-Rank Merges

ReLoRA is such a great trick You repeatedly train a LoRA, merge it, then train a new LoRA on top of that Each LoRA is only a low rank update, but a sum of low rank updates can be a full rank update, so you can get better results For DFT, I used a total of 13