Reaction
Weight decay’s role in LoRA training
A user says weight decay in LoRA looks more like regularizing toward the base model’s original weights than penalizing absolute weight magnitudes.
TLDR
A user finds it funny that, in the LoRA setting, weight decay seems much closer to regularizing toward the base model’s original weights than to penalizing absolute weight magnitudes.
Combined views
2.1K
1 Source, first seen ago
49 likes