Reaction
Weight decay’s role in LoRA training
A user says weight decay in LoRA looks more like regularizing toward the base model’s original weights than penalizing absolute weight magnitudes.
TLDR
A user finds it funny that, in the LoRA setting, weight decay seems much closer to regularizing toward the base model’s original weights than to penalizing absolute weight magnitudes.
Combined views
—
1 Source, first seen ago
