LoRA Mechanism in PEFT Documentation

Low-Rank Weight Updates

The documentation explains that LoRA accelerates fine-tuning by representing weight updates with two smaller matrices via low-rank decomposition. These matrices adapt to new data while keeping changes low. The original weight matrix remains frozen and does not receive further adjustments. This mechanism drastically reduces trainable parameters, making the process more efficient for large models.

Frozen Weights and Portability

Because original pre-trained weights stay frozen, users can maintain multiple lightweight LoRA models for various downstream tasks. The adapted weights combine with original weights to produce final results. This approach allows portable adaptation without altering the base model. The documentation notes LoRA is orthogonal to other parameter-efficient methods, enabling combination with various techniques for broader applicability.