What is LoRA?

LoRA stands for Low-Rank Adaptation. It freezes the pretrained model weights and inserts small trainable matrices into each transformer layer. These matrices have low rank, meaning few parameters, and their product is added to the original weights at inference. LoRA achieves results similar to full fine-tuning while training only a tiny fraction of the parameters.