LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

Researchers have introduced LoRA-GA^2, a new method designed to close the performance gap between standard Low-Rank Adaptation and full model fine-tuning. By implementing multi-step gradient adaptive alignment, the technique aims to improve the accuracy of large models while maintaining the lower memory requirements typical of parameter-efficient training.