OLORA+: A HYBRID APPROACH TO LOW-RANK ADAPTATION (DIGITAL TRANSFORMATION AND ARTIFICIAL INTELLIGENCE, 2025)
This paper introduces and analyzes OLoRA+, a novel parameter-efficient fine-tuning method that enhances Low-Rank Adaptation (LoRA) by combining the orthonormal initialization of OLoRA with the differential learning rate optimization of LoRA+. The study compared the performance of OLoRA+ against the standard OLoRA baseline on the TinyLlama-1.1B-Chat-v1.0 model. Using a subset of the alpaca dataset, a collection of 52K instruction-following demonstrations, with 1000 samples for training and 500 for testing, model performance was assessed using evaluation loss, BLEU, and ROUGE metrics. The obtained results show that OLoRA+ consistently outperforms the OLoRA baseline across all considered metrics. Additionally, the study reveals that OLoRA+ is effective with learning rate ratios both greater and less than one, uncovering a novel trade-off between "Refinement" and "Exploration" learning strategies. This confirms OLoRA+'s potential as a more versatile and powerful approach for LLM adaptation under limited resource conditions.
Keywords: NLP; PEFT; LLM; LoRA; OLoRA; LoRA+