OLORA+: A HYBRID APPROACH TO LOW-RANK ADAPTATION

Davronov Rifkat Rakhimovich; Kushmuratov Samariddin Ibodulla ugli

Authors

Davronov Rifkat Rakhimovich V.I.Romanovskiy Institute of Mathematics, Uzbekistan Academy of Sciences
Kushmuratov Samariddin Ibodulla ugli V.I.Romanovskiy Institute of Mathematics, Uzbekistan Academy of Sciences

Keywords:

NLP, PEFT, LLM, LoRA, OLoRA, LoRA+

Abstract

This paper introduces and analyzes OLoRA+, a novel parameter-efficient fine-tuning method that enhances Low-Rank Adaptation (LoRA) by combining the orthonormal initialization of OLoRA with the differential learning rate optimization of LoRA+. The study compared the performance of OLoRA+ against the standard OLoRA baseline on the TinyLlama-1.1B-Chat-v1.0 model. Using a subset of the alpaca dataset, a collection of 52K instruction-following demonstrations, with 1000 samples for training and 500 for testing, model performance was assessed using evaluation loss, BLEU, and ROUGE metrics. The obtained results show that OLoRA+ consistently outperforms the OLoRA baseline across all considered metrics. Additionally, the study reveals that OLoRA+ is effective with learning rate ratios both greater and less than one, uncovering a novel trade-off between "Refinement" and "Exploration" learning strategies. This confirms OLoRA+'s potential as a more versatile and powerful approach for LLM adaptation under limited resource conditions.

References

1. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., Liang, P. (2021). On the Opportunities and Risks of Foundation Models. arXiv. https://doi.org/10.48550/arXiv.2108.07258.

2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.

3. Büyükakyüz, K. (2024). OLORA: Orthonormal Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2406.01775

4. Hayou, S., Ghosh, N., & Yu, B. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. arXiv preprint arXiv:2402.12354.

5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of

Large Language Models. arXiv preprint arXiv:2106.09685.

6. Taori, R., et al. (2023). Stanford Alpaca: An Instruction-following LLaMA Model. Stanford Center for Research on Foundation Models (CRFM).

7. Wikipedia contributors. (n.d.). ROUGE (metric). In Wikipedia. Retrieved October 7, 2025, from https://en.wikipedia.org/wiki/ROUGE_(metric)

8. Wikipedia contributors. (n.d.). BLEU. In Wikipedia. Retrieved October 7, 2025, from https://en.wikipedia.org/wiki/BLEU

9. Loshchilov, I., & Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv. https://doi.org/10.48550/arXiv.1711.05101

10. Zhang, P., Zeng, G., Wang, T., & Lu, W. (2024). TinyLlama: An Open-Source Small Language Model. arXiv. https://doi.org/10.48550/arXiv.2401.02385

11. Wikipedia contributors. (n.d.). Evaluation function. In Wikipedia. Retrieved October 7, 2025, from https://en.wikipedia.org/wiki/Evaluation_function

12. Aghajanyan, A., Zettlemoyer, L., & Gupta, S. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv. https://doi.org/10.48550/arXiv.2012.13255

13. Meng, F., Wang, Z., & Zhang, M. (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2404.02948