YUQORI N-GRAM MODELLARINI O’ZBEK TILI MATNLARIGA QOʻLLASH

Botir Elov; Ruhillo Alayev; Abdulla Abdullayev; Narzillo Aloyev

Авторы

Botir Elov Alisher Navoiy nomidagi Toshkent davlat oʻzbek tili va adabiyoti universiteti
Ruhillo Alayev Mirzo Ulugʻbek nomidagi O‘zbekiston Milliy universiteti
Abdulla Abdullayev Urganch innovatsion university
Narzillo Aloyev Alisher Navoiy nomidagi Toshkent davlat o‘zbek tili va adabiyoti universiteti

Ключевые слова:

Til modellari, n-gram til modeli, o'rtacha logarifmik ehtimollik, modelni baholash, Laplas silliqlash, mashinali o’qitish

Аннотация

Tabiiy tilni qayta ishlashda kontekstdagi keyingi qaysi so’zni bashorat qilish vazifasi tilni modellashtirish deb ataladi. N-gram tahlili tilni qayta ishlashning muhim usuli boʻlib, u til tuzilishini tushunishga va gapdagi fragmentdan keyin qanday so`z kelishini bashorat qilishga yordam beradi. N-gramlarni tahlil qilish matn yaratish, imloni tuzatish va hissiyotlarni tahlil qilish kabi murakkab vazifalarda foydalidir. NLP modellari til qoliplarini yaxshiroq tushunishi va qaysi so'zlar birgalikda paydo bo'lishini o'rganish orqali samaraliroq bashorat qilishlari mumkin. N-gram modellari mashina tarjimasi, chatbotlar va qidiruv tizimlari kabi turli xil NLP ilovalarini yaxshilashga yordam beradi.

Библиографические ссылки

Booch, G., Jacobson, I., & Rumbaugh, J. (1996). The unified modeling language. Unix Review, 14(13), 5.

Tan, M., Zhou, W., Zheng, L., & Wang, S. (2012). A scalable distributed syntactic, semantic, and lexical language model. Computational Linguistics, 38(3), 631-671.

Brown, P. F., Della Pietra, V. J., Desouza, P. V., Lai, J. C., & Mercer, R. L. (1992). Class-based n-gram models of natural language. Computational linguistics, 18(4), 467-480.

Hockey, B. A., & Rayner, M. (2005, July). Comparison of grammar-based and statistical language models trained on the same data. In Proceedings of the AAAI Workshop on Spoken Language Understanding (pp. 9-10).

Mikolov, T. (2012). Statistical language models based on neural networks.

Kim, Y., Jernite, Y., Sontag, D., & Rush, A. (2016, March). Character-aware neural language models. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1).

B.Elov, Sh.Sirojiddinov, Sh.Hamroyeva, E.Adalı, Z.Xusainova (2023, September). Pos Taging of Uzbek Text Using Hidden Markov Model. In 2023 8th International Conference on Computer Science and Engineering (UBMK) (pp. 63-68). IEEE.

B.Elov, Sh.Khamroeva, R.Alayev, Z.Khusainova, U.Yodgorov (2023). Methods of processing the uzbek language corpus texts. International Journal of Open Information Technologies, 11(12), 143-151.

Liu, Y., & Zhang, M. (2018). Neural network methods for natural language processing.

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., ... & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45.

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Chen, S. F., Beeferman, D., & Rosenfeld, R. (1998). Evaluation metrics for language models.

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., ... & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45.

B.Elov (2022). N-gram til modellari vositasida o ‘zbek tilida matn generatsiya qilish. Computer linguistics: problems, solutions, prospects, 1(1).

B. Elov, A. Abdullayev, A., N.Xudoyberganov. (2024). O‘zbek tili korpusi matnlari asosida til modellarini yaratish. «Contemporary technologies of computational linguistics», 2(22.04), 344-353