Arabic Language and Large Language Models: A Scoping Review of Development, Applications, Dialectal Representation, Evaluation, and Research Gaps (2021–2026)
Keywords:
Arabic language, large language models, dialectal Arabic, model evaluationAbstract
Arabic-focused and multilingual large language models (LLMs) now support generation, translation, question answering, educational feedback, and specialist workflows, yet studies evaluate different linguistic varieties and outcomes. This scoping-review manuscript maps the development, applications, and evaluation of Arabic LLMs through a Population–Concept–Context framework. Targeted scholarly discovery yielded 49 candidate primary studies published in 2022–2026, provisionally charted by model strategy, benchmark, dialect, cultural context, and application. The preliminary map suggests that Modern Standard Arabic test accuracy does not establish dialectal fluency, factual reliability, or domain-specific utility. Native, translated, and synthetic datasets provide complementary but non-equivalent evidence. We propose evaluation incorporating provenance, human annotation, contamination safeguards, and context-specific outcomes. These 49 records are not a final systematically included set; database exports, independent screening, and full-text verification remain outstanding.
References
Aboukozzana, S., Khan, M. K. J., & Ali, A. (2025). AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.37
Abu Shairah, H., AlHarbi, S., AlHussein, A., Alsabea, S., Shaqaqi, O., AlShamlan, H., Knio, O., & Turkiyyah, G. (2025). ALARB: An Arabic Legal Argument Reasoning Benchmark. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.32
Al Kautsar, M. D., Almheiri, S., Ahsan, M., Elbouardi, B., Samih, Y., Ahmad, S., Keleg, A., El Herraoui, O., Elzeky, K., Freihat, A. A., Anwar, M., Xie, Z., Liang, J., Nasar, M. R. A., Nakov, P., & Koto, F. (2026). Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Vol. 1). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-long.963
Al Khatib, K., & Khader, M. (2025). Toward Culturally-Aware Arabic Debate Platforms with NLP Support. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.29
Alansari, A., & Luqman, H. (2025). AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.12
Alhassoun, M., Alkhanen, I. M., Alshalawi, N., Baazeem, I., & Alsanie, W. (2025). Saudi-Alignment Benchmark: Assessing LLMs Alignment with Cultural Norms and Domain Knowledge in the Saudi Context. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.11
Aljabari, A., Hamad, N., Khalilia, M., & Jarrar, M. (2025). WojoodOntology: Ontology-Driven LLM Prompting for Unified Information Extraction Tasks. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.14
Al-Matham, R., Darwish, K., Al-Rasheed, R., Alshammari, W., Alhoshan, M., Almazrua, A., Al Wazrah, A., Alheraki, M., Alam, F., Nakov, P., Alzahrani, N., AlBilali, E., Habash, N., El-Sheikh, A., Elmallah, M., Li, H., Mubarak, H., Anwar, M., Alyafeai, Z., ..., & AlOsaimy, A. (2025). BALSAM: A Platform for Benchmarking Arabic Large Language Models. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.21
Almazrouei, E., Cojocaru, R., Baldo, M., Malartic, Q., Alobeidli, H., Mazzotta, D., Penedo, G., Campesan, G., Farooq, M., Alhammadi, M., Launay, J., & Noune, B. (2023). AlGhafa Evaluation Benchmark for Arabic Language Models. In Proceedings of ArabicNLP 2023. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.arabicnlp-1.21
Altakrori, M. H., Habash, N., Lynn, T., Samih, Y., Freihat, A. A., Chirkunov, K., AbuOdeh, M., Florian, R., Nakov, P., & Aji, A. F. (2026). DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models. In Proceedings of the Fifteenth Language Resources and Evaluation Conference. Association for Computational Linguistics. https://doi.org/10.63317/3cy68duew55b
Althubaiti, S., Lodagala, V. S., Clark, T., Elshahawy, Y. A., Izham, D., Alrajeh, A., Tamran, A. B., & Ali, A. (2025). Octopus: Towards Building the Arabic Speech LLM Suite. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.35
Alwajih, F., Bhatia, G., & Abdul-Mageed, M. (2024). Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.27
Alwajih, F., Mekki, A. E., Mubarak, H., Hawasly, M., Mohamed, A., & Abdul-Mageed, M. (2025). PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.107
Amiraz, C., Fyodorov, Y., Haramaty, E., Karnin, Z., & Lewin-Eytan, L. (2025). The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.6
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Ben Hassine, S., Arrak, A., Addhoum, M., & Wilson, S. R. (2025). TounsiBench: Benchmarking Large Language Models for Tunisian Arabic. In Proceedings of EMNLP 2025. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.emnlp-main.1756
Bouchekif, A., Rashwani, S., Sbahi, H., Gaben, S., Al Khatib, M., & Ghaly, M. (2025). Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluation. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.20
Boussaha, B. E. A., Al Qadi, L., Farooq, M., Alsuwaidi, S., Campesan, G., Alzubaidi, A., Alyafeai, M., & Hacid, H. (2025a). 3LM: Bridging Arabic, STEM, and Code through Benchmarking. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.4
Boussaha, B. E. A., Alyafeai, M., Alzubaidi, A., Al Qadi, L., Alsuwaidi, S., & Hacid, H. (2025b). Adapting Falcon3-7B Language Model for Arabic: Methods, Challenges, and Outcomes. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.1
Demidova, A., Atwany, H., Rabih, N., Sha’ban, S., & Abdul-Mageed, M. (2024). John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.18
El-Haj, M. (2025). ArabJobs: A Multinational Corpus of Arabic Job Ads. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.2
El-Shangiti, A. O., Alwajih, F., & Abdul-Mageed, M. (2024). Arabic Automatic Story Generation with Large Language Models. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.13
ElFqih, K. A., & Monti, J. (2024). Large Language Models as Legal Translators of Arabic Legislation: Do ChatGPT and Gemini Care for Context and Terminology?. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.10
Eltahir, M., Sarraj, O., Alfrihidi, A. M., Alshatiri, T., Khurd, M., Bremoo, M., & Hussain, T. (2025). AutoArabic: A Three-Stage Framework for Localizing Video-Text Retrieval Benchmarks. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.23
Eltanbouly, S., Albatarni, S., Hassanein, S., & Elsayed, T. (2025). Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.16
Ersoy, A., Altinisik, E., Darwish, K. M., & Sencar, H. T. (2025). Tool Calling for Arabic LLMs: Data Strategies and Instruction Tuning. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.28
Gupta, V., Chowdhury, S. P., Zouhar, V., Rooein, D., & Sachan, M. (2025). Are Large Language Models for Education Reliable Across Languages?. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.bea-1.44
Hassib, M. E., El-Makky, N., & Torki, M. (2025). Open-domain Arabic Conversational Question Answering with Question Rewriting. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.7
Hawasly, M., Mohiuddin, T., Mubarak, H., & Boughorbel, S. (2025). ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.36
Hijazi, F., AlHarbi, S., AlHussein, A., Abu Shairah, H., AlZahrani, R., AlShamlan, H., Knio, O., & Turkiyyah, G. (2024). ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.20
Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Song, D., Chen, Z., Alharthi, M., An, B., He, J., Liu, Z., Chen, J., Li, J., Wang, B., Zhang, L., Sun, R., Wan, X., Li, H., & Xu, J. (2024). AceGPT, Localizing Large Language Models in Arabic. In Proceedings of NAACL 2024 (Vol. 1). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.450
Ibrahim, A., Hosseini, A., Helmy, H., Lakhdhar, W., & Serag, A. (2025). Bridging Dialectal Gaps in Arabic Medical LLMs through Model Merging. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.27
Khondaker, M. T. I., Naeem, N., Khan, F., Elmadany, A., & Abdul-Mageed, M. (2024). Benchmarking LLaMA-3 on Arabic Language Generation Tasks. In Proceedings of the Second Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.arabicnlp-1.24
Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A., Alraeesi, A., Almubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., Habash, N., Nakov, P., & Baldwin, T. (2024). ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.334
Kwon, S., Bhatia, G., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023). Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction. In Proceedings of ArabicNLP 2023. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.arabicnlp-1.9
Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5, 69. https://doi.org/10.1186/1748-5908-5-69
Mohammed, M. S., & Khalil, M. (2025). ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.8
Mousi, B., Durrani, N., & Dalvi, F. (2025b). An Exploration of Knowledge Editing for Arabic. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.34
Mousi, B., Durrani, N., Ahmad, F., Hasan, M. A., Hasanain, M., Kabbani, T., Dalvi, F., Chowdhury, S. A., & Alam, F. (2025a). AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics. https://aclanthology.org/2025.coling-main.283/
Mubarak, H., Al-Khalifa, H., & Alkhalefah, K. S. (2024). Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study. In Proceedings of LREC-COLING 2024. Association for Computational Linguistics. https://aclanthology.org/2024.lrec-main.705/
Mubarak, H., Mohamed, A., & Hawasly, M. (2025). AraSafe: Benchmarking Safety in Arabic LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.529
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18, 143. https://doi.org/10.1186/s12874-018-0611-x
Nabhani, S., & Al Khatib, K. (2025). Transfer or Translate? Argument Mining in Arabic with No Native Annotations. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.33
Nacar, O., Sibaee, S. T., Ahmed, S., Ben Atitallah, S., Ammar, A., Alhabashi, Y., Al-Batati, A. S., Alsehibani, A., Qandos, N., Elshehy, O., Abdelkader, M., & Koubaa, A. (2025). Towards Inclusive Arabic LLMs: A Culturally Aligned Benchmark in Arabic Large Language Model Evaluation. In Proceedings of the First Workshop on Language Models for Low-Resource Languages. Association for Computational Linguistics. https://aclanthology.org/2025.loreslm-1.29/
Nagoudi, E. M. B., Elmadany, A., & Abdul-Mageed, M. (2022). AraT5: Text-to-Text Transformers for Arabic Language Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.47
Noureldien, Y., Mohamed, A., & Attallah, F. (2025). Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for Arabic Word Sense Disambiguation. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.24
Peters, M. D. J., Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C., & Khalil, H. (2024). Scoping reviews. In JBI Manual for Evidence Synthesis. https://doi.org/10.46658/JBIMES-24-09
Qwaider, C., Chirkunov, K., Alhafni, B., Habash, N., & Briscoe, T. (2025). Evaluating Prompt Relevance in Arabic Automatic Essay Scoring: Insights from Synthetic and Real-World Data. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.13
Robinson, N. R., Abdelmoneim, S., Kantharuban, A., Alsboul, O., Lamsiyah, S., Marchisio, K., & Murray, K. (2026). AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026. In Proceedings of VarDial 2026. Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.vardial-1.1
Shang, G., Abdine, H., Chamma, A., Mohamed, A., Anwar, M., Bounhar, A., El Herraoui, O., Nakov, P., Vazirgiannis, M., & Xing, E. P. (2025). Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.25
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., ... Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Wang, K., Toibazar, D., & Mengibar, P. J. M. (2025). A-SEA³L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.9
Zbib, M. B., Hammoud, H. A. A. K., Mohanna, A., Rizk, N., Karnib, F., Moukaled, S., & Ghanem, B. (2026). AraLingBench: A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models. In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script. Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.abjadnlp-1.45
Zeinalipour, K., Saad, M. Z., Attafi, O., Maggini, M., & Gori, M. (2025). Shawarma Chats: A Benchmark Exact Dialogue & Evaluation Platter in Egyptian, Maghrebi & Modern Standard Arabic—A Triple-Dialect Feast for Hungry Language Models. In Proceedings of The Third Arabic Natural Language Processing Conference. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.arabicnlp-main.39
