Abu Dhabi [UAE], August 16: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) is pioneering research to refine artificial intelligence so that systems can genuinely grasp Arab culture and navigate its complex regional dialects.
In a landmark achievement, MBZUAI researchers have developed the first benchmark to measure AI models' ability to understand and engage with Arab culture across 13 national dialects, revealing a striking gap between the models' ability to understand what Arabs say and their ability to speak the way Arabs do in everyday life.
The study's lead researchers, Professor Fajri Koto, Assistant Professor in the Department of Natural Language Processing, and Muhammad Dehan, a researcher in the same department, told the Emirates News Agency (WAM) that bridging this gap could pave the way for the next stage in the development of Arabic-language AI, enabling it not only to understand the language but also to grasp the particular ways in which it is used across different Arab societies. Muhammad Dehan explained that Arabic unites more than 400 million speakers worldwide, but one of the main challenges in AI development is that models are trained almost exclusively on Modern Standard Arabic. As a result, they may appear fluent and capable in tests without having been sufficiently tested in natural conversations across the different dialects spoken in Arab countries while accurately reflecting the details of local cultures.
To address this gap, the researchers developed ArabCulture-Dialogue, which they describe as "the first benchmark for testing Arabic cultural reasoning in multi-turn conversations across Modern Standard Arabic and 13 national dialects from different Arab countries." The team recruited 26 native Arabic speakers across 13 countries to create authentic multi-turn conversations covering 12 topics from everyday life, ranging from weddings and food to parenting, agriculture, arts and games.
Professor Fajri Koto, Assistant Professor of Natural Language Processing at MBZUAI, said: "We gave the models three tasks: pick the culturally appropriate reply from a set of options; translate between Modern Standard Arabic and a specific dialect; and continue a conversation in a named dialect on request. The strongest models did well at recognizing the culturally appropriate answer, scoring in the mid-90s, even when the conversation shifted from MSA to dialect. But when we asked them to produce dialect, for example to translate a line into Emirati, or to continue a conversation in that dialect, performance dropped sharply."
The study found that models performed better when dealing with customs shared across the Arab world, while country-specific customs proved more difficult. North African dialect dialogues were among the most challenging overall, while Emirati dialogues were also among the most difficult for the models. The models succeeded in producing the dialect corresponding to the target country in only about half of cases.
Muhammad Dehan explained that "the paradox is that the cultural knowledge is already present within the models, but they sometimes require only a small amount of guidance," noting that specifying the country and region associated with a conversation led to improved accuracy.
The findings carry strategic importance to the UAE, which has made artificial intelligence a national priority through the UAE National Strategy for Artificial Intelligence 2031 and the development of home-grown AI models, including Jais. Professor Fajri Koto emphasised that the broader conclusion of the study carries a cautionary message for anyone developing Arabic-language AI models: a system that "supports Arabic" is not necessarily a system that understands Arabic across its different dialects and the cultures and identities embedded within them.
The team published its research paper, titled "Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues", which was presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2026), providing a foundation for further studies aimed at developing AI models that are better able to understand the Arab world's linguistic and cultural diversity while contributing to efforts to preserve local cultures and identities.
Source: Emirates News Agency
