HOW ACCURATELY DO ARTIFICIAL INTELLIGENCE CHATBOTS EXPLAIN INTERNATIONAL PROTECTION? A COMPARATIVE EVALUATION

OpenAlex Türkçe

Background: This study aims to comparatively evaluate the performance of artificial intelligence chatbots in explaining fundamental concepts and processes related to international protection status. Within the scope of the research, a set of 10 questions related to international protection status was posed to the OpenAI ChatGPT-5, Google Gemini 2.5 Flash, and Microsoft Copilot Think Deeper models in October 2025. Objective: The initial responses generated by the models were scored by five academics specialising in migration using a blind evaluation method. The evaluation process was based on the Handbook on Procedures and Criteria for Determining Refugee Status and Guidelines on International Protection developed by the United Nations High Commissioner for Refugees (UNHCR). Method: The responses were evaluated on a five-point scale according to their accuracy and comprehensiveness. Kendall's W coefficient was used to determine inter-rater agreement in the analysis of the data obtained, while the Friedman test was used to test the differences between the artificial intelligence models. Results: The findings show that all models are generally accurate but often provide limited and superficial explanations. It was found that ChatGPT's average scores were higher than those of other models, but there were significant differences between models, particularly in concepts open to legal interpretation, such as “complementary protection”. Conclusion: Consequently, it was concluded that artificial intelligence chatbots could serve as auxiliary tools providing rapid access to fundamental information in the field of international protection, but could not replace expert assessment in rights-based and legally consequential individual application processes.