---
title: "Can LLMs Faithfully Explain Themselves in — Mehrazar 2026"
description: "The paper asks whether the explanations LLMs give for their own decisions reflect how they actually decide, when the language is low-resource."
authors: ["Mobina Mehrazar", "Mohammad Amin Yousefi", "Parisa Beygi", "Behnam Bahrak"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.39/"
doi: "10.63317/4tds2kffmg7j"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/mehrazar-2026-can-llms-faithfully-explain-themselves/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/mehrazar-2026-can-llms-faithfully-explain-themselves/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/) / Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian

# Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian

**Authors:** Mobina Mehrazar, Mohammad Amin Yousefi, Parisa Beygi, Behnam Bahrak · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.39/) · [PDF](https://aclanthology.org/2026.lrec-1.39.pdf) · [DOI 10.63317/4tds2kffmg7j](https://doi.org/10.63317/4tds2kffmg7j) · [OpenAlex](https://openalex.org/W4416766459)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The paper asks whether the explanations LLMs give for their own decisions reflect how they actually decide, when the language is low-resource. The test case is emotion classification in Persian. The authors use a Persian emotion-labeled corpus containing both formal and informal texts. Models assign each text to one of six emotions and name the top-k input words behind that choice. Two prompt orders are compared: Predict-then-Explain (P-E) and Explain-then-Predict (E-P).

Faithfulness is measured by re-classifying each text with only the chosen words kept and with those words masked. Changes in prediction and confidence show whether the words carried the decision. Confidence comes from token-level log-probabilities rather than self-reported certainty. Because the raw scores were overconfident, they are corrected with temperature scaling. A two-stage human annotation, built to match the model output format, supplies reference influential words.

The models classify emotions well, but their explanations often do not match their actual reasoning. They also agree more with one another than with human annotators. The authors conclude that current explanation methods and metrics need improvement before LLMs can be trusted in multilingual, low-resource settings.

## Key points
- Six emotion classes are used: Sadness, Happiness, Anger, Surprise, Hatred, Fear. Labels are returned as numeric codes 0–5.
- Models evaluated include GPT-4-turbo, GPT-4o, Llama3.3-70B-instruct and DeepSeek-V3, all accessed via API.
- There are three input conditions: Full-text, Top-k only, and Top-k removed, where the chosen words are replaced by a `[removed]` placeholder. Each condition runs in a fresh conversation.
- Overconfidence is reduced with post-hoc temperature scaling (Guo et al., 2017), tuned on a held-out calibration set by minimizing Expected Calibration Error. Reliability diagrams are reported per model and paradigm.
- The authors present the work as the first to combine log-probability-based, calibrated confidence with self-explanation evaluation. It is also among the first such studies on Persian emotion detection.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mehrazar2026can,
  title = {{Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian}},
  author = {Mehrazar, Mobina and Yousefi, Mohammad Amin and Beygi, Parisa and Bahrak, Behnam},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4tds2kffmg7j},
  url = {https://doi.org/10.63317/4tds2kffmg7j}
}
```

- [← previous Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection](https://md.research.elamur.ai/free/papers/piot-2026-can-llms-evaluate-what-they/)
- [next → Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task](https://md.research.elamur.ai/free/papers/qiu-2026-can-llms-simulate-human-behavioral/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/)

## References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31,… [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Tim+Rockt%C3%A4schel%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2018%2E+e%2DSNLI%3A+Natural+Language+Inference+with+Natural+Language+Explanations%2E+In+S%2E+Bengio%2C+H%2E+Wallach%2C+H%2E+Larochelle%2C+K%2E+Grau) [our card](https://md.research.elamur.ai/free/papers/camburu-2018-e-snli-natural-language-inference/)
2. George Chrysostomou and Nikolaos Aletras. 2021. Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… [find](https://scholar.google.com/scholar?q=George+Chrysostomou+and+Nikolaos+Aletras%2E+2021%2E+Improving+the+Faithfulness+of+Attention%2Dbased+Explanations+with+Task%2Dspecific+Information+for+Text+Classification%2E+In+Proceedings+of+the+59th+Annual+Mee)
3. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A Benchmark to Evaluate Rationalized NLP Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Asso… [find](https://scholar.google.com/scholar?q=Jay+DeYoung%2C+Sarthak+Jain%2C+Nazneen+Fatema+Rajani%2C+Eric+Lehman%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Byron+C%2E+Wallace%2E+2020%2E+ERASER%3A+A+Benchmark+to+Evaluate+Rationalized+NLP+Models%2E+In+Proceedings+of+the)
4. DeepSeek-AI et al. 2025. DeepSeek-V3 Technical Report. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI+et+al%2E+2025%2E+DeepSeek%2DV3+Technical+Report%2E)
5. OpenAI et al. 2024a. GPT-4 Technical Report. [find](https://scholar.google.com/scholar?q=OpenAI+et+al%2E+2024a%2E+GPT%2D4+Technical+Report%2E) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
6. OpenAI et al. 2024b. GPT-4o System Card. [find](https://scholar.google.com/scholar?q=OpenAI+et+al%2E+2024b%2E+GPT%2D4o+System+Card%2E) [our card](https://md.research.elamur.ai/free/papers/openai-2024-gpt-4o-system-card/)
7. Aaron Grattafiori, Abhimanyu Dubey, Hugo Touvron, Angela Fan, Gabriel Synnaeve, Sergey Edunov, Armen Aghajanyan, Thomas Scialom, Roberta Raileanu, Soumith Chintala, and Meta AI. 2024. The Llama 3 Herd of Models. [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Hugo+Touvron%2C+Angela+Fan%2C+Gabriel+Synnaeve%2C+Sergey+Edunov%2C+Armen+Aghajanyan%2C+Thomas+Scialom%2C+Roberta+Raileanu%2C+Soumith+Chintala%2C+and+Meta+AI%2E+2024%2E+The+Llama+3+Herd)
8. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Geoff+Pleiss%2C+Yu+Sun%2C+and+Kilian+Q%2E+Weinberger%2E+2017%2E+On+Calibration+of+Modern+Neural+Networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning+%2D+Volume+70%2C+ICML%E2%80%9917%2C)
9. Henning Heyen, Amy Widdicombe, Noah Yamamoto Siegel, Philip Colin Treleaven, and Maria Perez-Ortiz. 2024. The Effect of Model Size on LLM post-hoc Explainability via LIME. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. [find](https://scholar.google.com/scholar?q=Henning+Heyen%2C+Amy+Widdicombe%2C+Noah+Yamamoto+Siegel%2C+Philip+Colin+Treleaven%2C+and+Maria+Perez%2DOrtiz%2E+2024%2E+The+Effect+of+Model+Size+on+LLM+post%2Dhoc+Explainability+via+LIME%2E+In+ICLR+2024+Workshop+on+Sec)
10. Miriam Horovicz and Roni Goldshmidt. 2024. TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), pages 1–8, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Miriam+Horovicz+and+Roni+Goldshmidt%2E+2024%2E+TokenSHAP%3A+Interpreting+Large+Language+Models+with+Monte+Carlo+Shapley+Value+Estimation%2E+In+Proceedings+of+the+1st+Workshop+on+NLP+for+Science+%28NLP4Science%29%2C)
11. Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin. 2023. Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations. [find](https://scholar.google.com/scholar?q=Shiyuan+Huang%2C+Siddarth+Mamidanna%2C+Shreedhar+Jangam%2C+Yilun+Zhou%2C+and+Leilani+H%2E+Gilpin%2E+2023%2E+Can+Large+Language+Models+Explain+Themselves%3F+A+Study+of+LLM%2DGenerated+Self%2DExplanations%2E)
12. Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. 2016. Investigating the Influence of Noise and Distractors on the Interpretation of Neural Networks. [find](https://scholar.google.com/scholar?q=Pieter%2DJan+Kindermans%2C+Kristof+Sch%C3%BCtt%2C+Klaus%2DRobert+M%C3%BCller%2C+and+Sven+D%C3%A4hne%2E+2016%2E+Investigating+the+Influence+of+Noise+and+Distractors+on+the+Interpretation+of+Neural+Networks%2E)
13. Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. 2024. The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Satyapriya+Krishna%2C+Tessa+Han%2C+Alex+Gu%2C+Steven+Wu%2C+Shahin+Jabbari%2C+and+Himabindu+Lakkaraju%2E+2024%2E+The+Disagreement+Problem+in+Explainable+Machine+Learning%3A+A+Practitioner%E2%80%99s+Perspective%2E+Transactions+o)
14. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Understanding Neural Networks through Representation Erasure. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2017%2E+Understanding+Neural+Networks+through+Representation+Erasure%2E)
15. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are Multilingual LLMs Culturally-diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… [find](https://scholar.google.com/scholar?q=Chen+Liu%2C+Fajri+Koto%2C+Timothy+Baldwin%2C+and+Iryna+Gurevych%2E+2024%2E+Are+Multilingual+LLMs+Culturally%2Ddiverse+Reasoners%3F+An+Investigation+into+Multicultural+Proverbs+and+Sayings%2E+In+Proceedings+of+the+202)
16. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Scott+M+Lundberg+and+Su%2DIn+Lee%2E+2017%2E+A+Unified+Approach+to+Interpreting+Model+Predictions%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+30%2E+Curran+Associates%2C+Inc%2E) [our card](https://md.research.elamur.ai/free/papers/lundberg-2017-unified-approach-interpreting-model-predictions/)
17. Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024. Are Self-explanations from Large Language Models Faithful? In Findings of the Association for Computational Linguistics: ACL 2024, pages 295–337, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Andreas+Madsen%2C+Sarath+Chandar%2C+and+Siva+Reddy%2E+2024%2E+Are+Self%2Dexplanations+from+Large+Language+Models+Faithful%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2024%2C+pages+295%E2%80%93337%2C+)
18. Hossein Mirzaee, Javad Peymanfard, Hamid Habibzadeh Moshtaghin, and Hossein Zeinali. 2025. ArmanEmo: a Persian Dataset for Text-based Emotion Detection. Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Hossein+Mirzaee%2C+Javad+Peymanfard%2C+Hamid+Habibzadeh+Moshtaghin%2C+and+Hossein+Zeinali%2E+2025%2E+ArmanEmo%3A+a+Persian+Dataset+for+Text%2Dbased+Emotion+Detection%2E+Language+Resources+and+Evaluation%2E)
19. Korbinian Randl, John Pavlopoulos, Aron Henriksson, and Tony Lindgren. 2025. Evaluating the Reliability of Self-explanations in Large Language Models. In Discovery Science, pages 36–51, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Korbinian+Randl%2C+John+Pavlopoulos%2C+Aron+Henriksson%2C+and+Tony+Lindgren%2E+2025%2E+Evaluating+the+Reliability+of+Self%2Dexplanations+in+Large+Language+Models%2E+In+Discovery+Science%2C+pages+36%E2%80%9351%2C+Cham%2E+Springer)
20. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Co… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+%22why+Should+I+Trust+You%3F%22%3A+Explaining+the+Predictions+of+Any+Classifier%2E+In+Proceedings+of+the+22nd+ACM+SIGKDD+International+Conference+on) [our card](https://md.research.elamur.ai/free/papers/ribeiro-2016-why-should-i-trust-explaining/)
21. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning Important Features Through Propagating Activation Differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3145–3153. JMLR.org. [find](https://scholar.google.com/scholar?q=Avanti+Shrikumar%2C+Peyton+Greenside%2C+and+Anshul+Kundaje%2E+2017%2E+Learning+Important+Features+Through+Propagating+Activation+Differences%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Lea) [our card](https://md.research.elamur.ai/free/papers/shrikumar-2017-learning-important-features-through-propagating/)
22. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 180–186, New York, NY, USA. Association for Computing… [find](https://scholar.google.com/scholar?q=Dylan+Slack%2C+Sophie+Hilgard%2C+Emily+Jia%2C+Sameer+Singh%2C+and+Himabindu+Lakkaraju%2E+2020%2E+Fooling+LIME+and+SHAP%3A+Adversarial+Attacks+on+Post+hoc+Explanation+Methods%2E+In+Proceedings+of+the+AAAI%2FACM+Conferen)
23. Jeevithashree Divya Venkatesh, Aparajita Jaiswal, and Gaurav Nanda. 2024. Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-tracking. Scientific Reports, 14:14295. [find](https://scholar.google.com/scholar?q=Jeevithashree+Divya+Venkatesh%2C+Aparajita+Jaiswal%2C+and+Gaurav+Nanda%2E+2024%2E+Comparing+Human+Text+Classification+Performance+and+Explainability+with+Large+Language+and+Machine+Learning+Models+Using+Eye%2Dt)
24. Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, and Kam-Fai Wong. 2024. A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models. [find](https://scholar.google.com/scholar?q=Boyang+Xue%2C+Hongru+Wang%2C+Rui+Wang%2C+Sheng+Wang%2C+Zezhong+Wang%2C+Yiming+Du%2C+Bin+Liang%2C+and+Kam%2DFai+Wong%2E+2024%2E+A+Comprehensive+Study+of+Multilingual+Confidence+Estimation+on+Large+Language+Models%2E)
25. Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3623–3643, Bangkok, Thail… [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Jena+Hwang%2C+Xiang+Ren%2C+and+Maarten+Sap%2E+2024%2E+Relying+on+the+Unreliable%3A+The+Impact+of+Language+Models%E2%80%99+Reluctance+to+Express+Uncertainty%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+th)
26. Input Format Average Reduction P-E E-P Full-text 16.60 12.17 GPT-4o Top-k only 8.77 7.07 Top-k removed 12.32 9.85 Full-text 39.03 20.65 GPT-4-turbo Top-k only 17.85 10.95 Top-k removed 9.67 5.03 Full-text 36.82 35.22 Llama3.3 70B-instruct Top-k only 18.71 14.99 Top-k removed 17.71 22.71 Full-text 33… [find](https://scholar.google.com/scholar?q=Input+Format+Average+Reduction+P%2DE+E%2DP+Full%2Dtext+16%2E60+12%2E17+GPT%2D4o+Top%2Dk+only+8%2E77+7%2E07+Top%2Dk+removed+12%2E32+9%2E85+Full%2Dtext+39%2E03+20%2E65+GPT%2D4%2Dturbo+Top%2Dk+only+17%2E85+10%2E95+Top%2Dk+removed+9%2E67+5%2E03+Full%2D)