--- title: "AMSN at PFB: Overview of the Automated Multilingual" description: "This EVALITA 2026 report describes one participating team's system for the AMSN shared task, despite \"Overview\" in its title. 3 references." authors: ["Ali Saleh Mohammadabad", "Mehregan Nazarmohsenifakori"] year: 2026 venue: "Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)" source: "https://aclanthology.org/2026.evalita-1.60/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mohammadabad-2026-amsn-pfb-overview-automated-multilingual/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammadabad-2026-amsn-pfb-overview-automated-multilingual/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task # AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task **Authors:** Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori · **Year:** 2026 · **Venue:** [Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)](https://md.research.elamur.ai/free/en/venues/evalita/) ([Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop 2026](https://md.research.elamur.ai/free/en/venues/evalita/2026/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.evalita-1.60/) · [PDF](https://aclanthology.org/2026.evalita-1.60.pdf) ## Summary This EVALITA 2026 report describes one participating team's system for the AMSN shared task, despite "Overview" in its title. The task is multiple-choice economics question answering in English, Italian and Turkish. Each item has five options (A–E, where E is usually a "none of the above" choice). Topics span micro- and macroeconomics, finance and quantitative methods, with 1,001 test questions per language. The authors compare three strategies: - fine-tuning mDeBERTa-v3; - querying GPT-4o and GPT-5 through the OpenAI API with a chain-of-thought system prompt; - a hybrid in which transformer classifiers flag likely errors in GPT-4o's output and substitute a new answer. Fine-tuning plateaus near 50%. GPT-4o with chain-of-thought gets 74% on English and Italian validation, and the hybrid raises this to roughly 78%. GPT-5 alone beats all of these, and the correction layer adds nothing on top of it. Zero-shot GPT-5 was therefore submitted, averaging 89.54% on test. The main takeaway is that a stronger foundation model can make engineered post-correction unnecessary. However, the results depend on a proprietary model whose exact snapshot is not reported. ## Key points - **Prompt selection:** Five prompting strategies were compared on only 50 validation questions. Chain of Thought and Expert Reasoning tied at 90%. Final settings were temperature 0.1 and up to 1500 output tokens, with regex-style extraction of an "ANSWER: X" line. - **Validation accuracy:** GPT-4o scored 74/74/65% and GPT-5 scored 82/81/73% (EN/IT/TR). Turkish trails the other two languages by roughly 8–9 points for both models. - **Test accuracy (GPT-5):** - EN: 89.11% (892/1,001) - IT: 91.21% (913/1,001) - TR: 88.31% (884/1,001) These are 7–15 points above validation. The authors attribute the gap to differences between the data splits rather than leakage, but they offer no verification. - **Hybrid system:** It consists of a binary mistake detector plus a five-way answer corrector, both trained on GPT-4o's validation predictions. The reported gain is inconsistent: 3–5 pp in the results section but about 6.5 pp in the conclusion. No Turkish figure is given, so its 77.5% average covers only EN and IT. - **Caveats:** - The GPT-5 version string is left as a placeholder, which hinders reproducibility. - Option E is predicted in only 1–2% of cases, while options A–C account for about 82–83% of predictions. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works 🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works ## Cite Cite as: Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori. 2026. AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mohammadabad2026amsn, title = {{AMSN at PFB: Overview of the Automated Multilingual Economics Question Answering Task}}, author = {Ali Saleh Mohammadabad and Mehregan Nazarmohsenifakori}, booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)}, year = {2026}, url = {https://aclanthology.org/2026.evalita-1.60/} } ``` - [← previous AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic](https://md.research.elamur.ai/free/en/papers/azime-2026-amharicstoryqa-multicultural-story-question-answering/) - [next → AMTA Best Thesis Award Abstract: Overcoming Vocabulary Challenges in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/salesky-2026-amta-best-thesis-award-abstract/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, in: International Conference on Learning Representations (ICLR), 2023. [find](https://scholar.google.com/scholar?q=P%2E+He%2C+J%2E+Gao%2C+W%2E+Chen%2C+Debertav3%3A+Improving+deberta+using+electra%2Dstyle+pre%2Dtraining+with+gradient%2Ddisentangled+embedding+sharing%2C+in%3A+International+Conference+on+Learning+Representations+%28ICLR%29%2C+202) 2. OpenAI, [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), arXiv preprint arXiv:2303.08774 (2023). 3. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), in: Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022, pp. 24824–24837. *Updated 1 November 2026*