--- title: "Pinetree at SemEval-2026 Task 7: A Large-Scale Failure" description: "This area has received less study than alignment measured with value surveys. 7 references, cited by 1 paper here." authors: ["Yen Yee Yam", "Hong Meng Yam"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.422/" doi: "10.18653/v1/2026.semeval-1.422" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yam-2026-pinetree-semeval-2026-task-7/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yam-2026-pinetree-semeval-2026-task-7/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/) / Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models # Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models **Authors:** [Yen Yee Yam](https://md.research.elamur.ai/free/en/authors/yen-yee-yam/) and [Hong Meng Yam](https://md.research.elamur.ai/free/en/authors/hong-meng-yam/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.422/) · [PDF](https://aclanthology.org/2026.semeval-1.422.pdf) · [DOI 10.18653/v1/2026.semeval-1.422](https://doi.org/10.18653/v1/2026.semeval-1.422) · [OpenAlex](https://openalex.org/W7166867063) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/yam-2026-pinetree-semeval-2026-task-7/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks how well large language models know routine, majority-default daily practices across cultures. This area has received less study than alignment measured with value surveys. For SemEval-2026 Task 7 Track 2 (47,014 multiple-choice questions across 30 locales, extending BLEnD), the authors queried a frozen GPT-5.2 with one structured prompt at temperature 0. They used no fine-tuning, retrieval, chain-of-thought or ensembling. The prompt steered the model toward contemporary, nationally mainstream behaviour and away from stereotypes and surface word overlap. This reached 88.85% micro and 90.55% macro accuracy, placing fourth. The main contribution is an analysis of all 5,241 wrong answers. Each was labelled with its BLEnD topic via question-ID lookup and tagged with rule-based cross-cutting flags. Relative to topic size, errors peak in Family and Work life. Western brand defaults are rare and confined to essentially one sports-team template. Numeric mistakes usually land one option away from the gold answer. The authors conclude that everyday cultural misalignment stems from sparse local knowledge rather than ideological bias. ## Key points - Results: 88.85% micro and 90.55% macro accuracy (rank #4). Country accuracy spans 69.96% (am-ET) to 99.28% (es-EC), and the weakest locales include ha-NG, ko-KP, fa-IR and ar-MA. - By absolute share, errors fall mostly in Food (39.42%), but within-topic error rates are highest for Family (21.04%) and Work life (20.45%). Rates are lowest for Sport (4.87%) and Holidays/Celebration/Leisure (6.90%). - Brand-attractor errors are only 2.50% of failures (131 cases): - 129 come from a "most popular sport team" question in four low-resource cultures. - 2 come from Australian payment-method questions. - Of 1,180 numeric errors, 71.1% (n=839) are rank-adjacent to the correct option, suggesting approximate rather than missing knowledge. - Heuristic error flags matched an independent GPT-5.2 blind labeler on 90.91% of the 4,061 non-numeric errors. The labeler confirmed 117 of the 131 brand cases. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 930 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Yen Yee Yam and Hong Meng Yam. 2026. Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yam2026pinetree, title = {{Pinetree at SemEval-2026 Task 7: A Large-Scale Failure Analysis of Cultural Grounding in Language Models}}, author = {Yen Yee Yam and Hong Meng Yam}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.422}, url = {https://doi.org/10.18653/v1/2026.semeval-1.422} } ``` - [← previous PhucNguyen@DravidianLangTech 2026: Political Multiclass Sentiment Analysis with XLM-RoBERTa and Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/nguyen-2026-phucnguyen-dravidianlangtech-2026-political-multiclass/) - [next → Pixel Phantoms at SemEval-2026 Task 3: Language-Specific Transformer Regression for Dimensional Aspect-Based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/s-2026-pixel-phantoms-semeval-2026-task/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/16/) ## References 7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073. 2. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. [Assessing cross-cultural alignment between chatgpt and human societies: An empirical study](https://md.research.elamur.ai/free/en/papers/cao-2023-assessing-cross-cultural-alignment-between/). Preprint, arXiv:2303.17466. 3. Yoshinari Fujinuma, Siddharth Varia, Nishant Sankaran, Srikar Appalaraju, Bonan Min, and Yogarshi Vyas. 2023. A multi-modal multilingual benchmark for document image classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14361–14376. [find](https://scholar.google.com/scholar?q=Yoshinari+Fujinuma%2C+Siddharth+Varia%2C+Nishant+Sankaran%2C+Srikar+Appalaraju%2C+Bonan+Min%2C+and+Yogarshi+Vyas%2E+2023%2E+A+multi%2Dmodal+multilingual+benchmark+for+document+image+classification%2E+In+Findings+of+the) 4. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. [find](https://scholar.google.com/scholar?q=Debanjan+Ghosh%2C+Kai+North%2C+Ekaterina+Kochmar%2C+Mamoru+Komachi%2C+and+Marcos+Zampieri%2C+editors%2E+2026%2E+Proceedings+of+the+20th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D2026%29%2E+Association+for+) 5. Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744. 6. Candace Ross, Boris Katz, and Andrei Barbu. 2021. Measuring social biases in grounded vision and language embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 998–1008, Online. Association… [find](https://scholar.google.com/scholar?q=Candace+Ross%2C+Boris+Katz%2C+and+Andrei+Barbu%2E+2021%2E+Measuring+social+biases+in+grounded+vision+and+language+embeddings%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Associa) 7. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. [Cultural bias and cultural alignment of large language models](https://md.research.elamur.ai/free/en/papers/tao-2023-cultural-bias-cultural-alignment-large/). PNAS Nexus, 3(9):pgae346. *Updated 2 November 2026*