---
title: "FORK — paper by Palta and Rudinger (2023)"
description: "The authors focus on food customs and build FORK (Food ORiented cultural commonsense… 24 references, cited by 52 papers here."
authors: ["Shramay Palta", "Rachel Rudinger"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.631/"
doi: "10.18653/v1/2023.findings-acl.631"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models

# FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models

**Authors:** Shramay Palta and [Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.631/) · [PDF](https://aclanthology.org/2023.findings-acl.631.pdf) · [DOI 10.18653/v1/2023.findings-acl.631](https://doi.org/10.18653/v1/2023.findings-acl.631) · [OpenAlex](https://openalex.org/W4385572212)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Commonsense QA systems are usually assumed to encode universal knowledge. Many everyday facts, however, depend on culture, such as whether to tip or which utensil fits a dish. This paper asks whether such systems quietly adopt a US viewpoint. The authors focus on food customs and build FORK (Food ORiented cultural commonsense Knowledge), a small hand-written test set in the CommonsenseQA format. Each question has two candidate answers: one matches US practice and the other a non-US practice. Questions come in three kinds. Underspecified items name no culture, Implicit items signal a culture only through context cues such as a dish name, and Explicit items state the country or city. Seven encoder models fine-tuned on CommonsenseQA were evaluated. With no cultural cue, most models picked the US answer. They were also far more accurate on US-contextualized questions than on non-US ones, even when the country was stated. This points to a systematic US-leaning bias, likely inherited from English pretraining text. The work is an early probe of cultural assumptions in commonsense reasoning.

## Key points
- FORK has 184 two-option questions: 31 Underspecified, 70 Implicit (14 US, 56 non-US) and 83 Explicit (13 US, 70 non-US). About 91.84% concern the US, China, Japan or India. Themes are eating utensils, tipping and general customs.
- Six annotators (two each from the US, China and India) validated the data. Cohen's kappa was 1.0 for US, 0.93 for China and 0.52 for India, with disagreement for India mostly on regional differences in tipping.
- The models tested were bert-base, bert-large, roberta-base, roberta-large, DistilBERT, deberta-v3-base and deberta-v3-large. Each was fine-tuned on the CommonsenseQA train split for 3 epochs.
- On Underspecified questions, roberta-large chose the US option 90.32% of the time and bert-large 83.87%. Every model except DistilBERT favored US answers in most cases.
- roberta-large scored 100% on US Explicit questions but only 22.85% on non-US ones. On non-US Implicit questions, just DistilBERT and bert-base exceeded 50%, while bert-large fell to 26.78%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 58 works

## Cite

Cite as: Shramay Palta and Rachel Rudinger. 2023. FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{palta2023fork,
  title = {{FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models}},
  author = {Shramay Palta and Rachel Rudinger},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.631},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.631}
}
```

- [← previous CREPE: Open-Domain Question Answering with False Presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/)
- [next → CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?](https://md.research.elamur.ai/free/en/papers/wei-2023-cmath-language-model-pass/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2020. Towards an atlas of cultural commonsense for machine reasoning. arXiv preprint arXiv:2009.05664. — [arXiv:2009.05664](https://arxiv.org/abs/2009.05664) [find](https://scholar.google.com/scholar?q=Anurag+Acharya%2C+Kartik+Talamadupula%2C+and+Mark+A+Finlayson%2E+2020%2E+Towards+an+atlas+of+cultural+commonsense+for+machine+reasoning%2E+arXiv+preprint+arXiv%3A2009%2E05664%2E+%E2%80%94+%5BarXiv%3A2009%2E05664%5D%28https%3A%2F%2Farxiv%2Eorg)
2. Ewa S. Callahan and Susan C. Herring. 2011. Cultural bias in wikipedia content on famous persons. Journal of the American Society for Information Science and Technology, 62(10):1899–1915. [find](https://scholar.google.com/scholar?q=Ewa+S%2E+Callahan+and+Susan+C%2E+Herring%2E+2011%2E+Cultural+bias+in+wikipedia+content+on+famous+persons%2E+Journal+of+the+American+Society+for+Information+Science+and+Technology%2C+62%2810%29%3A1899%E2%80%931915%2E)
3. Zheng Chen and Grant Henning. 1985. Linguistic and cultural bias in language proficiency tests. Language Testing, 2(2):155–163. [find](https://scholar.google.com/scholar?q=Zheng+Chen+and+Grant+Henning%2E+1985%2E+Linguistic+and+cultural+bias+in+language+proficiency+tests%2E+Language+Testing%2C+2%282%29%3A155%E2%80%93163%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). arXiv preprint arXiv:2111.09543.
6. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. [Challenges and strategies in cross-cultural N…](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
7. Diana Kendall. 2015. Sociology in our times: The essentials, 10 edition. CENGAGE Learning Custom Publishing, Mason, OH. [find](https://scholar.google.com/scholar?q=Diana+Kendall%2E+2015%2E+Sociology+in+our+times%3A+The+essentials%2C+10+edition%2E+CENGAGE+Learning+Custom+Publishing%2C+Mason%2C+OH%2E)
8. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. [UNQOVERing stereotyping biases via underspecified questions](https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489, Online. Association for Computational Linguistics.
9. Dekang Lin and Patrick Pantel. 2001. Dirt @sbt@discovery of inference rules from text. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’01, page 323–328, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Patrick+Pantel%2E+2001%2E+Dirt+%40sbt%40discovery+of+inference+rules+from+text%2E+In+Proceedings+of+the+Seventh+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9901)
10. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). arXiv preprint arXiv:2201.05955.
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
12. John McCarthy. 1959. Programs with common sense. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+1959%2E+Programs+with+common+sense%2E)
13. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. [CrowS-pairs: A challenge dataset for measuring social biases in masked language models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
14. Michaela Regneri, Alexander Koller, and Manfred Pinkal. 2010. Learning script knowledge with web experiments. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 979–988, Uppsala, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michaela+Regneri%2C+Alexander+Koller%2C+and+Manfred+Pinkal%2E+2010%2E+Learning+script+knowledge+with+web+experiments%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics)
15. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. [Gender bias in coreference resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1…
16. Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1681–1686, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Pushpendre+Rastogi%2C+Francis+Ferraro%2C+and+Benjamin+Van+Durme%2E+2015%2E+Script+induction+as+language+modeling%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Languag)
17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
18. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–3035. [find](https://scholar.google.com/scholar?q=Atomic%3A+An+atlas+of+machine+commonsense+for+if%2Dthen+reasoning%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volume+33%2C+pages+3027%E2%80%933035%2E)
19. Roger C Schank and Robert P Abelson. 1975. Scripts, plans, and knowledge. In IJCAI, volume 75, pages 151–157. [find](https://scholar.google.com/scholar?q=Roger+C+Schank+and+Robert+P+Abelson%2E+1975%2E+Scripts%2C+plans%2C+and+knowledge%2E+In+IJCAI%2C+volume+75%2C+pages+151%E2%80%93157%2E)
20. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
21. Yufei Tian, Tuhin Chakrabarty, Fred Morstatter, and Nanyun Peng. 2021. Identifying distributional perspectives from colingual groups. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 178–190, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yufei+Tian%2C+Tuhin+Chakrabarty%2C+Fred+Morstatter%2C+and+Nanyun+Peng%2E+2021%2E+Identifying+distributional+perspectives+from+colingual+groups%2E+In+Proceedings+of+the+Ninth+International+Workshop+on+Natural+Lang)
22. Benjamin Van Durme and Lenhart Schubert. 2008. Open knowledge extraction through compositional language processing. In Semantics in Text Processing. [find](https://scholar.google.com/scholar?q=Benjamin+Van+Durme+and+Lenhart+Schubert%2E+2008%2E+Open+knowledge+extraction+through+compositional+language+processing%2E+In+Semantics+in+Text+Processing%2E)
23. STEP 2008 Conference Proceedings, pages 239–254. College Publications. [find](https://scholar.google.com/scholar?q=STEP+2008+Conference+Proceedings%2C+pages+239%E2%80%93254%2E+College+Publications%2E)
24. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. [Gender bias in coreference resolution: Evaluation and debiasing methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…

*Updated 24 October 2026*