---
title: "The Ghost in the Machine has an American accent: value conflict"
description: "The authors study this problem for GPT-3. 81 references, cited by 65 papers here. Among them: 7."
authors: ["Rebecca L Johnson", "Giada Pistilli", "Natalia Menedez-Gonzalez", "Leslye Denisse Dias Duran", "Enrico Panai", "Julija Kalpokiene", "Donald Jay Bertulfo"]
year: 2022
venue: "arXiv:2203.07785"
source: "https://arxiv.org/abs/2203.07785"
doi: "10.48550/arxiv.2203.07785"
arxiv: "2203.07785"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/johnson-2022-ghost-machine-has-american-accent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/johnson-2022-ghost-machine-has-american-accent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / The Ghost in the Machine has an American accent: value conflict in GPT-3

# The Ghost in the Machine has an American accent: value conflict in GPT-3

**Authors:** Rebecca L Johnson, Giada Pistilli, Natalia Menedez-Gonzalez, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene and Donald Jay Bertulfo · **Year:** 2022 · **Venue:** arXiv:2203.07785 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2203.07785) · [PDF](https://arxiv.org/pdf/2203.07785) · [DOI 10.48550/arxiv.2203.07785](https://doi.org/10.48550/arxiv.2203.07785) · [arXiv 2203.07785](https://arxiv.org/abs/2203.07785) · [OpenAlex](https://openalex.org/W4221164682)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/johnson-2022-ghost-machine-has-american-accent/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models absorb the cultural values of the people whose writing fills their training data. Aligning them with "human values" is therefore hard, because those values differ across societies and sometimes clash legitimately. The authors study this problem for GPT-3.

First, they compare the composition of its training corpus, dominated by English web text, with global distributions of languages and internet access. They also compare it with national value profiles from the World Values Survey (WVS). Second, they probe the model in an exploratory, qualitative way. GPT-3 receives value-laden passages from several languages and countries, including some whose stance diverges from prevailing US opinion. The generated outputs are then checked for shifts in the values expressed. When such shifts occurred, they often moved toward positions the WVS reports as dominant in the United States.

To interpret these "value mutations", the paper draws on moral value pluralism (MVP), especially Nagel's typology of conflicts and values. The authors argue this lens can help developers spot cultural skew and guide fine-tuning toward more pluralist alignment.

## Key points
- About 60% of GPT-3's training data comes from a filtered CommonCrawl. The remainder includes a scraped-links dataset, two books corpora and English-language Wikipedia, and over 93% of the total is English.
- The method is stress testing with value-rich input texts, judged against WVS national profiles. Altered values in outputs frequently matched dominant US views, which gives the paper its "American accent" title.
- Following Nagel, the paper separates strong conflicts (values that condemn each other) from weak ones (incompatibilities a community can tolerate). It uses five value categories: obligations, rights, utility, perfectionist ends and private commitments.
- The authors propose a sixth category: fair sharing of collective responsibility for global issues such as the environment and sustainability.
- They recommend anchoring LLMs to basic rights, for example the Universal Declaration of Human Rights, while accommodating conflicting cultural value systems. They also call for an MVP roadmap for fine-tuning and careful human-in-the-loop design. The study is framed as exploratory "slow research" without a quantitative benchmark.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Rebecca L Johnson, Giada Pistilli, Natalia Menedez-Gonzalez, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene and Donald Jay Bertulfo. 2022. The Ghost in the Machine has an American accent: value conflict in GPT-3. arXiv:2203.07785.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{johnson2022ghost,
  title = {{The Ghost in the Machine has an American accent: value conflict in GPT-3}},
  author = {Rebecca L Johnson, Giada Pistilli, Natalia Menedez-Gonzalez, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene and Donald Jay Bertulfo},
  year = {2022},
  doi = {10.48550/arxiv.2203.07785},
  eprint = {2203.07785},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.07785}
}
```

- [← previous InPars: Data Augmentation for Information Retrieval using Large Language Models](https://md.research.elamur.ai/free/en/papers/bonifacio-2022-inpars-data-augmentation-information-retrieval/)
- [next → Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://md.research.elamur.ai/free/en/papers/sartran-2022-transformer-grammars-augmenting-transformer-language/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

81 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Floridi, L. and Chiriatti, M. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30, 4 (2020), 681-694. [find](https://scholar.google.com/scholar?q=%5B1%5D+Floridi%2C+L%2E+and+Chiriatti%2C+M%2E+GPT%2D3%3A+Its+nature%2C+scope%2C+limits%2C+and+consequences%2E+Minds+and+Machines%2C+30%2C+4+%282020%29%2C+681%2D694%2E)
2. [2] Abid, A., Farooqi, M. and Zou, J. [Persistent anti-muslim bias in large language models](https://md.research.elamur.ai/free/en/papers/abid-2021-persistent-anti-muslim-bias-large/). arXiv:2101.05783 (2021).
3. [3] Solaiman, I. and Dennison, C. [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://md.research.elamur.ai/free/en/papers/solaiman-2021-process-adapting-language-models-society/). arXiv:2106.10328 (2021).
4. [4] Tamkin, A., Brundage, M., Clark, J. and Ganguli, D. [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://md.research.elamur.ai/free/en/papers/tamkin-2021-understanding-capabilities-limitations-societal-impact/). arXiv:2102.02503 (2021).
5. [5] Hofstede, G. Culture's consequences: Comparing values, behaviors, institutions and organizations across nations. Sage publications, 2001. [find](https://scholar.google.com/scholar?q=%5B5%5D+Hofstede%2C+G%2E+Culture%27s+consequences%3A+Comparing+values%2C+behaviors%2C+institutions+and+organizations+across+nations%2E+Sage+publications%2C+2001%2E)
6. [6] Rokeach, M. Understanding human values. Simon and Schuster, 2008. [find](https://scholar.google.com/scholar?q=%5B6%5D+Rokeach%2C+M%2E+Understanding+human+values%2E+Simon+and+Schuster%2C+2008%2E)
7. [7] Jonkers, P. How to Respond to Conflicts Over Value Pluralism? Journal of Nationalism, Memory, and Language-Politics (2019). [find](https://scholar.google.com/scholar?q=%5B7%5D+Jonkers%2C+P%2E+How+to+Respond+to+Conflicts+Over+Value+Pluralism%3F+Journal+of+Nationalism%2C+Memory%2C+and+Language%2DPolitics+%282019%29%2E)
8. [8] Yudkowsky, E. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker (2016). [find](https://scholar.google.com/scholar?q=%5B8%5D+Yudkowsky%2C+E%2E+The+AI+alignment+problem%3A+why+it+is+hard%2C+and+where+to+start%2E+Symbolic+Systems+Distinguished+Speaker+%282016%29%2E)
9. [9] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A. and Brunskill, E. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv:2108.07258 (2021).
10. [10] Hume, D. A treatise of human nature. Clarendon Press, 1896. [find](https://scholar.google.com/scholar?q=%5B10%5D+Hume%2C+D%2E+A+treatise+of+human+nature%2E+Clarendon+Press%2C+1896%2E)
11. [11] Gabriel, I. Artificial intelligence, values, and alignment. Minds and Machines, 30, 3 (2020), 411-437. [find](https://scholar.google.com/scholar?q=%5B11%5D+Gabriel%2C+I%2E+Artificial+intelligence%2C+values%2C+and+alignment%2E+Minds+and+Machines%2C+30%2C+3+%282020%29%2C+411%2D437%2E)
12. [12] Russell, S. Human-compatible artificial intelligence. Human-Like Machine Intelligence, Oxford University Press Oxford, 2021. [find](https://scholar.google.com/scholar?q=%5B12%5D+Russell%2C+S%2E+Human%2Dcompatible+artificial+intelligence%2E+Human%2DLike+Machine+Intelligence%2C+Oxford+University+Press+Oxford%2C+2021%2E)
13. [13] Wiener, N. Some Moral and Technical Consequences of Automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers. Science, 131, 3410 (1960), 1355-1358. [find](https://scholar.google.com/scholar?q=%5B13%5D+Wiener%2C+N%2E+Some+Moral+and+Technical+Consequences+of+Automation%3A+As+machines+learn+they+may+develop+unforeseen+strategies+at+rates+that+baffle+their+programmers%2E+Science%2C+131%2C+3410+%281960%29%2C+1355%2D13)
14. [14] Habermas, J. Moral consciousness and communicative action. MIT press, 1990. [find](https://scholar.google.com/scholar?q=%5B14%5D+Habermas%2C+J%2E+Moral+consciousness+and+communicative+action%2E+MIT+press%2C+1990%2E)
15. [15] Barthes, R. Elements of Semiology. 1964. [find](https://scholar.google.com/scholar?q=%5B15%5D+Barthes%2C+R%2E+Elements+of+Semiology%2E+1964%2E)
16. New York: Hill & Wang (1967), 41-47. [find](https://scholar.google.com/scholar?q=New+York%3A+Hill+%26+Wang+%281967%29%2C+41%2D47%2E)
17. [16] Peeters, B. Language and cultural values. International Journal of Language and Culture, 2, 2 (2015), 133-141. [find](https://scholar.google.com/scholar?q=%5B16%5D+Peeters%2C+B%2E+Language+and+cultural+values%2E+International+Journal+of+Language+and+Culture%2C+2%2C+2+%282015%29%2C+133%2D141%2E)
18. [18] Stearn, R. L. Technology and world trade : proceedings of a symposium, November 16-17, 1966. U.S. Department of Commerce, National Bureau of Standards, Washington, 1967. [find](https://scholar.google.com/scholar?q=%5B18%5D+Stearn%2C+R%2E+L%2E+Technology+and+world+trade+%3A+proceedings+of+a+symposium%2C+November+16%2D17%2C+1966%2E+U%2ES%2E+Department+of+Commerce%2C+National+Bureau+of+Standards%2C+Washington%2C+1967%2E)
19. [19] Lakoff, G. and Johnson, M. Metaphors we live by. University of Chicago press, 2008. [find](https://scholar.google.com/scholar?q=%5B19%5D+Lakoff%2C+G%2E+and+Johnson%2C+M%2E+Metaphors+we+live+by%2E+University+of+Chicago+press%2C+2008%2E)
20. [20] Peeters, B. Tall poppies and egalitarianism in Australian discourse: From key word to cultural value. English worldwide, 25, 1 (2004), 1-25. [find](https://scholar.google.com/scholar?q=%5B20%5D+Peeters%2C+B%2E+Tall+poppies+and+egalitarianism+in+Australian+discourse%3A+From+key+word+to+cultural+value%2E+English+worldwide%2C+25%2C+1+%282004%29%2C+1%2D25%2E)
21. [21] Nieminen, T. The nail that sticks out: the practice of individuality in the East Asian classroom (2015). [find](https://scholar.google.com/scholar?q=%5B21%5D+Nieminen%2C+T%2E+The+nail+that+sticks+out%3A+the+practice+of+individuality+in+the+East+Asian+classroom+%282015%29%2E)
22. [22] Walker, J., Tepper, S. J. and Gilovich, T. People are more tolerant of inequality when it is expressed in terms of individuals rather than groups at the top. Proceedings of the National Academy of Sciences, 118, 43 (2021). [find](https://scholar.google.com/scholar?q=%5B22%5D+Walker%2C+J%2E%2C+Tepper%2C+S%2E+J%2E+and+Gilovich%2C+T%2E+People+are+more+tolerant+of+inequality+when+it+is+expressed+in+terms+of+individuals+rather+than+groups+at+the+top%2E+Proceedings+of+the+National+Academy+o)
23. [23] Stephens, G. J., Silbert, L. J. and Hasson, U. Speaker– listener neural coupling underlies successful communication. Proceedings of the National Academy of Sciences, 107, 32 (2010), 14425-14430. [find](https://scholar.google.com/scholar?q=%5B23%5D+Stephens%2C+G%2E+J%2E%2C+Silbert%2C+L%2E+J%2E+and+Hasson%2C+U%2E+Speaker%E2%80%93+listener+neural+coupling+underlies+successful+communication%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+107%2C+32+%282010%29%2C+14425%2D14430%2E)
24. [24] Clark, H. H. Using language. Cambridge university press, 1996. [find](https://scholar.google.com/scholar?q=%5B24%5D+Clark%2C+H%2E+H%2E+Using+language%2E+Cambridge+university+press%2C+1996%2E)
25. [25] Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V. and Kalai, A. T. [Man is to computer programmer as woman is to homemaker? debiasing word embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/). Advances in neural information processing systems, 29 (2016).
26. [26] Izzidien, A. Word vector embeddings hold social ontological relations capable of reflecting meaningful fairness assessments. AI & SOCIETY, 37, 1 (2022/03/01 2022), 299-318. [find](https://scholar.google.com/scholar?q=%5B26%5D+Izzidien%2C+A%2E+Word+vector+embeddings+hold+social+ontological+relations+capable+of+reflecting+meaningful+fairness+assessments%2E+AI+%26+SOCIETY%2C+37%2C+1+%282022%2F03%2F01+2022%29%2C+299%2D318%2E)
27. [27] Topal, M. O., Bas, A. and van Heerden, I. Exploring transformers in natural language generation: Gpt, bert, and xlnet. arXiv:2102.08036 (2021). — [arXiv:2102.08036](https://arxiv.org/abs/2102.08036) [find](https://scholar.google.com/scholar?q=%5B27%5D+Topal%2C+M%2E+O%2E%2C+Bas%2C+A%2E+and+van+Heerden%2C+I%2E+Exploring+transformers+in+natural+language+generation%3A+Gpt%2C+bert%2C+and+xlnet%2E+arXiv%3A2102%2E08036+%282021%29%2E+%E2%80%94+%5BarXiv%3A2102%2E08036%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E080)
28. [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the Advances in neural information processing systems, 2017
29. [29] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. and Galstyan, A. [A survey on bias and fairness in machine learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/). ACM Computing Surveys (CSUR), 54, 6 (2021), 1-35.
30. [30] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B. and Kasirzadeh, A. [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/). arXiv:2112.04359 (2021).
31. [31] Hymes, D. Models of the interaction of language and social life: toward a descriptive theory. Intercultural discourse and communication: The essential readings, 1 (2005), 4-16. [find](https://scholar.google.com/scholar?q=%5B31%5D+Hymes%2C+D%2E+Models+of+the+interaction+of+language+and+social+life%3A+toward+a+descriptive+theory%2E+Intercultural+discourse+and+communication%3A+The+essential+readings%2C+1+%282005%29%2C+4%2D16%2E)
32. [32] Fishman, J. What Do You Lose When You Lose Your Language? (1996). [find](https://scholar.google.com/scholar?q=%5B32%5D+Fishman%2C+J%2E+What+Do+You+Lose+When+You+Lose+Your+Language%3F+%281996%29%2E)
33. [17] Lakoff, G. Women, fire, and dangerous things : what categories reveal about the mind. University of Chicago Press, Chicago, 1987. [find](https://scholar.google.com/scholar?q=%5B17%5D+Lakoff%2C+G%2E+Women%2C+fire%2C+and+dangerous+things+%3A+what+categories+reveal+about+the+mind%2E+University+of+Chicago+Press%2C+Chicago%2C+1987%2E)
34. [33] Dewey, J. The logic of judgments of practise. The Journal of philosophy, psychology and scientific methods, 12, 19 (1915), 505-523. [find](https://scholar.google.com/scholar?q=%5B33%5D+Dewey%2C+J%2E+The+logic+of+judgments+of+practise%2E+The+Journal+of+philosophy%2C+psychology+and+scientific+methods%2C+12%2C+19+%281915%29%2C+505%2D523%2E)
35. [34] Jean, S. Les valeurs du temps présent: une enquête européenne. Paris: Presses universitaires de France (1983). [find](https://scholar.google.com/scholar?q=%5B34%5D+Jean%2C+S%2E+Les+valeurs+du+temps+pr%C3%A9sent%3A+une+enqu%C3%AAte+europ%C3%A9enne%2E+Paris%3A+Presses+universitaires+de+France+%281983%29%2E)
36. [35] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G. and Askell, A. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv:2005.14165 (2020). the Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021
37. [48] Lucy, L. and Bamman, D. [Gender and Representation Bias in GPT-3 Generated Stories](https://md.research.elamur.ai/free/en/papers/lucy-2021-gender-representation-bias-gpt-3/). In Proceedings of the The Third Workshop on Narrative Understanding, 2021
38. [49] Fazelpour, S. and De-Arteaga, M. Diversity in Sociotechnical Machine Learning Systems. arXiv:2107.09163 (2021). — [arXiv:2107.09163](https://arxiv.org/abs/2107.09163) [find](https://scholar.google.com/scholar?q=%5B49%5D+Fazelpour%2C+S%2E+and+De%2DArteaga%2C+M%2E+Diversity+in+Sociotechnical+Machine+Learning+Systems%2E+arXiv%3A2107%2E09163+%282021%29%2E+%E2%80%94+%5BarXiv%3A2107%2E09163%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2107%2E09163%29)
39. [36] CommonCrawl, Distribution of Languages. 2021, [https://commoncrawl.github.io/cc-crawl-statistics/plots/languages](https://commoncrawl.github.io/cc-crawl-statistics/plots/languages) [↗](https://commoncrawl.github.io/cc-crawl-statistics/plots/languages)
40. [50] Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L. and Du, Y. [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv:2201.08239 (2022).
41. [37] Eberhard David M., S. G. F., Fenning Charles D.,, Summary by language size. Ethnologue Languages of the World, 2019, [http://www.ethnologue.com](http://www.ethnologue.com) [↗](http://www.ethnologue.com)
42. [51] Galston, W. A. Liberal pluralism: The implications of value pluralism for political theory and practice. Cambridge University Press, 2002. [find](https://scholar.google.com/scholar?q=%5B51%5D+Galston%2C+W%2E+A%2E+Liberal+pluralism%3A+The+implications+of+value+pluralism+for+political+theory+and+practice%2E+Cambridge+University+Press%2C+2002%2E)
43. [38] Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the The 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021 [find](https://scholar.google.com/scholar?q=%5B38%5D+Bender%2C+E%2E+M%2E%2C+Gebru%2C+T%2E%2C+McMillan%2DMajor%2C+A%2E+and+Shmitchell%2C+S%2E+On+the+Dangers+of+Stochastic+Parrots%3A+Can+Language+Models+Be+Too+Big%3F+In+Proceedings+of+the+The+2021+ACM+Conference+on+Fairness%2C+Ac)
44. [52] Berlin, I. Four essays on liberty. (1969). [find](https://scholar.google.com/scholar?q=%5B52%5D+Berlin%2C+I%2E+Four+essays+on+liberty%2E+%281969%29%2E)
45. [39] Johnson, J., Internet usage worldwide - statistics & facts. 2021, [https://www.statista.com/topics/1145/internet-usage-worldwide/](https://www.statista.com/topics/1145/internet-usage-worldwide/) [↗](https://www.statista.com/topics/1145/internet-usage-worldwide/)
46. [54] McSweeney, B. The essentials of scholarship: A reply to Geert Hofstede. Human relations, 55, 11 (2002), 1363-1372. [find](https://scholar.google.com/scholar?q=%5B54%5D+McSweeney%2C+B%2E+The+essentials+of+scholarship%3A+A+reply+to+Geert+Hofstede%2E+Human+relations%2C+55%2C+11+%282002%29%2C+1363%2D1372%2E)
47. [40] International Telecommunication Union, Individuals using the Internet (% of population) - Sub-Saharan Africa. The World Bank, — [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) [https://data.worldbank.org/indicator/IT.NET.USER.ZS?locati](https://data.worldbank.org/indicator/IT.NET.USER.ZS?locati) ons=ZG [41] Nadeem, M., Bethke, A. and Reddy, S. Stereoset: Measuring stereotypical bias in pretrained languag… [↗](https://data.worldbank.org/indicator/IT.NET.USER.ZS?locati)
48. [42] Lu, K., Mardziel, P., Wu, F., Amancharla, P. and Datta, A. [Gender bias in neural natural language processing](https://md.research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/). Logic, Language, and Security, Springer, 2020.
49. [43] Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y. and Shieber, S. M. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Proceedings of the NeurIPS, 2020 [find](https://scholar.google.com/scholar?q=%5B43%5D+Vig%2C+J%2E%2C+Gehrmann%2C+S%2E%2C+Belinkov%2C+Y%2E%2C+Qian%2C+S%2E%2C+Nevo%2C+D%2E%2C+Singer%2C+Y%2E+and+Shieber%2C+S%2E+M%2E+Investigating+Gender+Bias+in+Language+Models+Using+Causal+Mediation+Analysis%2E+In+Proceedings+of+the+NeurIPS%2C)
50. [44] Russell, S. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019. [find](https://scholar.google.com/scholar?q=%5B44%5D+Russell%2C+S%2E+Human+compatible%3A+Artificial+intelligence+and+the+problem+of+control%2E+Penguin%2C+2019%2E)
51. [45] Christian, B. The Alignment Problem: How Can Machines Learn Human Values? Atlantic Books, 2021. [find](https://scholar.google.com/scholar?q=%5B45%5D+Christian%2C+B%2E+The+Alignment+Problem%3A+How+Can+Machines+Learn+Human+Values%3F+Atlantic+Books%2C+2021%2E)
52. [46] Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W. and Gupta, R. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the The 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021 [find](https://scholar.google.com/scholar?q=%5B46%5D+Dhamala%2C+J%2E%2C+Sun%2C+T%2E%2C+Kumar%2C+V%2E%2C+Krishna%2C+S%2E%2C+Pruksachatkun%2C+Y%2E%2C+Chang%2C+K%2E%2DW%2E+and+Gupta%2C+R%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+In+Proceedings+of+the)
53. [47] Guo, W. and Caliskan, A. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of [53] Tausch, A. Hofstede, Inglehart and Beyond. New Directions in Empirical Global Value Research. SSRN (2015). [find](https://scholar.google.com/scholar?q=%5B47%5D+Guo%2C+W%2E+and+Caliskan%2C+A%2E+Detecting+emergent+intersectional+biases%3A+Contextualized+word+embeddings+contain+a+distribution+of+human%2Dlike+biases%2E+In+Proceedings+of+%5B53%5D+Tausch%2C+A%2E+Hofstede%2C+Inglehar)
54. [55] Schwartz, S. H. and Bardi, A. Value hierarchies across cultures: Taking a similarities perspective. Journal of cross-cultural Psychology, 32, 3 (2001), 268-290. [find](https://scholar.google.com/scholar?q=%5B55%5D+Schwartz%2C+S%2E+H%2E+and+Bardi%2C+A%2E+Value+hierarchies+across+cultures%3A+Taking+a+similarities+perspective%2E+Journal+of+cross%2Dcultural+Psychology%2C+32%2C+3+%282001%29%2C+268%2D290%2E)
55. [56] World Values Survey, WVS wave 7. 2022, [https://www.worldvaluessurvey.org/WVSContents.jsp](https://www.worldvaluessurvey.org/WVSContents.jsp) [↗](https://www.worldvaluessurvey.org/WVSContents.jsp)
56. [57] Weber, M. and Kalberg, S. The Protestant ethic and the spirit of capitalism. Routledge, 2013. [find](https://scholar.google.com/scholar?q=%5B57%5D+Weber%2C+M%2E+and+Kalberg%2C+S%2E+The+Protestant+ethic+and+the+spirit+of+capitalism%2E+Routledge%2C+2013%2E)
57. [58] Gray, M. L. and Suri, S. Ghost work: how to stop Silicon Valley from building a new global underclass. Eamon Dolan Books, 2019. [find](https://scholar.google.com/scholar?q=%5B58%5D+Gray%2C+M%2E+L%2E+and+Suri%2C+S%2E+Ghost+work%3A+how+to+stop+Silicon+Valley+from+building+a+new+global+underclass%2E+Eamon+Dolan+Books%2C+2019%2E)
58. [59] Davani, A. M., Díaz, M. and Prabhakaran, V. [Dealing with disagreements: Looking beyond the majority vote in subjective annotations](https://md.research.elamur.ai/free/en/papers/davani-2022-dealing-disagreements-looking-beyond-majority/). arXiv:2110.05719 (2021).
59. [60] Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M. and Dean, J. [Carbon emissions and large neural network training](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/). arXiv:2104.10350 (2021).
60. [61] Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M. and Le, Q. V. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv:2109.01652 (2021).
61. [62] Nikolich, A. and Puchkova, A. Fine-tuning GPT-3 for Russian Text Summarization. arXiv:2108.03502 (2021). — [arXiv:2108.03502](https://arxiv.org/abs/2108.03502) [find](https://scholar.google.com/scholar?q=%5B62%5D+Nikolich%2C+A%2E+and+Puchkova%2C+A%2E+Fine%2Dtuning+GPT%2D3+for+Russian+Text+Summarization%2E+arXiv%3A2108%2E03502+%282021%29%2E+%E2%80%94+%5BarXiv%3A2108%2E03502%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2108%2E03502%29)
62. [63] Moradi, M., Blagec, K., Haberl, F. and Samwald, M. GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain. arXiv:2109.02555 (2021). — [arXiv:2109.02555](https://arxiv.org/abs/2109.02555) [find](https://scholar.google.com/scholar?q=%5B63%5D+Moradi%2C+M%2E%2C+Blagec%2C+K%2E%2C+Haberl%2C+F%2E+and+Samwald%2C+M%2E+GPT%2D3+Models+are+Poor+Few%2DShot+Learners+in+the+Biomedical+Domain%2E+arXiv%3A2109%2E02555+%282021%29%2E+%E2%80%94+%5BarXiv%3A2109%2E02555%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E02555)
63. [64] Kirstain, Y., Lewis, P., Riedel, S. and Levy, O. A Few More Examples May Be Worth Billions of Parameters. arXiv:2110.04374 (2021). [find](https://scholar.google.com/scholar?q=%5B64%5D+Kirstain%2C+Y%2E%2C+Lewis%2C+P%2E%2C+Riedel%2C+S%2E+and+Levy%2C+O%2E+A+Few+More+Examples+May+Be+Worth+Billions+of+Parameters%2E+arXiv%3A2110%2E04374+%282021%29%2E)
64. [65] Reynolds, L. and McDonell, K. [Prompt programming for large language models: Beyond the few-shot paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/). In Proceedings of the The 2021 CHI Conference on Human Factors in Computing Systems., 2021
65. [66] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R. and Young, S. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446 (2021). [find](https://scholar.google.com/scholar?q=%5B66%5D+Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E+and+Young%2C+S%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+g)
66. [67] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K. and Ray, A. InstructGPT: [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). [https://github.com/openai/following-instructions-human-feedback](https://github.com/openai/following-instructions-human-feedback) (2022). [↗](https://github.com/openai/following-instructions-human-feedback)
67. [68] Prabhakaran, V., Davani, A. M. and Díaz, M. [On releasing Annotator-Level labels and information in datasets](https://md.research.elamur.ai/free/en/papers/prabhakaran-2021-releasing-annotator-level-labels-information/). arXiv:2110.05699 (2021).
68. [69] Newman, J. and Head, B. The national context of wicked problems: Comparing policies on gun violence in the US, Canada, and Australia. Journal of comparative policy analysis: research and practice, 19, 1 (2017), 40-53. [find](https://scholar.google.com/scholar?q=%5B69%5D+Newman%2C+J%2E+and+Head%2C+B%2E+The+national+context+of+wicked+problems%3A+Comparing+policies+on+gun+violence+in+the+US%2C+Canada%2C+and+Australia%2E+Journal+of+comparative+policy+analysis%3A+research+and+practice)
69. [79] Freedman, J. Secularism as a barrier to integration? The French dilemma. International Migration, 42, 3 (2004), 5-27. [find](https://scholar.google.com/scholar?q=%5B79%5D+Freedman%2C+J%2E+Secularism+as+a+barrier+to+integration%3F+The+French+dilemma%2E+International+Migration%2C+42%2C+3+%282004%29%2C+5%2D27%2E)
70. [80] United Nations General Assembly Convention on the elimination of all forms of discrimination against women. 2006, [https://www.ohchr.org/en/hrbodies/cedaw/pages/cedawindex](https://www.ohchr.org/en/hrbodies/cedaw/pages/cedawindex). aspx [81] Malcolm, X., Baldwin, J. and McCummins, L. The ballot or the bullet. [↗](https://www.ohchr.org/en/hrbodies/cedaw/pages/cedawindex)
71. Paul Winley Records, 1987. [find](https://scholar.google.com/scholar?q=Paul+Winley+Records%2C+1987%2E)
72. [82] Tarana Burke, Power of Women Speech. Iowa State University, 2018, [https://awpc.cattcenter.iastate.edu/2018/09/26/full-power-of-women-speech-april-13-2018/](https://awpc.cattcenter.iastate.edu/2018/09/26/full-power-of-women-speech-april-13-2018/) [↗](https://awpc.cattcenter.iastate.edu/2018/09/26/full-power-of-women-speech-april-13-2018/)
73. [83] UNESCO Draft text of the recommendation on the ethics of artificial intelligence., 2021, [https://unesdoc.unesco.org/ark:/48223/pf0000377897](https://unesdoc.unesco.org/ark:/48223/pf0000377897) [↗](https://unesdoc.unesco.org/ark:/48223/pf0000377897)
74. [84] Nagel, T. Mortal questions. Cambridge University Press, Cambridge [Eng.] ;, 1979. [find](https://scholar.google.com/scholar?q=%5B84%5D+Nagel%2C+T%2E+Mortal+questions%2E+Cambridge+University+Press%2C+Cambridge+%5BEng%2E%5D+%3B%2C+1979%2E)
75. [85] Nagel, T. The Fragmentation of Value. Mortal Questions, Cambridge University Press, Cambridge, 2012. [find](https://scholar.google.com/scholar?q=%5B85%5D+Nagel%2C+T%2E+The+Fragmentation+of+Value%2E+Mortal+Questions%2C+Cambridge+University+Press%2C+Cambridge%2C+2012%2E)
76. [70] Small Arms Survey, Global Fireams Holdings. [https://www.smallarmssurvey.org/database/global-firearms-holdings](https://www.smallarmssurvey.org/database/global-firearms-holdings) [71] Research, E. The Essential Report. 2016. [↗](https://www.smallarmssurvey.org/database/global-firearms-holdings)
77. [72] Ipsos. What do you think is more important?...To protect the right of Americans to own guns or control gun ownership? , Pew Research Center for the People & the Press, U.S.A, 2019. [find](https://scholar.google.com/scholar?q=%5B72%5D+Ipsos%2E+What+do+you+think+is+more+important%3F%2E%2E%2ETo+protect+the+right+of+Americans+to+own+guns+or+control+gun+ownership%3F+%2C+Pew+Research+Center+for+the+People+%26+the+Press%2C+U%2ES%2EA%2C+2019%2E)
78. [73] Australasian Police Ministers’ COUNCIL (APMC) National Firearms Agreement. 1996, [74] De Beauvoir, S. “Introduction” to The Second Sex. na, 1997. [find](https://scholar.google.com/scholar?q=%5B73%5D+Australasian+Police+Ministers%E2%80%99+COUNCIL+%28APMC%29+National+Firearms+Agreement%2E+1996%2C+%5B74%5D+De+Beauvoir%2C+S%2E+%E2%80%9CIntroduction%E2%80%9D+to+The+Second+Sex%2E+na%2C+1997%2E)
79. [75] Jones, M., \#IWD2019: Perceptions of violence against women in France and the United States. 2019, [https://www.ipsos.com/en/iwd2019-perceptions-violence-against-women-france-and-united-states](https://www.ipsos.com/en/iwd2019-perceptions-violence-against-women-france-and-united-states) [↗](https://www.ipsos.com/en/iwd2019-perceptions-violence-against-women-france-and-united-states)
80. [76] Merkel, A. Sommerpressekonferenz von Bundeskanzlerin Merkel. Thema: Aktuelle Themen der Innen-und Außenpolitik. August, 2015, [77] Stasi, B. Rapport au Président de la République. Commission de réflexion sur l'application du principe de laïcité dans la republique., France, 2003. [find](https://scholar.google.com/scholar?q=%5B76%5D+Merkel%2C+A%2E+Sommerpressekonferenz+von+Bundeskanzlerin+Merkel%2E+Thema%3A+Aktuelle+Themen+der+Innen%2Dund+Au%C3%9Fenpolitik%2E+August%2C+2015%2C+%5B77%5D+Stasi%2C+B%2E+Rapport+au+Pr%C3%A9sident+de+la+R%C3%A9publique%2E+Commission+de+r)
81. [78] Hauser, J. Education, secularism, and illiberalism: Marginalisation of Muslims by the French state. French Cultural Studies, 32, 2 (2021), 149-162. [find](https://scholar.google.com/scholar?q=%5B78%5D+Hauser%2C+J%2E+Education%2C+secularism%2C+and+illiberalism%3A+Marginalisation+of+Muslims+by+the+French+state%2E+French+Cultural+Studies%2C+32%2C+2+%282021%29%2C+149%2D162%2E)

*Updated 2 November 2026*