---
title: "Linearity of Relation Decoding in Transformer Language Models"
description: "The authors test whether this step works like a linear relational embedding (LRE): an affine map from… 45 references, cited by 79 papers here."
authors: ["Evan Hernandez", "Arnab Sen Sharma", "Tal Haklay", "Kevin Meng", "Martin Wattenberg", "Jacob Andreas", "Yonatan Belinkov", "David Bau"]
year: 2023
venue: "arXiv:2308.09124"
source: "https://arxiv.org/abs/2308.09124"
doi: "10.48550/arxiv.2308.09124"
arxiv: "2308.09124"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hernandez-2023-linearity-relation-decoding-transformer-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hernandez-2023-linearity-relation-decoding-transformer-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Linearity of Relation Decoding in Transformer Language Models

# Linearity of Relation Decoding in Transformer Language Models

**Authors:** [Evan Hernandez](https://md.research.elamur.ai/free/en/authors/evan-hernandez/), [Arnab Sen Sharma](https://md.research.elamur.ai/free/en/authors/arnab-sen-sharma/), Tal Haklay, Kevin Meng, [Martin Wattenberg](https://md.research.elamur.ai/free/en/authors/martin-wattenberg/), [Jacob Andreas](https://md.research.elamur.ai/free/en/authors/jacob-andreas/), [Yonatan Belinkov](https://md.research.elamur.ai/free/en/authors/yonatan-belinkov/) and [David Bau](https://md.research.elamur.ai/free/en/authors/david-bau/) · **Year:** 2023 · **Venue:** arXiv:2308.09124 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.09124) · [PDF](https://arxiv.org/pdf/2308.09124) · [DOI 10.48550/arxiv.2308.09124](https://doi.org/10.48550/arxiv.2308.09124) · [arXiv 2308.09124](https://arxiv.org/abs/2308.09124) · [OpenAlex](https://openalex.org/W4386043297)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/hernandez-2023-linearity-relation-decoding-transformer-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Transformer language models clearly store relational facts, such as entity–attribute or word–synonym pairs. Earlier work showed where this information sits, but not how the model turns a subject's hidden state into the predicted object. The authors test whether this step works like a linear relational embedding (LRE): an affine map from an intermediate-layer subject representation to the final-layer state that is decoded into the object token. They estimate each map from the model's Jacobian, averaged over a few example prompts. They then scale up its slope by a constant β, because layer normalization causes the raw Jacobian to understate how strongly the output changes.

Each LRE is judged on two criteria:
- **Faithfulness:** whether it reproduces the model's top prediction.
- **Causality:** whether inverting it lets one edit the subject so the model outputs a different object.

The authors tested 47 relations in GPT and LLaMA models. In 48% of them they found robust LREs, while other relations were predicted correctly but resisted linear decoding. So LMs use a simple, interpretable mechanism alongside more opaque ones.

## Key points
- The dataset has 47 relations covering more than 10k facts. They span factual, commonsense, linguistic and implicit-bias relations, e.g. (banana, has color, yellow).
- The models analyzed include GPT-J, GPT2-xl and LLaMA-13B. The subject state is taken at the subject's last token.
- Results on GPT-J vary widely by relation. Faithfulness is about 0.98 for adjective comparative but only about 0.06 for company CEO.
- Predictions count as correct if the first token matches. This can inflate scores where many objects share a first token, as in person university.
- The authors introduce the "attribute lens," which shows per-layer object distributions for a given relation, in the spirit of Logit Lens.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 126 works · 45,130 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 54 works · 7,237 citations
- [Northeastern University](https://md.research.elamur.ai/free/en/universities/northeastern-university/) — University, Boston, United States · 68 works · 7,316 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov and David Bau. 2023. Linearity of Relation Decoding in Transformer Language Models. arXiv:2308.09124.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hernandez2023linearity,
  title = {{Linearity of Relation Decoding in Transformer Language Models}},
  author = {Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov and David Bau},
  year = {2023},
  doi = {10.48550/arxiv.2308.09124},
  eprint = {2308.09124},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.09124}
}
```

- [← previous Does Writing with Language Models Reduce Content Diversity?](https://md.research.elamur.ai/free/en/papers/padmakumar-2023-writing-language-models-reduce-content/)
- [next → Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment](https://md.research.elamur.ai/free/en/papers/tanwar-2023-multilingual-llms-are-better-cross/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
2. Yonatan Belinkov. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219, 2022.
3. Yonatan Belinkov and James Glass. [Analysis methods in neural language processing: A survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/). Transactions of the Association for Computational Linguistics, 7:49–72, 2019.
4. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. [What do neural machine translation models learn about morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) arXiv preprint arXiv:1704.03471, 2017.
5. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). arXiv preprint arXiv:2303.08112, 2023.
6. Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web. Scientific american, 284(5): 34–43, 2001. [find](https://scholar.google.com/scholar?q=Tim+Berners%2DLee%2C+James+Hendler%2C+and+Ora+Lassila%2E+The+semantic+web%2E+Scientific+american%2C+284%285%29%3A+34%E2%80%9343%2C+2001%2E)
7. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247–1250, 2008. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SIGMOD+i)
8. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. [Crawling the internal knowledge-base of language models](https://md.research.elamur.ai/free/en/papers/cohen-2023-crawling-internal-knowledge-base-language/). arXiv preprint arXiv:2301.12810, 2023.
9. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.[01070, 2018](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/).
10. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+Knowledge+neurons+in+pretrained+transformers%2E+arXiv+preprint+arXiv%3A2104%2E08696%2C+2021%2E)
11. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Shortcutting transformers with linear transformations. arXiv preprint arXiv:2303.09435, 2023. — [arXiv:2303.09435](https://arxiv.org/abs/2303.09435) [find](https://scholar.google.com/scholar?q=Alexander+Yom+Din%2C+Taelin+Karidi%2C+Leshem+Choshen%2C+and+Mor+Geva%2E+Jump+to+conclusions%3A+Shortcutting+transformers+with+linear+transformations%2E+arXiv+preprint+arXiv%3A2303%2E09435%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E09435%5D%28)
12. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. [Probing for semantic evidence of composition by means of simple classification tasks](https://md.research.elamur.ai/free/en/papers/ettinger-2016-probing-semantic-evidence-composition-means/). In Proceedings of the 1st workshop on evaluating vector-space representations for nlp, pp. 134–139, 2016.
13. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+arXiv+preprint+arXiv%3A2012%2E14913%2C+2020%2E)
14. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). arXiv preprint arXiv:2203.14680, 2022.
15. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+arXiv+preprint+arXiv%3A2304%2E14767%2C+2023%2E)
16. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. arXiv preprint arXiv:2301.04213, 2023. [find](https://scholar.google.com/scholar?q=Peter+Hase%2C+Mohit+Bansal%2C+Been+Kim%2C+and+Asma+Ghandeharioun%2E+Does+localization+inform+editing%3F+surprising+differences+in+causality%2Dbased+localization+vs%2E+knowledge+editing+in+language+models%2E+arXiv+pre)
17. Evan Hernandez, Belinda Z Li, and Jacob Andreas. [Inspecting and editing knowledge representations in language models](https://md.research.elamur.ai/free/en/papers/hernandez-2023-inspecting-editing-knowledge-representations-language/). arXiv preprint arXiv:2304.00740, 2023.
18. John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. arXiv preprint arXiv:1909.03368, 2019. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+Designing+and+interpreting+probes+with+control+tasks%2E+arXiv+preprint+arXiv%3A1909%2E03368%2C+2019%2E)
19. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926, 2018. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+Visualisation+and%E2%80%99diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artificial+In)
20. Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2):494–514, 2021. [find](https://scholar.google.com/scholar?q=Shaoxiong+Ji%2C+Shirui+Pan%2C+Erik+Cambria%2C+Pekka+Marttinen%2C+and+S+Yu+Philip%2E+A+survey+on+knowledge+graphs%3A+Representation%2C+acquisition%2C+and+applications%2E+IEEE+Transactions+on+Neural+Networks+and+Learning)
21. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438, 2020.
22. Douglas B Lenat. Cyc: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):33–38, 1995. [find](https://scholar.google.com/scholar?q=Douglas+B+Lenat%2E+Cyc%3A+A+large%2Dscale+investment+in+knowledge+infrastructure%2E+Communications+of+the+ACM%2C+38%2811%29%3A33%E2%80%9338%2C+1995%2E)
23. Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021. [find](https://scholar.google.com/scholar?q=Belinda+Z+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+arXiv+preprint+arXiv%3A2106%2E00737%2C+2021%2E)
24. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+Yang+Liu%2C+and+Xuan+Zhu%2E+Learning+entity+and+relation+embeddings+for+knowledge+graph+completion%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+)
25. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2C+2022%2E)
26. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.16130, 2023. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+Language+models+implement+simple+word2vec%2Dstyle+vector+arithmetic%2E+arXiv+preprint+arXiv%3A2305%2E16130%2C+2023%2E)
27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781, 2013.
28. George A Miller. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11): 39–41, 1995.
29. Marvin Minsky. A framework for representing knowledge, 1974. nostalgebraist. interpreting gpt: the logit lens, 2020. URL [https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens). [↗](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)
30. Alberto Paccanaro and Geoffrey E. Hinton. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions on Knowledge and Data Engineering, 13(2): 232–244, 2001. [find](https://scholar.google.com/scholar?q=Alberto+Paccanaro+and+Geoffrey+E%2E+Hinton%2E+Learning+distributed+representations+of+concepts+using+linear+relational+embedding%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+13%282%29%3A+232%E2%80%93244%2C+2001%2E)
31. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32, 2019.
32. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
33. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
34. Richard H Richens. Preprogramming for mechanical translation. Mechanical Translation, 3(1): 20–25, 1956. [find](https://scholar.google.com/scholar?q=Richard+H+Richens%2E+Preprogramming+for+mechanical+translation%2E+Mechanical+Translation%2C+3%281%29%3A+20%E2%80%9325%2C+1956%2E)
35. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418–5426, 2020. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
36. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986. [find](https://scholar.google.com/scholar?q=David+E+Rumelhart%2C+Geoffrey+E+Hinton%2C+and+Ronald+J+Williams%2E+Learning+representations+by+back%2Dpropagating+errors%2E+nature%2C+323%286088%29%3A533%E2%80%93536%2C+1986%2E)
37. Xing Shi, Inkit Padhi, and Kevin Knight. [Does string-based neural mt learn source syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1526–1534, 2016.
38. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. [Autoprompt: Eliciting knowledge from language models with automatically generated prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222–4235, 2020.
39. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
41. Elena Voita and Ivan Titov. [Information-theoretic probing with minimum description length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/). arXiv preprint arXiv:2003.12298, 2020.
42. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
43. Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12): 2724–2743, 2017. [find](https://scholar.google.com/scholar?q=Quan+Wang%2C+Zhendong+Mao%2C+Bin+Wang%2C+and+Li+Guo%2E+Knowledge+graph+embedding%3A+A+survey+of+approaches+and+applications%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+29%2812%29%3A+2724%E2%80%932743%2C+2017%2E)
44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771, 2019.
45. Jinfa Yang, Yongjie Shi, Xin Tong, Robin Wang, Taiyan Chen, and Xianghua Ying. Improving knowledge graph embedding using affine transformations of entities corresponding to each relation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 508–517, 2021. [find](https://scholar.google.com/scholar?q=Jinfa+Yang%2C+Yongjie+Shi%2C+Xin+Tong%2C+Robin+Wang%2C+Taiyan+Chen%2C+and+Xianghua+Ying%2E+Improving+knowledge+graph+embedding+using+affine+transformations+of+entities+corresponding+to+each+relation%2E+In+Findings+)

*Updated 2 November 2026*