---
title: "Transcendence — paper by Zhang et al. (2024)"
description: "The authors examine this assumption and define \"transcendence\" as a generative model earning… 37 references, cited by 6 papers here."
authors: ["Edwin Zhang", "Vincent Zhu", "Naomi Saphra", "Anat Kleiman", "Benjamin L. Edelman", "Milind Tambe", "Sham M. Kakade", "Eran Malach"]
year: 2024
venue: "arXiv:2406.11741"
source: "https://arxiv.org/abs/2406.11741"
doi: "10.48550/arxiv.2406.11741"
arxiv: "2406.11741"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-transcendence-generative-models-outperform-experts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-transcendence-generative-models-outperform-experts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / Transcendence: Generative Models Can Outperform The Experts That Train Them

# Transcendence: Generative Models Can Outperform The Experts That Train Them

**Authors:** Edwin Zhang, Vincent Zhu, [Naomi Saphra](https://md.research.elamur.ai/free/en/authors/naomi-saphra/), Anat Kleiman, [Benjamin L. Edelman](https://md.research.elamur.ai/free/en/authors/benjamin-l-edelman/), Milind Tambe, [Sham M. Kakade](https://md.research.elamur.ai/free/en/authors/sham-m-kakade/) and [Eran Malach](https://md.research.elamur.ai/free/en/authors/eran-malach/) · **Year:** 2024 · **Venue:** [arXiv:2406.11741](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.11741) · [PDF](https://arxiv.org/pdf/2406.11741) · [DOI 10.48550/arxiv.2406.11741](https://doi.org/10.48550/arxiv.2406.11741) · [arXiv 2406.11741](https://arxiv.org/abs/2406.11741) · [OpenAlex](https://openalex.org/W4415796469)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/zhang-2024-transcendence-generative-models-outperform-experts/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Models trained with cross-entropy learn to mimic the distribution of their training data. This suggests they cannot outperform the humans who produced that data. The authors examine this assumption and define "transcendence" as a generative model earning higher reward than every expert behind its training set.

They first prove that, in an idealized setting, simply reproducing the averaged mixture of experts never beats the strongest expert. Transcendence becomes possible with low-temperature sampling, which acts like a majority vote across experts and filters out their individual, diverse mistakes ("wisdom of the crowd"). This happens exactly when the arg-max predictor is better than the best expert.

To test the theory, they trained ChessFormer, an autoregressive decoder-only transformer, on human chess transcripts with an upper limit on player rating. At low temperature, some models played above the strongest rating they saw in training. The paper offers a formal explanation for how imitation learning can exceed its teachers, while stressing that the effect comes from removing errors, not from new reasoning.

## Key points
- The paper formalizes transcendence and gives three main results:
  - Plain imitation without temperature changes cannot achieve it (Proposition 1).
  - Low-temperature sampling achieves it if and only if the arg-max predictor beats every expert (Proposition 2).
  - A single expert who errs uniformly at random can be "denoised" this way (Proposition 3).
- ChessFormer 1000 and ChessFormer 1300 surpassed the top Glicko-2 rating in their training data. These ratings are reported with 95% confidence intervals.
- Most of the gain at low temperature comes from large improvements on a small share of game states, likely the decisive ones.
- Transcendence requires diverse training data; models trained on less varied datasets did not exceed their experts.
- Supporting experiments include natural-language question answering and a toy linear model. The toy model was trained on 10K examples labeled by Gaussian-noisy experts. The work appeared at NeurIPS 2024, and code, data and models are at transcendence.eddie.win.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

## Cite

Cite as: Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade and Eran Malach. 2024. Transcendence: Generative Models Can Outperform The Experts That Train Them. arXiv:2406.11741.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024transcendence,
  title = {{Transcendence: Generative Models Can Outperform The Experts That Train Them}},
  author = {Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade and Eran Malach},
  year = {2024},
  doi = {10.48550/arxiv.2406.11741},
  eprint = {2406.11741},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11741}
}
```

- [← previous Synthetic Arabic Medical Dialogues Using Advanced Multi-Agent LLM Techniques](https://md.research.elamur.ai/free/en/papers/almutairi-2024-synthetic-arabic-medical-dialogues-advanced/)
- [next → What Causes the Failure of Explicit to Implicit Discourse Relation Recognition?](https://md.research.elamur.ai/free/en/papers/liu-2024-what-causes-failure-explicit-implicit/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. Breiman. [https://api.semanticscholar.org/CorpusID:47328136](https://api.semanticscholar.org/CorpusID:47328136). URL [↗](https://api.semanticscholar.org/CorpusID:47328136)
2. C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
3. M. Campbell, A. J. Hoane, and F.-h. Hsu. Deep Blue. Artificial Intelligence, 134(1):57–83, Jan. 2002. ISSN 0004-3702. doi: 10.1016/S0004-3702(01)00129-1. — [doi:10.1016/s0004-3702(01)00129-1](https://doi.org/10.1016/s0004-3702%2801%2900129-1) [find](https://scholar.google.com/scholar?q=M%2E+Campbell%2C+A%2E+J%2E+Hoane%2C+and+F%2E%2Dh%2E+Hsu%2E+Deep+Blue%2E+Artificial+Intelligence%2C+134%281%29%3A57%E2%80%9383%2C+Jan%2E+2002%2E+ISSN+0004%2D3702%2E+doi%3A+10%2E1016%2FS0004%2D3702%2801%2900129%2D1%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2Fs0004%2D3702%2801%2900129%2D1%5D%28https%3A%2F%2F)
4. L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+K%2E+Lu%2C+A%2E+Rajeswaran%2C+K%2E+Lee%2C+A%2E+Grover%2C+M%2E+Laskin%2C+P%2E+Abbeel%2C+A%2E+Srinivas%2C+and+I%2E+Mordatch%2E+Decision+transformer%3A+Reinforcement+learning+via+sequence+modeling%2C+2021%2E)
5. X. Feng, Y. Luo, Z. Wang, H. Tang, M. Yang, K. Shao, D. Mguni, Y. Du, and J. Wang. [ChessGPT: Bridging Policy Learning and Language Modeling](https://md.research.elamur.ai/free/en/papers/feng-2023-chessgpt-bridging-policy-learning-language/). Advances in Neural Information Processing Systems, 36:7216–7262, Dec. 2023.
6. Y. Freund and R. E. Schapire. A short introduction to boosting, 1999. [https://api.semanticscholar.org/CorpusID:9621074](https://api.semanticscholar.org/CorpusID:9621074). URL [↗](https://api.semanticscholar.org/CorpusID:9621074)
7. M. E. Glickman. Example of the glicko-2 system. Boston University, 28, 2012. [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Glickman%2E+Example+of+the+glicko%2D2+system%2E+Boston+University%2C+28%2C+2012%2E)
8. M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Janner%2C+Q%2E+Li%2C+and+S%2E+Levine%2E+Offline+reinforcement+learning+as+one+big+sequence+modeling+problem%2C+2021%2E)
9. A. Jiang, L. Soriano Marcolino, A. D. Procaccia, T. Sandholm, N. Shah, and M. Tambe. Diverse randomized agents vote to win. Advances in Neural Information Processing Systems, 27, 2014. [find](https://scholar.google.com/scholar?q=A%2E+Jiang%2C+L%2E+Soriano+Marcolino%2C+A%2E+D%2E+Procaccia%2C+T%2E+Sandholm%2C+N%2E+Shah%2C+and+M%2E+Tambe%2E+Diverse+randomized+agents+vote+to+win%2E+Advances+in+Neural+Information+Processing+Systems%2C+27%2C+2014%2E)
10. S. M. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. URL [https://api.semanticscholar.org/CorpusID:31442909](https://api.semanticscholar.org/CorpusID:31442909). [↗](https://api.semanticscholar.org/CorpusID:31442909)
11. A. Karpathy. NanoGPT. [https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT), 2022. [↗](https://github.com/karpathy/nanoGPT)
12. A. Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint arXiv:2403.15498, 2024. — [arXiv:2403.15498](https://arxiv.org/abs/2403.15498) [find](https://scholar.google.com/scholar?q=A%2E+Karvonen%2E+Emergent+world+models+and+latent+variable+estimation+in+chess%2Dplaying+language+models%2E+arXiv+preprint+arXiv%3A2403%2E15498%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E15498%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E15498%29)
13. D. P. Kingma and J. Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
14. S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Levine%2C+A%2E+Kumar%2C+G%2E+Tucker%2C+and+J%2E+Fu%2E+Offline+reinforcement+learning%3A+Tutorial%2C+review%2C+and+perspectives+on+open+problems%2E+arXiv+preprint+arXiv%3A2005%2E01643%2C+2020%2E)
15. J. Li, E. Zhang, M. Yin, Q. Bai, Y.-X. Wang, and W. Y. Wang. Offline reinforcement learning with closed-form policy improvement operators. In International Conference on Machine Learning, pages 20485–20528. PMLR, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+E%2E+Zhang%2C+M%2E+Yin%2C+Q%2E+Bai%2C+Y%2E%2DX%2E+Wang%2C+and+W%2E+Y%2E+Wang%2E+Offline+reinforcement+learning+with+closed%2Dform+policy+improvement+operators%2E+In+International+Conference+on+Machine+Learning%2C+pages+20485%E2%80%93)
16. T. Lin, L. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Lin%2C+L%2E+Kong%2C+S%2E+U%2E+Stich%2C+and+M%2E+Jaggi%2E+Ensemble+distillation+for+robust+model+fusion+in+federated+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A2351%E2%80%932363%2C+2020%2E)
17. L. S. Marcolino, A. X. Jiang, and M. Tambe. Multi-agent team formation: Diversity beats strength? In IJCAI, volume 13, 2013. [find](https://scholar.google.com/scholar?q=L%2E+S%2E+Marcolino%2C+A%2E+X%2E+Jiang%2C+and+M%2E+Tambe%2E+Multi%2Dagent+team+formation%3A+Diversity+beats+strength%3F+In+IJCAI%2C+volume+13%2C+2013%2E)
18. R. McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson. Aligning Superhuman AI with Human Behavior: Chess as a Model System. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1677–1687, Aug. 2020. doi: 10.1145/3394486.3403219. — [doi:10.1145/3394486.3403219](https://doi.org/10.1145/3394486.3403219) [find](https://scholar.google.com/scholar?q=R%2E+McIlroy%2DYoung%2C+S%2E+Sen%2C+J%2E+Kleinberg%2C+and+A%2E+Anderson%2E+Aligning+Superhuman+AI+with+Human+Behavior%3A+Chess+as+a+Model+System%2E+In+Proceedings+of+the+26th+ACM+SIGKDD+International+Conference+on+Knowledg)
19. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023. [find](https://scholar.google.com/scholar?q=H%2E+B%2E+McMahan%2C+E%2E+Moore%2C+D%2E+Ramage%2C+S%2E+Hampson%2C+and+B%2E+A%2E+y+Arcas%2E+Communication%2Defficient+learning+of+deep+networks+from+decentralized+data%2C+2023%2E)
20. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+A%2E+Rusu%2C+J%2E+Veness%2C+M%2E+G%2E+Bellemare%2C+A%2E+Graves%2C+M%2E+Riedmiller%2C+A%2E+K%2E+Fidjeland%2C+G%2E+Ostrovski%2C+et+al%2E+Human%2Dlevel+control+through+deep+reinforcement+learning%2E+nat)
21. R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=R%2E+Munos%2C+T%2E+Stepleton%2C+A%2E+Harutyunyan%2C+and+M%2E+Bellemare%2E+Safe+and+efficient+off%2Dpolicy+reinforcement+learning%2E+Advances+in+neural+information+processing+systems%2C+29%2C+2016%2E)
22. Y. Nasu. Efficiently Updatable Neural-Network-based Evaluation Functions for Computer Shogi, 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Nasu%2E+Efficiently+Updatable+Neural%2DNetwork%2Dbased+Evaluation+Functions+for+Computer+Shogi%2C+2018%2E)
23. D. Noever, M. Ciolino, and J. Kalin. The Chess Transformer: Mastering Play using Generative Language Models, Sept. 2020. [find](https://scholar.google.com/scholar?q=D%2E+Noever%2C+M%2E+Ciolino%2C+and+J%2E+Kalin%2E+The+Chess+Transformer%3A+Mastering+Play+using+Generative+Language+Models%2C+Sept%2E+2020%2E)
24. A. Odonnat, V. Feofanov, and I. Redko. Leveraging ensemble diversity for robust self-training in the presence of sample selection bias, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Odonnat%2C+V%2E+Feofanov%2C+and+I%2E+Redko%2E+Leveraging+ensemble+diversity+for+robust+self%2Dtraining+in+the+presence+of+sample+selection+bias%2C+2024%2E)
25. Pete. AlphaZero Crushes Stockfish In New 1,000-Game Match. [https://www.chess.com/news/view/updated-alphazero-crushes-stockfish-in-new-1-000-game-match](https://www.chess.com/news/view/updated-alphazero-crushes-stockfish-in-new-1-000-game-match), Dec. 2018. [↗](https://www.chess.com/news/view/updated-alphazero-crushes-stockfish-in-new-1-000-game-match)
26. C. E. Shannon. XXII. Programming a computer for playing chess. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 41(314):256–275, Mar. 1950. ISSN 1941-5982, 1941-5990. doi: 10.1080/14786445008521796. — [doi:10.1080/14786445008521796](https://doi.org/10.1080/14786445008521796) [find](https://scholar.google.com/scholar?q=C%2E+E%2E+Shannon%2E+XXII%2E+Programming+a+computer+for+playing+chess%2E+The+London%2C+Edinburgh%2C+and+Dublin+Philosophical+Magazine+and+Journal+of+Science%2C+41%28314%29%3A256%E2%80%93275%2C+Mar%2E+1950%2E+ISSN+1941%2D5982%2C+1941%2D5990%2E+d)
27. D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/), Dec. 2017.
28. L. Soriano Marcolino, H. Xu, A. Xin Jiang, M. Tambe, and E. Bowring. Give a Hard Problem to a Diverse Team: Exploring Large Action Spaces. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), June 2014. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v28i1.8880. — [doi:10.1609/aaai.v28i1.8880](https://doi.org/10.1609/aaai.v28i1.8880) [find](https://scholar.google.com/scholar?q=L%2E+Soriano+Marcolino%2C+H%2E+Xu%2C+A%2E+Xin+Jiang%2C+M%2E+Tambe%2C+and+E%2E+Bowring%2E+Give+a+Hard+Problem+to+a+Diverse+Team%3A+Exploring+Large+Action+Spaces%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence)
29. The Stockfish developers (see AUTHORS file). Stockfish, 2024. URL [https://stockfishchess.org/](https://stockfishchess.org/). [↗](https://stockfishchess.org/)
30. S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel. Chess as a Testbed for Language Model State Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385–11393, June 2022. doi: 10.1609/aaai.v36i10.21390. — [doi:10.1609/aaai.v36i10.21390](https://doi.org/10.1609/aaai.v36i10.21390) [find](https://scholar.google.com/scholar?q=S%2E+Toshniwal%2C+S%2E+Wiseman%2C+K%2E+Livescu%2C+and+K%2E+Gimpel%2E+Chess+as+a+Testbed+for+Language+Model+State+Tracking%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+36%2C+pages+11385%E2%80%931139)
31. A. Touati, A. Zhang, J. Pineau, and P. Vincent. Stable policy optimization via off-policy divergence regularization. In Conference on Uncertainty in Artificial Intelligence, pages 1328–1337. PMLR, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Touati%2C+A%2E+Zhang%2C+J%2E+Pineau%2C+and+P%2E+Vincent%2E+Stable+policy+optimization+via+off%2Dpolicy+divergence+regularization%2E+In+Conference+on+Uncertainty+in+Artificial+Intelligence%2C+pages+1328%E2%80%931337%2E+PMLR%2C+202)
32. A. Turing. Chess (1953). In B. J. Copeland, editor, The Essential Turing, page 0. Oxford University Press, Sept. 2004. ISBN 978-0-19-825079-1. doi: 10.1093/oso/9780198250791.003.0023. — [doi:10.1093/oso/9780198250791.003.0023](https://doi.org/10.1093/oso/9780198250791.003.0023) [find](https://scholar.google.com/scholar?q=A%2E+Turing%2E+Chess+%281953%29%2E+In+B%2E+J%2E+Copeland%2C+editor%2C+The+Essential+Turing%2C+page+0%2E+Oxford+University+Press%2C+Sept%2E+2004%2E+ISBN+978%2D0%2D19%2D825079%2D1%2E+doi%3A+10%2E1093%2Foso%2F9780198250791%2E003%2E0023%2E+%E2%80%94+%5Bdoi%3A10%2E1093%2Fo)
33. A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470, 2021. [find](https://scholar.google.com/scholar?q=A%2E+N%2E+Uma%2C+T%2E+Fornaciari%2C+D%2E+Hovy%2C+S%2E+Paun%2C+B%2E+Plank%2C+and+M%2E+Poesio%2E+Learning+from+disagreement%3A+A+survey%2E+Journal+of+Artificial+Intelligence+Research%2C+72%3A1385%E2%80%931470%2C+2021%2E)
34. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. [find](https://scholar.google.com/scholar?q=L%2E+Van+der+Maaten+and+G%2E+Hinton%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%2811%29%2C+2008%2E)
35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
36. Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698, 2020. [find](https://scholar.google.com/scholar?q=Q%2E+Xie%2C+M%2E%2DT%2E+Luong%2C+E%2E+Hovy%2C+and+Q%2E+V%2E+Le%2E+Self%2Dtraining+with+noisy+student+improves+imagenet+classification%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recognition%2C+pag)
37. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.

*Updated 3 November 2026*