---
title: "JuICe — paper by Agashe et al. (2019) — summary and ideas"
description: "The paper addresses code generation in interactive computing, where a programmer… 60% BLEU. 35 references, cited by 28 papers here."
authors: ["Rajas Agashe", "Srinivasan Iyer", "Luke Zettlemoyer"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1546/"
doi: "10.18653/v1/d19-1546"
arxiv: "1910.02216"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agashe-2019-juice-large-scale-distantly-supervised/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agashe-2019-juice-large-scale-distantly-supervised/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

# JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

**Authors:** Rajas Agashe, [Srinivasan Iyer](https://md.research.elamur.ai/free/en/authors/srinivasan-iyer/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1546/) · [PDF](https://aclanthology.org/D19-1546.pdf) · [DOI 10.18653/v1/d19-1546](https://doi.org/10.18653/v1/d19-1546) · [arXiv 1910.02216](https://arxiv.org/abs/1910.02216) · [OpenAlex](https://openalex.org/W2970004377)

**Citations:** 47 citations (OpenAlex, 2026-10-14) · [28](https://md.research.elamur.ai/free/en/papers/agashe-2019-juice-large-scale-distantly-supervised/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses code generation in interactive computing, where a programmer writes code cell by cell in Jupyter notebooks. Here the next code cell must be inferred from the nearest markdown instruction and from the earlier NL and code cells. The authors built JuICe from roughly 659K public GitHub notebooks, written in Python with English markdown, and extracted about 1.5M target code cells along with their preceding context. This large, noisy collection serves as distant supervision. For evaluation, they used nbgrader assignment notebooks, which instructors write with carefully crafted instructions and reference solutions, to create a human-curated dev/test set.

Two tasks are defined: predicting the set of API calls in a cell, and generating the full cell. Seq2seq LSTM models with sub-word units beat retrieval baselines. Both more training data and more context cells improve results, though the benefit stops growing beyond about three context cells. Performance remains well below the estimated human level. This makes JuICe a demanding benchmark for context-aware, open-domain code generation.

## Key points
- Training set: 1,518,049 examples from ~659K GitHub notebooks created before May 2019. Notebooks with more than three times as many code cells as markdown cells were filtered out.
- Curated evaluation data: 1,744 dev and 1,981 test examples (3,725 in total), drawn from 13,905 collected nbgrader notebooks. About 62% of these examples depend on variables or methods defined in earlier cells.
- Metrics: full code generation uses EM and corpus BLEU, while the API sequence task uses set-based precision/recall. The best baseline is an LSTM given 3 context cells and the full training data.
- Gains from additional data level off after about 200K examples. Estimated human performance is about 60% BLEU / 23% EM for code generation and 84% precision / 85% recall for API sequences.
- Error analysis of 50 incorrect outputs found that 39% misread the NL intent entirely, 26% failed at contextual reasoning, and 15% were semantically equivalent to the reference but worded differently.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Rajas Agashe, Srinivasan Iyer and Luke Zettlemoyer. 2019. JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{agashe2019juice,
  title = {{JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation}},
  author = {Rajas Agashe, Srinivasan Iyer and Luke Zettlemoyer},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1546},
  eprint = {1910.02216},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1546}
}
```

- [← previous Automatic Gender Identification and Reinflection in Arabic](https://md.research.elamur.ai/free/en/papers/habash-2019-automatic-gender-identification-reinflection-arabic/)
- [next → The discourse commitments of illocutionary reportatives](https://md.research.elamur.ai/free/en/papers/faller-2019-discourse-commitments-illocutionary-reportatives/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning, pages 2091–2100. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis%2C+Hao+Peng%2C+and+Charles+Sutton%2E+2016%2E+A+convolutional+attention+network+for+extreme+summarization+of+source+code%2E+In+International+Conference+on+Machine+Learning%2C+pages+2091%E2%80%932100%2E)
2. Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language modeling. In Proceedings of the 10th Working Conference on Mining Software Repositories, pages 207–216. IEEE Press. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis+and+Charles+Sutton%2E+2013%2E+Mining+source+code+repositories+at+massive+scale+using+language+modeling%2E+In+Proceedings+of+the+10th+Working+Conference+on+Mining+Software+Repositories%2C+p)
3. Miltiadis Allamanis and Charles Sutton. 2014. Mining idioms from source code. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 472– 483. ACM. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis+and+Charles+Sutton%2E+2014%2E+Mining+idioms+from+source+code%2E+In+Proceedings+of+the+22nd+ACM+SIGSOFT+International+Symposium+on+Foundations+of+Software+Engineering%2C+pages+472%E2%80%93+483%2E+ACM)
4. Miltiadis Allamanis, Daniel Tarlow, Andrew Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In Proceedings of The 32nd International Conference on Machine Learning, pages 2123–2132. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis%2C+Daniel+Tarlow%2C+Andrew+Gordon%2C+and+Yi+Wei%2E+2015%2E+Bimodal+modelling+of+source+code+and+natural+language%2E+In+Proceedings+of+The+32nd+International+Conference+on+Machine+Learning%2C+pag)
5. Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989. [find](https://scholar.google.com/scholar?q=Matej+Balog%2C+Alexander+L+Gaunt%2C+Marc+Brockschmidt%2C+Sebastian+Nowozin%2C+and+Daniel+Tarlow%2E+2016%2E+Deepcoder%3A+Learning+to+write+programs%2E+arXiv+preprint+arXiv%3A1611%2E01989%2E)
6. Li Dong and Mirella Lapata. 2016. [Language to logical form with neural attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33–43. Association for Computational Linguistics.
7. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016a. [Incorporating copying mechanism in sequence-to-sequence learning](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640, Berlin, Germany. Association for Computa…
8. Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016b. Deep api learning. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 631–642. ACM. [find](https://scholar.google.com/scholar?q=Xiaodong+Gu%2C+Hongyu+Zhang%2C+Dongmei+Zhang%2C+and+Sunghun+Kim%2E+2016b%2E+Deep+api+learning%2E+In+Proceedings+of+the+2016+24th+ACM+SIGSOFT+International+Symposium+on+Foundations+of+Software+Engineering%2C+pages+6)
9. Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016c. Deep api learning. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pages 631–642, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Xiaodong+Gu%2C+Hongyu+Zhang%2C+Dongmei+Zhang%2C+and+Sunghun+Kim%2E+2016c%2E+Deep+api+learning%2E+In+Proceedings+of+the+2016+24th+ACM+SIGSOFT+International+Symposium+on+Foundations+of+Software+Engineering%2C+FSE+201)
10. Jessica B Hamrick. 2016. Creating and grading ipython/jupyter notebook assignments with nbgrader. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pages 242–242. ACM. [find](https://scholar.google.com/scholar?q=Jessica+B+Hamrick%2E+2016%2E+Creating+and+grading+ipython%2Fjupyter+notebook+assignments+with+nbgrader%2E+In+Proceedings+of+the+47th+ACM+Technical+Symposium+on+Computing+Science+Education%2C+pages+242%E2%80%93242%2E+ACM%2E)
11. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+Jayant+Krishnamurthy%2C+and+Luke+Zettlemoyer%2E+2017%2E)
12. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963–973, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Learning+a+neural+semantic+parser+from+user+feedback%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+963%E2%80%93973%2C+Vancouver%2C+Cana)
13. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+and+Luke+Zettlemoyer%2E+2018%2E+Mapping+language+to+code+in+programmatic+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Lang)
14. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. [Search-based neural structured learning for sequential question answering](https://md.research.elamur.ai/free/en/papers/iyyer-2017-search-based-neural-structured-learning/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831.
15. Diederik Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR.
16. Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. 2016. Jupyter notebooks-a publishing format for reproducible computational workflows. [find](https://scholar.google.com/scholar?q=Thomas+Kluyver%2C+Benjamin+Ragan%2DKelley%2C+Fernando+Pe%CC%81rez%2C+Brian+E+Granger%2C+Matthias+Bussonnier%2C+Jonathan+Frederic%2C+Kyle+Kelley%2C+Jessica+B+Hamrick%2C+Jason+Grout%2C+Sylvain+Corlay%2C+et+al%2E+2016%2E+Jupyter+noteb)
17. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516–1526, Copenhagen, Denmark. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matt+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Na)
18. Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 826–836. Association for… [find](https://scholar.google.com/scholar?q=Nate+Kushman+and+Regina+Barzilay%2E+2013%2E+Using+semantic+unification+to+generate+regular+expressions+from+natural+language%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Ass)
19. Percy Liang, I. Michael Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2). [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+I%2E+Michael+Jordan%2C+and+Dan+Klein%2E+2013%2E+Learning+dependency%2Dbased+compositional+semantics%2E+Computational+Linguistics%2C+39%282%29%2E)
20. Wang Ling, Phil Blunsom, Edward Grefenstette, Moritz Karl Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599–609.… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Phil+Blunsom%2C+Edward+Grefenstette%2C+Moritz+Karl+Hermann%2C+Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+Fumin+Wang%2C+and+Andrew+Senior%2E+2016%2E+Latent+predictor+networks+for+code+generation%2E+In+Proceedings+of+the+54th+Ann)
21. Nicholas Locascio, Karthik Narasimhan, Eduardo De Leon, Nate Kushman, and Regina Barzilay. 2016. Neural generation of regular expressions from natural language with minimal domain knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1918–1923, A… [find](https://scholar.google.com/scholar?q=Nicholas+Locascio%2C+Karthik+Narasimhan%2C+Eduardo+De+Leon%2C+Nate+Kushman%2C+and+Regina+Barzilay%2E+2016%2E+Neural+generation+of+regular+expressions+from+natural+language+with+minimal+domain+knowledge%2E+In+Procee)
22. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. arXiv preprint arXiv:1606.05378. [find](https://scholar.google.com/scholar?q=Reginald+Long%2C+Panupong+Pasupat%2C+and+Percy+Liang%2E+2016%2E+Simpler+context%2Ddependent+logical+forms+via+model+projections%2E+arXiv+preprint+arXiv%3A1606%2E05378%2E)
23. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). arXiv preprint arXiv:1508.04025.
24. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on, pag… [find](https://scholar.google.com/scholar?q=Yusuke+Oda%2C+Hiroyuki+Fudaba%2C+Graham+Neubig%2C+Hideaki+Hata%2C+Sakriani+Sakti%2C+Tomoki+Toda%2C+and+Satoshi+Nakamura%2E+2015%2E+Learning+to+generate+pseudo%2Dcode+from+source+code+using+statistical+machine+translati)
25. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). In International Conference on Mining Software Repositories, MSR, pages 476–486. ACM.
26. Pengcheng Yin and Graham Neubig. 2017. [A syntactic neural model for general-purpose code generation](https://md.research.elamur.ai/free/en/papers/yin-2017-syntactic-neural-model-general-purpose/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440–450, Vancouver, Canada. Association for Computational Linguistics.
27. Luke S. Zettlemoyer and Michael Collins. 2005. [Learning to map sentences to logical form: structured classification with probabilistic categorial grammars](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/). In UAI ’05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence.
28. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of NAACL-HLT 2019: Demonstrations.
29. Luke S Zettlemoyer and Michael Collins. 2009. [Learning context-dependent mappings from sentences to logical form](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2009-learning-context-dependent-mappings-sentences/). In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pa…
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
31. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. [Seq2sql: Generating structured queries from natural language using reinforcement learning](https://md.research.elamur.ai/free/en/papers/zhong-2017-seq2sql-generating-structured-queries-natural/). arXiv preprint arXiv:1709.00103.
32. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139–1149, Vancouver, Canada. Association for Computatio… [find](https://scholar.google.com/scholar?q=Maxim+Rabinovich%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2017%2E+Abstract+syntax+networks+for+code+generation+and+semantic+parsing%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
34. Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volu… [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Srinivasan+Iyer%2C+and+Yoav+Artzi%2E+2018%2E+Learning+to+map+context%2Ddependent+sentences+to+executable+formal+queries%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.

*Updated 31 October 2026*