---
title: "Position Encoding with Random Float Sampling Enhances"
description: "The authors propose Random Float Sampling (RFS), a position indexing scheme that needs no fine-tuning. 80% accuracy. 47 references."
authors: ["Atsushi Shimizu", "Shohei Taniguchi", "Yutaka Matsuo"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.261/"
doi: "10.18653/v1/2026.findings-eacl.261"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shimizu-2026-position-encoding-random-float-sampling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shimizu-2026-position-encoding-random-float-sampling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers

# Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers

**Authors:** Atsushi Shimizu, [Shohei Taniguchi](https://md.research.elamur.ai/free/en/authors/shohei-taniguchi/) and [Yutaka Matsuo](https://md.research.elamur.ai/free/en/authors/yutaka-matsuo/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.261/) · [PDF](https://aclanthology.org/2026.findings-eacl.261.pdf) · [DOI 10.18653/v1/2026.findings-eacl.261](https://doi.org/10.18653/v1/2026.findings-eacl.261) · [OpenAlex](https://openalex.org/W7140081622)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Transformers often fail on inputs longer than their training context. Standard integer position indices beyond the trained range are out-of-distribution (OOD), and training on longer sequences is expensive. The authors propose Random Float Sampling (RFS), a position indexing scheme that needs no fine-tuning. During training, each sequence gets sorted random floats drawn uniformly from [0, 1) as position indices. At inference, indices are spaced evenly across the same interval, with the denominator using max(n_tr, n_in). All indices are then multiplied by a scale L, set to 1,000.

Because inference indices always stay inside the range seen in training, and random training exposes the model to many index spacings, the model learns to rely on relative distances. RFS plugs into absolute sinusoidal encoding, RoPE and ALiBi.

On synthetic length-generalization tasks, RFS clearly outperforms NoPE, simple extension, randomly sampled integer positions, and position interpolation applied with no fine-tuning. In GPT-2-style language modeling, it nearly closes the gap between in-distribution and longer test inputs. This is useful for small models with short context windows.

## Key points
- **Synthetic benchmark:** six exact-match tasks following Kazemnejad et al. (2023): copy, reverse, addition, sort, summation and SCAN. The model is a decoder-only "base" Transformer with the T5-Base tokenizer, trained from scratch.
- **Copy task, sinusoidal PE:**
  - RFS keeps about 80% accuracy at twice the training length.
  - NoPE falls to about 20% already at 1.5× length.
- **Ablation (copy task, RoPE, average OOD accuracy):**
  - Uniform sampling gives 83.7%.
  - Normal sampling gives 0.0%, and Beta(0.5, 0.5) gives 15.5%.
  - CDF-matched variants of these reach 81.9% and 81.2%.
  - L = 10 and L = 100,000 barely change results (83.4% and 83.6%).
- **Language modeling setup:** a GPT-2 model (12 layers, 12 heads, 768-dim) trained on OpenWebText. It is evaluated zero-shot on HellaSwag, RACE, ARC-Easy, ARC-Challenge, OpenBookQA, WinoGrande and BoolQ. The context window is set to each task's median token count, which splits test sets into ID and OOD halves.
- **Language modeling results:**
  - Average OOD accuracy rises from 34.47% to 35.78%, nearly matching the baseline's ID score of 35.84%.
  - The largest gains are on WinoGrande (46.20 → 50.17) and BoolQ (56.25 → 60.42).
  - There is no OOD gain on HellaSwag, RACE or ARC-Easy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Atsushi Shimizu, Shohei Taniguchi and Yutaka Matsuo. 2026. Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shimizu2026position,
  title = {{Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers}},
  author = {Atsushi Shimizu, Shohei Taniguchi and Yutaka Matsuo},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.261},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.261}
}
```

- [← previous POS Tagging with Generative LLMs for Historical Germanic Low-Resource Languages: An Evaluation Against Fine-Tuned BERT](https://md.research.elamur.ai/free/en/papers/miani-2026-pos-tagging-generative-llms-historical/)
- [next → Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric](https://md.research.elamur.ai/free/en/papers/perez-2026-position-semiotic-hermeneutic-approach-evaluating/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, C… — [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://md.research.elamur.ai/free/en/papers/allal-2025-smollm2-smol-goes-big-data/)
2. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai. Preprint, arXiv:2506.02153. [find](https://scholar.google.com/scholar?q=Peter+Belcak%2C+Greg+Heinrich%2C+Shizhe+Diao%2C+Yonggan+Fu%2C+Xin+Dong%2C+Saurav+Muralidharan%2C+Yingyan+Celine+Lin%2C+and+Pavlo+Molchanov%2E+2025%2E+Small+language+models+are+the+future+of+agentic+ai%2E+Preprint%2C+arXiv%3A)
3. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. Preprint, arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+Preprint%2C+arXiv%3A2306%2E15595%2E)
4. Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022. Kerple: Kernelized relative positional embedding for length extrapolation. In Advances in Neural Information Processing Systems, volume 35, pages 8386–8399. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Peter+J+Ramadge%2C+and+Alexander+Rudnicky%2E+2022%2E+Kerple%3A+Kernelized+relative+positional+embedding+for+length+extrapolation%2E+In+Advances+in+Neural+Information+Processing+Syste)
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457.
6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
7. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [LongRoPE: Extending LLM context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Res…
8. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
9. Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019. Openwebtext corpus. [http://Skylion007.github.io/OpenWebTextCorpus](http://Skylion007.github.io/OpenWebTextCorpus). [↗](http://Skylion007.github.io/OpenWebTextCorpus)
10. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382–1390, Abu Dhabi, United Arab Emirates. Association for C… [find](https://scholar.google.com/scholar?q=Adi+Haviv%2C+Ori+Ram%2C+Ofir+Press%2C+Peter+Izsak%2C+and+Omer+Levy%2E+2022%2E+Transformer+language+models+without+positional+encodings+still+learn+positional+information%2E+In+Findings+of+the+Association+for+Comput)
11. Kazuki Irie. 2024. Why are positional encodings nonessential for deep autoregressive transformers? revisiting a petroglyph. Preprint, arXiv:2501.00659. — [arXiv:2501.00659](https://arxiv.org/abs/2501.00659) [find](https://scholar.google.com/scholar?q=Kazuki+Irie%2E+2024%2E+Why+are+positional+encodings+nonessential+for+deep+autoregressive+transformers%3F+revisiting+a+petroglyph%2E+Preprint%2C+arXiv%3A2501%2E00659%2E+%E2%80%94+%5BarXiv%3A2501%2E00659%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E)
12. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems, volume 36, pages 24892–24928. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Amirhossein+Kazemnejad%2C+Inkit+Padhi%2C+Karthikeyan+Natesan+Ramamurthy%2C+Payel+Das%2C+and+Siva+Reddy%2E+2023%2E+The+impact+of+positional+encoding+on+length+generalization+in+transformers%2E+In+Advances+in+Neural+)
13. Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. 2021. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3309–3321, Online and Punta Cana, Dominican Republic. Association for Compu… [find](https://scholar.google.com/scholar?q=Shun+Kiyono%2C+Sosuke+Kobayashi%2C+Jun+Suzuki%2C+and+Kentaro+Inui%2E+2021%2E+SHAPE%3A+Shifted+absolute+position+embedding+for+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+La)
14. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [RACE: Large-scale ReAding comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
16. Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2873–2882. PMLR. [find](https://scholar.google.com/scholar?q=Brenden+Lake+and+Marco+Baroni%2E+2018%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Proceedings+of+the+35th+International+Conference+o)
17. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). Preprint, arXiv:2309.05463.
18. Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021. Cape: Encoding relative positions with continuous augmented positional embeddings. In Advances in Neural Information Processing Systems, volume 34, pages 16079–16092. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Tatiana+Likhomanenko%2C+Qiantong+Xu%2C+Gabriel+Synnaeve%2C+Ronan+Collobert%2C+and+Alex+Rogozhnikov%2E+2021%2E+Cape%3A+Encoding+relative+positions+with+continuous+augmented+positional+embeddings%2E+In+Advances+in+Neur)
19. LiquidAI. 2025. Introducing liquid nanos — frontier-grade performance on everyday devices. [find](https://scholar.google.com/scholar?q=LiquidAI%2E+2025%2E+Introducing+liquid+nanos+%E2%80%94+frontier%2Dgrade+performance+on+everyday+devices%2E)
20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+EMNLP%2E)
21. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). Preprint, arXiv:2501.19393.
22. Ken M. Nakanishi. 2025. Scalable-softmax is superior for attention. Preprint, arXiv:2501.19399. — [arXiv:2501.19399](https://arxiv.org/abs/2501.19399) [find](https://scholar.google.com/scholar?q=Ken+M%2E+Nakanishi%2E+2025%2E+Scalable%2Dsoftmax+is+superior+for+attention%2E+Preprint%2C+arXiv%3A2501%2E19399%2E+%E2%80%94+%5BarXiv%3A2501%2E19399%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E19399%29)
23. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). Preprint, arXiv:2112.00…
24. Yui Oka, Taku Hasegawa, Kyosuke Nishida, and Kuniko Saito. 2025. Wavelet-based positional representation for long context. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yui+Oka%2C+Taku+Hasegawa%2C+Kyosuke+Nishida%2C+and+Kuniko+Saito%2E+2025%2E+Wavelet%2Dbased+positional+representation+for+long+context%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
25. Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. 2022. Making transformers solve compositional tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3591– 3607, Dublin, Ireland. Association for Computation… [find](https://scholar.google.com/scholar?q=Santiago+Ontanon%2C+Joshua+Ainslie%2C+Zachary+Fisher%2C+and+Vaclav+Cvicek%2E+2022%2E+Making+transformers+solve+compositional+tasks%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational)
26. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2024%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat)
27. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
28. Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2021%2E+Shortformer%3A+Better+language+modeling+using+shorter+inputs%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+)
29. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–2…
30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(1).
32. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo… [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+2023%2E+Randomized+positional+encodings+boost+length+generalization+of+transfor)
33. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2019%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+arXiv+preprint+arXiv%3A1907%2E10641%2E)
34. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). In International Conference on Learning Representations.
35. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc.
36. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
38. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
41. Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. 2025. Rope to nope and back again: A new hybrid attention strategy. Preprint, arXiv:2501.18795. — [arXiv:2501.18795](https://arxiv.org/abs/2501.18795) [find](https://scholar.google.com/scholar?q=Bowen+Yang%2C+Bharat+Venkitesh%2C+Dwarak+Talupuru%2C+Hangyu+Lin%2C+David+Cairuz%2C+Phil+Blunsom%2C+and+Acyr+Locatelli%2E+2025%2E+Rope+to+nope+and+back+again%3A+A+new+hybrid+attention+strategy%2E+Preprint%2C+arXiv%3A2501%2E1879)
42. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The Eleventh International Conference on Learning Representations.
43. Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. In Advances in Neural Information Processing Systems, volume 36, pages 78808– 78823. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+Daniel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+Megabyte%3A+Predicting+million%2Dbyte+sequences+with+multiscale+transformers%2E+In+Advances+in+Neural+Informa)
44. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
45. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). Preprint, arXiv:2401.02385.
46. Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, and Yu Li. 2024. DAPE: Data-adaptive positional encoding for length extrapolation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Chuanyang+Zheng%2C+Yihang+Gao%2C+Han+Shi%2C+Minbin+Huang%2C+Jingyao+Li%2C+Jing+Xiong%2C+Xiaozhe+Ren%2C+Michael+Ng%2C+Xin+Jiang%2C+Zhenguo+Li%2C+and+Yu+Li%2E+2024%2E+DAPE%3A+Data%2Dadaptive+positional+encoding+for+length+extrapol)
47. Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. 2024. Transformers can achieve length generalization but not robustly. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Uri+Alon%2C+Xinyun+Chen%2C+Xuezhi+Wang%2C+Rishabh+Agarwal%2C+and+Denny+Zhou%2E+2024%2E+Transformers+can+achieve+length+generalization+but+not+robustly%2E+In+ICLR+2024+Workshop+on+Mathematical+and+Emp)

*Updated 1 November 2026*