---
title: "Repeat After Me — paper by Jelassi et al. (2024)"
description: "On the theory side, they build a two-layer transformer that copies by matching n-grams. 42 references, cited by 54 papers here."
authors: ["Samy Jelassi", "David Brandfonbrener", "Sham M. Kakade", "Eran Malach"]
year: 2024
venue: "arXiv:2402.01032"
source: "https://arxiv.org/abs/2402.01032"
doi: "10.48550/arxiv.2402.01032"
arxiv: "2402.01032"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jelassi-2024-repeat-after-me-transformers-better/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jelassi-2024-repeat-after-me-transformers-better/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Repeat After Me: Transformers are Better than State Space Models at Copying

# Repeat After Me: Transformers are Better than State Space Models at Copying

**Authors:** [Samy Jelassi](https://md.research.elamur.ai/free/en/authors/samy-jelassi/), [David Brandfonbrener](https://md.research.elamur.ai/free/en/authors/david-brandfonbrener/), [Sham M. Kakade](https://md.research.elamur.ai/free/en/authors/sham-m-kakade/) and [Eran Malach](https://md.research.elamur.ai/free/en/authors/eran-malach/) · **Year:** 2024 · **Venue:** arXiv:2402.01032 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01032) · [PDF](https://arxiv.org/pdf/2402.01032) · [DOI 10.48550/arxiv.2402.01032](https://doi.org/10.48550/arxiv.2402.01032) · [arXiv 2402.01032](https://arxiv.org/abs/2402.01032) · [OpenAlex](https://openalex.org/W4391555700)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/jelassi-2024-repeat-after-me-transformers-better/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks what models with a constant-size memory give up in exchange for cheap inference. The authors call this family "generalized state space models" (GSSMs). It includes S4, Mamba, LSTMs and linear-attention variants. The authors find that the main loss is the ability to reproduce or retrieve arbitrary parts of the input.

On the theory side, they build a two-layer transformer that copies by matching n-grams. It uses a positional scheme they introduce, Hard-ALiBi, and handles strings whose length grows exponentially with the number of heads. They also show that no GSSM can reliably copy inputs containing more bits than its latent state holds.

Experiments test whether these limits matter in practice:
- **Synthetic copying:** with roughly 160M-parameter models, transformers learn copying with far fewer examples and handle longer strings much better.
- **Pretrained models:** Pythia beats Mamba of similar size on copying and in-context lookup, even though Mamba achieves lower perplexity on the Pile.

The results suggest that architecture affects downstream ability beyond perplexity. The authors argue for hybrid designs that give state space models attention-like retrieval.

## Key points
- Theory: a two-block transformer copies with small error, failing only when the input has repeated n-grams. A GSSM's copyable length is capped by mem(S), the number of bits in its state.
- Hard-ALiBi masks attention outside a window of size m, with a different window per head. It helps empirically and lets transformers generalize past the training context length.
- In synthetic training on strings of length ≤ 300, transformers reach high accuracy far sooner than Mamba. An LSTM fails to learn the task within the same sample budget.
- In 1-shot phone-book lookup without finetuning, Pythia (410M/1.4B/2.8B) clearly outperforms Mamba (360M/1.4B/2.8B) at every size.
- The paper was published at ICML 2024. The authors acknowledge GSSM strengths, such as tracking state over long sequences (flip-flop tasks) and processing very long inputs like DNA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Samy Jelassi, David Brandfonbrener, Sham M. Kakade and Eran Malach. 2024. Repeat After Me: Transformers are Better than State Space Models at Copying. arXiv:2402.01032.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jelassi2024repeat,
  title = {{Repeat After Me: Transformers are Better than State Space Models at Copying}},
  author = {Samy Jelassi, David Brandfonbrener, Sham M. Kakade and Eran Malach},
  year = {2024},
  doi = {10.48550/arxiv.2402.01032},
  eprint = {2402.01032},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01032}
}
```

- [← previous RegMix: Data Mixture as Regression for Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/)
- [next → TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space](https://md.research.elamur.ai/free/en/papers/zhang-2024-truthx-alleviating-hallucinations-editing-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12973, 2024. [find](https://scholar.google.com/scholar?q=Akyu%CC%88rek%2C+E%2E%2C+Wang%2C+B%2E%2C+Kim%2C+Y%2E%2C+and+Andreas%2C+J%2E+In%2Dcontext+language+learning%3A+Arhitectures+and+algorithms%2E+arXiv+preprint+arXiv%3A2401%2E12973%2C+2024%2E)
2. Edelman, B. L., Goel, S., Kakade, S., and Zhang, C. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, pp. 5793–5831. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Edelman%2C+B%2E+L%2E%2C+Goel%2C+S%2E%2C+Kakade%2C+S%2E%2C+and+Zhang%2C+C%2E+Inductive+biases+and+variable+creation+in+self%2Dattention+mechanisms%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+5793%E2%80%935831%2E+PMLR%2C+2022%2E)
3. Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Anil%2C+C%2E%2C+Wu%2C+Y%2E%2C+Andreassen%2C+A%2E%2C+Lewkowycz%2C+A%2E%2C+Misra%2C+Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+et+al%2E+The+pile%3A+An+800gb+dat)
4. Merrill, W., Weiss, G., Goldberg, Y., Schwartz, R., Smith, N. A., and Yahav, E. A formal hierarchy of rnn architectures. arXiv preprint arXiv:2004.08500, 2020. [find](https://scholar.google.com/scholar?q=Merrill%2C+W%2E%2C+Weiss%2C+G%2E%2C+Goldberg%2C+Y%2E%2C+Schwartz%2C+R%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Yahav%2C+E%2E+A+formal+hierarchy+of+rnn+architectures%2E+arXiv+preprint+arXiv%3A2004%2E08500%2C+2020%2E)
5. Grazzi, R., Siems, J., Schrodi, S., Brox, T., and Hutter, F. Is mamba capable of in-context learning? arXiv preprint arXiv:2402.03170, 2024. — [arXiv:2402.03170](https://arxiv.org/abs/2402.03170) [find](https://scholar.google.com/scholar?q=Grazzi%2C+R%2E%2C+Siems%2C+J%2E%2C+Schrodi%2C+S%2E%2C+Brox%2C+T%2E%2C+and+Hutter%2C+F%2E+Is+mamba+capable+of+in%2Dcontext+learning%3F+arXiv+preprint+arXiv%3A2402%2E03170%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E03170%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E03170%29)
6. Merrill, W., Sabharwal, A., and Smith, N. A. Saturated Transformers are Constant-Depth Threshold Circuits. Transactions of the Association for Computational Linguistics, 10:843–856, 08 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00493. URL https://doi.org/10. 1162/tacl_a_00493. [find](https://scholar.google.com/scholar?q=Merrill%2C+W%2E%2C+Sabharwal%2C+A%2E%2C+and+Smith%2C+N%2E+A%2E+Saturated+Transformers+are+Constant%2DDepth+Threshold+Circuits%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+10%3A843%E2%80%93856%2C+08+2022%2E+ISSN+2307)
7. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E+and+Dao%2C+T%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
8. Miller, G. A. The magic number seven plus or minus two: Some limits on our capacity for processing information. Psychological review, 63:91–97, 1956. [find](https://scholar.google.com/scholar?q=Miller%2C+G%2E+A%2E+The+magic+number+seven+plus+or+minus+two%3A+Some+limits+on+our+capacity+for+processing+information%2E+Psychological+review%2C+63%3A91%E2%80%9397%2C+1956%2E)
9. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Goel%2C+K%2E%2C+and+Re%CC%81%2C+C%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+arXiv+preprint+arXiv%3A2111%2E00396%2C+2021%2E)
10. Nguyen, E., Poli, M., Faizi, M., Thomas, A., Birch-Sykes, C., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. arXiv preprint arXiv:2306.15794, 2023. [find](https://scholar.google.com/scholar?q=Nguyen%2C+E%2E%2C+Poli%2C+M%2E%2C+Faizi%2C+M%2E%2C+Thomas%2C+A%2E%2C+Birch%2DSykes%2C+C%2E%2C+Wornow%2C+M%2E%2C+Patel%2C+A%2E%2C+Rabideau%2C+C%2E%2C+Massaroli%2C+S%2E%2C+Bengio%2C+Y%2E%2C+et+al%2E+Hyenadna%3A+Long%2Drange+genomic+sequence+modeling+at+single+nucleotide)
11. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
12. Jelassi, S., d’Ascoli, S., Domingo-Enrich, C., Wu, Y., Li, Y., and Charton, F. [Length generalization in arithmetic transformers](https://md.research.elamur.ai/free/en/papers/jelassi-2023-length-generalization-arithmetic-transformers/). arXiv preprint arXiv:2306.15400, 2023.
13. Kamradt, G. Llmtest needleinahaystack. //github.com/gkamradt/LLMTest_ NeedleInAHaystack, 2023. [find](https://scholar.google.com/scholar?q=Kamradt%2C+G%2E+Llmtest+needleinahaystack%2E+%2F%2Fgithub%2Ecom%2Fgkamradt%2FLLMTest%5F+NeedleInAHaystack%2C+2023%2E)
14. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895, 2022. https:
15. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Katharopoulos%2C+A%2E%2C+Vyas%2C+A%2E%2C+Pappas%2C+N%2E%2C+and+Fleuret%2C+F%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+learning%2C+pp%2E+5156%E2%80%935165%2E+)
16. Park, J., Park, J., Xiong, Z., Lee, N., Cho, J., Oymak, S., Lee, K., and Papailiopoulos, D. [Can mamba learn how to learn? a comparative study on in-context learning tasks](https://md.research.elamur.ai/free/en/papers/park-2024-mamba-learn-learn-comparative-study/). arXiv preprint arXiv:2402.04248, 2024.
17. Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., and Reddy, S. [The impact of positional encoding on length generalization in transformers](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/). arXiv preprint arXiv:2305.19466, 2023.
18. Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310–1318. Pmlr, 2013. [find](https://scholar.google.com/scholar?q=Pascanu%2C+R%2E%2C+Mikolov%2C+T%2E%2C+and+Bengio%2C+Y%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+1310%E2%80%931318%2E+Pmlr%2C+2013%2E)
19. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Exposing attention glitches with flip-flop language modeling. arXiv preprint arXiv:2306.00946, 2023a. [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Ash%2C+J%2E+T%2E%2C+Goel%2C+S%2E%2C+Krishnamurthy%2C+A%2E%2C+and+Zhang%2C+C%2E+Exposing+attention+glitches+with+flip%2Dflop+language+modeling%2E+arXiv+preprint+arXiv%3A2306%2E00946%2C+2023a%2E)
20. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32, 2019.
21. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172, 2023b.
22. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Alcaide%2C+E%2E%2C+Anthony%2C+Q%2E%2C+Albalak%2C+A%2E%2C+Arcadinho%2C+S%2E%2C+Cao%2C+H%2E%2C+Cheng%2C+X%2E%2C+Chung%2C+M%2E%2C+Grella%2C+M%2E%2C+GV%2C+K%2E+K%2E%2C+et+al%2E+Rwkv%3A+Reinventing+rnns+for+the+transformer+era%2E+arXiv+preprint+arXiv%3A2305%2E1)
23. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
24. Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, S. How context affects language models’ factual predictions. arXiv preprint arXiv:2005.[04611, 2020](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/).
25. McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., and Celikyilmaz, A. [How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/). Transactions of the Association for Computational Linguistics, 11:652–670, 2023.
26. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
27. Merrill, W. Sequential neural networks as automata. arXiv preprint arXiv:1906.01615, 2019. [find](https://scholar.google.com/scholar?q=Merrill%2C+W%2E+Sequential+neural+networks+as+automata%2E+arXiv+preprint+arXiv%3A1906%2E01615%2C+2019%2E)
28. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners, 2019. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 160025533. machines with transformers. Advances in Neural Information Processing Systems, 35:12071–12083, 2022. [↗](https://api.semanticscholar.org/CorpusID:)
29. Weiss, G., Goldberg, Y., and Yahav, E. Thinking like transformers. In International Conference on Machine Learning, pp. 11080–11090. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Weiss%2C+G%2E%2C+Goldberg%2C+Y%2E%2C+and+Yahav%2C+E%2E+Thinking+like+transformers%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+11080%E2%80%9311090%2E+PMLR%2C+2021%2E)
30. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
31. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771, 2019.
32. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.[03822, 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/).
33. Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. [What algorithms can transformers learn? a study in length generalization](https://md.research.elamur.ai/free/en/papers/zhou-2023-algorithms-transformers-learn-study-length/). arXiv preprint arXiv:2310.16028, 2023.
34. Ruoss, A., Delétang, G., Genewein, T., Grau-Moya, J., Csordás, R., Bennani, M., Legg, S., and Veness, J. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023. [find](https://scholar.google.com/scholar?q=Ruoss%2C+A%2E%2C+Dele%CC%81tang%2C+G%2E%2C+Genewein%2C+T%2E%2C+Grau%2DMoya%2C+J%2E%2C+Csorda%CC%81s%2C+R%2E%2C+Bennani%2C+M%2E%2C+Legg%2C+S%2E%2C+and+Veness%2C+J%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+arXiv+preprint+a)
35. Sanford, C., Hsu, D., and Telgarsky, M. Representational strengths and limitations of transformers. arXiv preprint arXiv:2306.02896, 2023. — [arXiv:2306.02896](https://arxiv.org/abs/2306.02896) [find](https://scholar.google.com/scholar?q=Sanford%2C+C%2E%2C+Hsu%2C+D%2E%2C+and+Telgarsky%2C+M%2E+Representational+strengths+and+limitations+of+transformers%2E+arXiv+preprint+arXiv%3A2306%2E02896%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E02896%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E02896%29)
36. Shelton, T. The Ingenious Gentleman Don Quixote of La Mancha. 1612. Written by Miguel de Cervantes, translated by Thomas Shelton. Shen, R., Bubeck, S., Eldan, R., Lee, Y. T., Li, Y., and Zhang, Y. [Positional description matters for transformers arithmetic](https://md.research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/). arXiv preprint arXiv:2311.14737, 2023.
37. Strobl, L., Merrill, W., Weiss, G., Chiang, D., and Angluin, D. Transformers as recognizers of formal languages: A survey on expressivity. arXiv preprint arXiv:2311.00208, 2023. [find](https://scholar.google.com/scholar?q=Strobl%2C+L%2E%2C+Merrill%2C+W%2E%2C+Weiss%2C+G%2E%2C+Chiang%2C+D%2E%2C+and+Angluin%2C+D%2E+Transformers+as+recognizers+of+formal+languages%3A+A+survey+on+expressivity%2E+arXiv+preprint+arXiv%3A2311%2E00208%2C+2023%2E)
38. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, pp. 127063, 2023.
39. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621, 2023.
40. Tikochinski, R., Goldstein, A., Meiri, Y., Hasson, U., and Reichart, R. An incremental large language model for long text processing in the brain. 2024. [find](https://scholar.google.com/scholar?q=Tikochinski%2C+R%2E%2C+Goldstein%2C+A%2E%2C+Meiri%2C+Y%2E%2C+Hasson%2C+U%2E%2C+and+Reichart%2C+R%2E+An+incremental+large+language+model+for+long+text+processing+in+the+brain%2E+2024%2E)
41. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
42. Wei, C., Chen, Y., and Ma, T. Statistically meaningful approximation: a case study on approximating turing A. Experimental setup [find](https://scholar.google.com/scholar?q=Wei%2C+C%2E%2C+Chen%2C+Y%2E%2C+and+Ma%2C+T%2E+Statistically+meaningful+approximation%3A+a+case+study+on+approximating+turing+A%2E+Experimental+setup)

*Updated 3 November 2026*