---
title: "Reka Core, Flash, and Edge — paper by Ormazabal et al. (2024)"
description: "The Reka Team presents three multimodal language models trained from scratch: Edge (7B, dense)… 41 references, cited by 39 papers here."
authors: ["Reka Team: Aitor Ormazabal", "Che Zheng", "Cyprien de Masson d'Autume", "Dani Yogatama", "Deyu Fu", "Donovan Ong", "Eric Chen", "Eugenie Lamprecht", "Hai Pham", "Isaac Ong", "Kaloyan Aleksiev", "Lei Li", "Matthew Henderson", "Max Bain", "Mikel Artetxe", "Nishant Relan", "Piotr Padlewski", "Qi Liu", "Ren Chen", "Samuel Phua", "Yazheng Yang", "Yi Tay", "Yuqi Wang", "Zhongkai Zhu", "Zhihui Xie"]
year: 2024
venue: "arXiv:2404.12387"
source: "https://arxiv.org/abs/2404.12387"
doi: "10.48550/arxiv.2404.12387"
arxiv: "2404.12387"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models

# Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models

**Authors:** Reka Team: Aitor Ormazabal, Che Zheng, Cyprien de Masson d'Autume, [Dani Yogatama](https://md.research.elamur.ai/free/en/authors/dani-yogatama/), Deyu Fu, [Donovan Ong](https://md.research.elamur.ai/free/en/authors/donovan-ong/), Eric Chen, [Eugenie Lamprecht](https://md.research.elamur.ai/free/en/authors/eugenie-lamprecht/), [Hai Pham](https://md.research.elamur.ai/free/en/authors/hai-pham/), Isaac Ong, [Kaloyan Aleksiev](https://md.research.elamur.ai/free/en/authors/kaloyan-aleksiev/), [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/), [Matthew Henderson](https://md.research.elamur.ai/free/en/authors/matthew-henderson/), Max Bain, [Mikel Artetxe](https://md.research.elamur.ai/free/en/authors/mikel-artetxe/), Nishant Relan, [Piotr Padlewski](https://md.research.elamur.ai/free/en/authors/piotr-padlewski/), [Qi Liu](https://md.research.elamur.ai/free/en/authors/qi-liu/), Ren Chen, Samuel Phua, Yazheng Yang, [Yi Tay](https://md.research.elamur.ai/free/en/authors/yi-tay/), [Yuqi Wang](https://md.research.elamur.ai/free/en/authors/yuqi-wang/), Zhongkai Zhu and Zhihui Xie · **Year:** 2024 · **Venue:** arXiv:2404.12387 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.12387) · [PDF](https://arxiv.org/pdf/2404.12387) · [DOI 10.48550/arxiv.2404.12387](https://doi.org/10.48550/arxiv.2404.12387) · [arXiv 2404.12387](https://arxiv.org/abs/2404.12387) · [OpenAlex](https://openalex.org/W4394972274)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The Reka Team presents three multimodal language models trained from scratch: Edge (7B, dense), Flash (21B, dense) and Core, the largest, whose training was still ongoing when the report was written. The work aims to deliver frontier-level quality across modalities with a startup's limited compute. The models take text, image, video and audio as input and produce only text. They use a modular encoder-decoder design built on a "Noam"-style Transformer backbone with SwiGLU, Grouped Query Attention, RoPE and RMSNorm. Pretraining used trillions of filtered tokens, with code, STEM and multilingual data deliberately up-weighted, followed by SFT and PPO-based RLHF. Edge beats Gemma 7B and Mistral 7B on most benchmarks. Flash often matches or surpasses models of roughly 70B parameters and Grok-1. Core reaches 83.2 on MMLU, is competitive with GPT-4V on MMMU and VQAv2, and ranks second in blind multimodal-chat human evaluation, ahead of Claude 3 Opus. The report also documents practical problems with GPU cluster reliability, which matters for small teams building large models.

## Key points
- **Training data:** Flash was trained on about 5T text tokens and Edge on about 4.5T, with a knowledge cutoff of November 2023. The mix is roughly 25% code, 30% STEM, 25% web crawl and 10% math-related data.
- **Multilingual coverage:** About 15% of the data is deliberately multilingual, spanning 32 tier-weighted languages, and the full multilingual Wikipedia (110 languages) is also included. Ukrainian does not appear in the extracted P1/P2 tier list, although Russian, Czech and Bulgarian do, so Ukrainian likely receives only baseline-level coverage.
- **Context and tokenizer:** The standard context length is 8K, with 128K long-context variants of Flash and Core that appear to extrapolate to 256K on passkey retrieval. Long-context SFT data comes from "reverse instruction tuning". Flash and Edge use a 100K sentencepiece vocabulary based on tiktoken.
- **Compute and reliability:** Peak compute was about 2.5K H100s plus 2.5K A100s. Weekly node failures varied widely, from 0.2 at one provider to more than 30 during another provider's early phase.
- **Domain results:** On medical benchmarks, Core averages 81.3 across MedMCQA, PubMedQA and MMLU (Medical), above Med-PaLM-2 and close to GPT-4. Edge scores 66.2 on GSM8K and 54.3 on HumanEval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Reka Team: Aitor Ormazabal, Che Zheng, Cyprien de Masson d'Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu and Zhihui Xie. 2024. Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models. arXiv:2404.12387.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ormazabal2024reka,
  title = {{Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models}},
  author = {Reka Team: Aitor Ormazabal, Che Zheng, Cyprien {de Masson d'Autume}, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu and Zhihui Xie},
  year = {2024},
  doi = {10.48550/arxiv.2404.12387},
  eprint = {2404.12387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.12387}
}
```

- [← previous LUQ: Long-text Uncertainty Quantification for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/)
- [next → Auxiliary task demands mask the capabilities of smaller language models](https://md.research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

41 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/), 2023.
2. Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+2024%2E)
3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). CoRR, abs/1910.11856, 2019.
4. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+Nelson+Elhage%2C+Zac+Hatfield%2DDodds%2C+Danny+Hernandez%2C+Jackson+Kernion%2C)
5. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. [The belebele benchmark: a parallel reading comprehension dataset in 122 language variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/), 2023.
6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne H… — [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+Journal)
11. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. [Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454–470, 2020.
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
14. Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone,… [find](https://scholar.google.com/scholar?q=Gemma%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+Pouya+Tafti%2C+L%C3%A9onard+Hussenot%2C+Pier+Giuseppe+)
15. Google, Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robi… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
16. Gemini Team Google. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
17. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2021.
19. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
21. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… [find](https://scholar.google.com/scholar?q=Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Mikel+Artetxe%2C+Tianlu+Wang%2C+Shuohui+Chen%2C+Daniel+Simig%2C+Myle+Ott%2C+Naman+Goyal%2C+Shruti+Bhosale%2C+Jingfei+Du%2C+Ramakanth+Pasunuru%2C+Sam+Shleifer%2C+Punit+Singh+Koura%2C+Vishrav)
22. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
23. OpenAI. Gpt-4v(ision) system card. 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4v%28ision%29+system+card%2E+2024%2E)
24. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
25. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach De-Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
26. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr…
27. Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stepha… [find](https://scholar.google.com/scholar?q=Viorica+P%C4%83tr%C4%83ucean%2C+Lucas+Smaira%2C+Ankush+Gupta%2C+Adri%C3%A0+Recasens+Continente%2C+Larisa+Markeeva%2C+Dylan+Banarse%2C+Skanda+Koppula%2C+Joseph+Heyward%2C+Mateusz+Malinowski%2C+Yi+Yang%2C+Carl+Doersch%2C+Tatiana+Matejovico)
28. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
30. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), 2023.
31. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. [↗](http://arxiv.org/abs/1707.06347)
32. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E)
33. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
34. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Y… — [Towards Expert-Level Medical Question Answering with Large Language Models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/)
35. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864, 2021.
36. Yi Tay. Training great llms entirely from ground up in the wilderness as a startup. 2024. [find](https://scholar.google.com/scholar?q=Yi+Tay%2E+Training+great+llms+entirely+from+ground+up+in+the+wilderness+as+a+startup%2E+2024%2E)
37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
38. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021.
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Conference on Neural Information Processing Systems (NeurIPS), 2022. xAI. Announcing grok. 2023. — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
40. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
41. Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+Root+mean+square+layer+normalization%2C+2019%2E)

*Updated 30 October 2026*