---
title: "Nemotron-4 340B Technical Report — Adler 2024"
description: "Nemotron-4 340B is NVIDIA's effort to give the community a very large, openly licensed model family that can also produce synthetic training data."
authors: ["Nvidia: Bo Adler", "Niket Agarwal", "Ashwath Aithal", "Dong H. Anh", "Pallab Bhattacharya", "Annika Brundyn", "Jared Casper", "Bryan Catanzaro", "Sharon Clay", "Jonathan Cohen", "Sirshak Das", "Ayush Dattagupta", "Olivier Delalleau", "Leon Derczynski", "Yi Dong", "Daniel Egert", "Ellie Evans", "Aleksander Ficek", "Denys Fridman", "Shaona Ghosh", "Boris Ginsburg", "Igor Gitman", "Tomasz Grzegorzek", "Robert Hero", "Jining Huang", "Vibhu Jawa", "Joseph Jennings", "Aastha Jhunjhunwala", "John Kamalu", "Sadaf Khan", "Oleksii Kuchaiev", "Patrick LeGresley", "Hui Li", "Jiwei Liu", "Zihan Liu", "Eileen Long", "Ameya Sunil Mahabaleshwarkar", "Somshubra Majumdar", "James Maki", "Miguel Martinez", "Maer Rodrigues de Melo", "Ivan Moshkov", "Deepak Narayanan", "Sean Narenthiran", "Jesus Navarro", "Phong Nguyen", "Osvald Nitski", "Vahid Noroozi", "Guruprasad Nutheti", "Christopher Parisien", "Jupinder Parmar", "Mostofa Patwary", "Krzysztof Pawelec", "Wei Ping", "Shrimai Prabhumoye", "Rajarshi Roy", "Trisha Saar", "Vasanth Rao Naik Sabavat", "Sanjeev Satheesh", "Jane Polak Scowcroft", "Jason Sewall", "Pavel Shamis", "Gerald Shen", "Mohammad Shoeybi", "Dave Sizer", "Misha Smelyanskiy", "Felipe Soares", "Makesh Narsimhan Sreedhar", "Dan Su", "Sandeep Subramanian", "Shengyang Sun", "Shubham Toshniwal", "Hao Wang", "Zhilin Wang", "Jiaxuan You", "Jiaqi Zeng", "Jimmy Zhang", "Jing Zhang", "Vivienne Zhang", "Yian Zhang", "Chen Zhu"]
year: 2024
venue: "arXiv:2406.11704"
source: "https://arxiv.org/abs/2406.11704"
doi: "10.48550/arxiv.2406.11704"
arxiv: "2406.11704"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Nemotron-4 340B Technical Report

# Nemotron-4 340B Technical Report

**Authors:** Nvidia: Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, [Jared Casper](https://md.research.elamur.ai/free/en/authors/jared-casper/), [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/), Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta et al. (81 authors) · **Year:** 2024 · **Venue:** arXiv:2406.11704 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.11704) · [PDF](https://arxiv.org/pdf/2406.11704) · [DOI 10.48550/arxiv.2406.11704](https://doi.org/10.48550/arxiv.2406.11704) · [arXiv 2406.11704](https://arxiv.org/abs/2406.11704) · [OpenAlex](https://openalex.org/W4399795412)

**Citations:** 7 citations (openalex-snapshot · 2026-10-14) · [95](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Nemotron-4 340B is NVIDIA's effort to give the community a very large, openly licensed model family that can also produce synthetic training data. They release three checkpoints under the NVIDIA Open Model License: Nemotron-4-340B-Base, Nemotron-4-340B-Instruct and Nemotron-4-340B-Reward. The base model is a decoder-only Transformer that uses GQA, RoPE and squared ReLU. It was pretrained on 9T tokens of English, multilingual and code text. In the last 1T tokens, the mix shifted toward higher-quality sources and added some question-answering data. Alignment used SFT followed by preference tuning with DPO and RLHF. A multi-attribute regression reward model, trained on the human-labelled HelpSteer2 set, guided this step. Almost all alignment data was machine-generated, and the generation pipeline is released as well. It covers prompt creation, responses, dialogues, filtering and ranking. The base model matches or beats Llama-3 70B, Mixtral 8x22B and Qwen-2 72B on most reasoning benchmarks, and the reward model placed first on RewardBench. The work shows that a permissively licensed model can produce the alignment data it needs and can also generate training data for smaller models.

## Key points
- Pretraining used 9T tokens: 70% English, 15% multilingual (53 natural languages) and 15% code (43 programming languages). It ran as 8T tokens of main pretraining plus 1T tokens of continued training.
- The model has about 9.4B embedding and 331.6B non-embedding parameters. Training ran on 768 DGX H100 nodes (up to 6,144 GPUs) at roughly 41–42% MFU. In FP8, the model fits on a single 8-GPU DGX H100.
- Nemotron-4-340B-Base scores ARC-c 94.28, Winogrande 89.50, Hellaswag 90.53, BBH 85.44, MMLU 81.10 and HumanEval 57.32.
- Over 98% of the alignment data is synthetic. HelpSteer2 (about 10k human preference annotations) is released to train the reward model, which beat GPT-4o-0513 and Gemini 1.5 Pro-0514 on RewardBench.
- Safety testing used AEGIS and the Garak scanner. Overall safety was comparable to Llama-3-70B-Instruct, but weak spots remain: jailbreaks, some malware requests and adversarial hallucination.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 84 not yet.

- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://md.research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions](https://md.research.elamur.ai/free/en/papers/noel-2026-eiffel-novel-benchmark-measure-bias/)
- [GhostWriter: Hidden AI-Generated Texts over Multiple Languages, Domains and Generators](https://md.research.elamur.ai/free/en/papers/schaaf-2026-ghostwriter-hidden-ai-generated-texts/)
- [STRUDEL: Unrolling a Benchmark for Evaluating Vision-Language Models on Structured Diagram Understanding across Domains](https://md.research.elamur.ai/free/en/papers/steinigen-2026-strudel-unrolling-benchmark-evaluating-vision/)
- [Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models](https://md.research.elamur.ai/free/en/papers/takayama-2026-evaluating-effect-question-wording-variations/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{adler2024nemotron,
  title = {{Nemotron-4 340B Technical Report}},
  author = {Adler, Nvidia: Bo and Agarwal, Niket and Aithal, Ashwath and Anh, Dong H. and Bhattacharya, Pallab and Brundyn, Annika and Casper, Jared and Catanzaro, Bryan and Clay, Sharon and Cohen, Jonathan and Das, Sirshak and Dattagupta, Ayush and Delalleau, Olivier and Derczynski, Leon and Dong, Yi and Egert, Daniel and Evans, Ellie and Ficek, Aleksander and Fridman, Denys and Ghosh, Shaona and Ginsburg, Boris and Gitman, Igor and Grzegorzek, Tomasz and Hero, Robert and Huang, Jining and Jawa, Vibhu and Jennings, Joseph and Jhunjhunwala, Aastha and Kamalu, John and Khan, Sadaf and Kuchaiev, Oleksii and LeGresley, Patrick and Li, Hui and Liu, Jiwei and Liu, Zihan and Long, Eileen and Mahabaleshwarkar, Ameya Sunil and Majumdar, Somshubra and Maki, James and Martinez, Miguel and Melo, Maer Rodrigues de and Moshkov, Ivan and Narayanan, Deepak and Narenthiran, Sean and Navarro, Jesus and Nguyen, Phong and Nitski, Osvald and Noroozi, Vahid and Nutheti, Guruprasad and Parisien, Christopher and Parmar, Jupinder and Patwary, Mostofa and Pawelec, Krzysztof and Ping, Wei and Prabhumoye, Shrimai and Roy, Rajarshi and Saar, Trisha and Sabavat, Vasanth Rao Naik and Satheesh, Sanjeev and Scowcroft, Jane Polak and Sewall, Jason and Shamis, Pavel and Shen, Gerald and Shoeybi, Mohammad and Sizer, Dave and Smelyanskiy, Misha and Soares, Felipe and Sreedhar, Makesh Narsimhan and Su, Dan and Subramanian, Sandeep and Sun, Shengyang and Toshniwal, Shubham and Wang, Hao and Wang, Zhilin and You, Jiaxuan and Zeng, Jiaqi and Zhang, Jimmy and Zhang, Jing and Zhang, Vivienne and Zhang, Yian and Zhu, Chen},
  year = {2024},
  doi = {10.48550/arxiv.2406.11704},
  eprint = {2406.11704},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11704}
}
```

- [← previous Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://md.research.elamur.ai/free/en/papers/zhang-2024-negative-preference-optimization-catastrophic-collapse/)
- [next → Neo-Assyrian Metaphors through the Telescope: Linguistic Patterns involving Body Part Constructions in the State Archives Letter Corpus](https://md.research.elamur.ai/free/en/papers/ong-2024-neo-assyrian-metaphors-through-telescope/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

72 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. NVLink and NVSwitch. [https://www.nvidia.com/en-us/data-center/nvlink/](https://www.nvidia.com/en-us/data-center/nvlink/). Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv preprint arXiv:2305.13245, 2023. [↗](https://www.nvidia.com/en-us/data-center/nvlink/) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
2. Allen AI. Reward bench leaderboard. [https://huggingface.co/spaces/allenai/reward-bench](https://huggingface.co/spaces/allenai/reward-bench), 2024. [↗](https://huggingface.co/spaces/allenai/reward-bench)
3. Importantly, the safety model can make both false positive and false negative errors. Future work will include human ground truth labels to quantify the false prediction rate. * indicates equal contribution † indicates work done while at NVIDIA AI Anthropic. The claude 3 model family: Opus, sonnet,… [find](https://scholar.google.com/scholar?q=Importantly%2C+the+safety+model+can+make+both+false+positive+and+false+negative+errors%2E+Future+work+will+include+human+ground+truth+labels+to+quantify+the+false+prediction+rate%2E+%2A+indicates+equal+contri)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+Cai%2C+Michael+Terry%2C+Quoc+Le%2C+and+Charles+Sutton%2E+Program+Synthesis+with+Large+Language+Mo) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
5. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024… [find](https://scholar.google.com/scholar?q=Mohammad+Gheshlaghi+Azar%2C+Zhaohan+Daniel+Guo%2C+Bilal+Piot%2C+Remi+Munos%2C+Mark+Rowland%2C+Michal+Valko%2C+and+Daniele+Calandriello%2E+A+general+theoretical+paradigm+to+understand+learning+from+human+preferences) [our card](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/)
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
7. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. [find](https://scholar.google.com/scholar?q=Collin+Burns%2C+Pavel+Izmailov%2C+Jan+Hendrik+Kirchner%2C+Bowen+Baker%2C+Leo+Gao%2C+Leopold+Aschenbrenner%2C+Yining+Chen%2C+Adrien+Ecoffet%2C+Manas+Joglekar%2C+Jan+Leike%2C+et+al%2E+Weak%2Dto%2Dstrong+generalization%3A+Eliciting) [our card](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/)
8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+Mich) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122, 2021b. [find](https://scholar.google.com/scholar?q=Zhiyu+Chen%2C+Wenhu+Chen%2C+Charese+Smiley%2C+Sameena+Shah%2C+Iana+Borova%2C+Dylan+Langdon%2C+Reema+Moussa%2C+Matt+Beane%2C+Ting%2DHao+Huang%2C+Bryan+Routledge%2C+et+al%2E+Finqa%3A+A+dataset+of+numerical+reasoning+over+financi) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-finqa-dataset-numerical-reasoning-over/)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+et+al%2E+PaLM%3A+Scaling+Language+Modeling+w)
11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think You have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+Think+You+have+Solved+Question+Answering%3F+Try+ARC%2C+the+AI2+Reasoning+Challenge%2E+arXiv+pre)
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
13. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
14. Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. garak: A Framework for Security Probing Large Language Models. arXiv preprint arXiv:2406.11036, 2024. — [arXiv:2406.11036](https://arxiv.org/abs/2406.11036) [find](https://scholar.google.com/scholar?q=Leon+Derczynski%2C+Erick+Galinkin%2C+Jeffrey+Martin%2C+Subho+Majumdar%2C+and+Nanna+Inie%2E+garak%3A+A+Framework+for+Security+Probing+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2406%2E11036%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E110)
15. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yulin+Chen%2C+Bokai+Xu%2C+Yujia+Qin%2C+Zhi+Zheng%2C+Shengding+Hu%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Bowen+Zhou%2E+Enhancing+chat+language+models+by+scaling+high%2Dquality+instructional+conversations%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/)
16. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bala%CC%81zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
17. Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024. — [arXiv:2405.19316](https://arxiv.org/abs/2405.19316) [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Jacob+Eisenstein%2C+Vicky+Zayats%2C+Alekh+Agarwal%2C+Ahmad+Beirami%2C+Chirag+Nagpal%2C+Pete+Shaw%2C+and+Jonathan+Berant%2E+Robust+preference+optimization+through+reward+model+distillation%2E+arXiv+preprin)
18. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-shot Language Model Evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
19. Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts. arXiv preprint arXiv:2404.05993, 2024. — [arXiv:2404.05993](https://arxiv.org/abs/2404.05993) [find](https://scholar.google.com/scholar?q=Shaona+Ghosh%2C+Prasoon+Varshney%2C+Erick+Galinkin%2C+and+Christopher+Parisien%2E+AEGIS%3A+Online+Adaptive+AI+Content+Safety+Moderation+with+Ensemble+of+LLM+Experts%2E+arXiv+preprint+arXiv%3A2404%2E05993%2C+2024%2E+%E2%80%94+%5Bar)
20. Glaive AI. glaive-function-calling-v2. glaive-function-calling-v2, 2023. [https://huggingface.co/datasets/glaiveai/](https://huggingface.co/datasets/glaiveai/) [↗](https://huggingface.co/datasets/glaiveai/)
21. Loubna Ben Allal Anton Lozhkov Colin Raffel Leandro Werra Thomas Wolf Guilherme Penedo, Hynek Kydlı́ček. Fineweb: decanting the web for the finest text data at scale. [https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1), 2024. [↗](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1)
22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+Massive+Multitask+Language+Understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
23. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+math+dataset%2E+In+Thirty%2Dfifth+Confe) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
24. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. [find](https://scholar.google.com/scholar?q=Hakan+Inan%2C+Kartikeya+Upasani%2C+Jianfeng+Chi%2C+Rashi+Rungta%2C+Krithika+Iyer%2C+Yuning+Mao%2C+Michael+Tontchev%2C+Qing+Hu%2C+Brian+Fuller%2C+Davide+Testuggine%2C+et+al%2E+Llama+guard%3A+Llm%2Dbased+input%2Doutput+safeguard+f) [our card](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/)
25. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
26. Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396–403, 2015. [find](https://scholar.google.com/scholar?q=Ankur+Joshi%2C+Saket+Kale%2C+Satish+Chandel%2C+and+D+Kumar+Pal%2E+Likert+scale%3A+Explored+and+explained%2E+British+journal+of+applied+science+%26+technology%2C+7%284%29%3A396%E2%80%93403%2C+2015%2E)
27. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing Activation Recomputation in Large Transformer Models, 2022. [find](https://scholar.google.com/scholar?q=Vijay+Korthikanti%2C+Jared+Casper%2C+Sangkug+Lym%2C+Lawrence+McAfee%2C+Michael+Andersch%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+Reducing+Activation+Recomputation+in+Large+Transformer+Models%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/)
28. Taku Kudo and John Richardson. Sentencepiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. arXiv preprint arXiv:1808.06226, 2018. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+Sentencepiece%3A+A+Simple+and+Language+Independent+Subword+Tokenizer+and+Detokenizer+for+Neural+Text+Processing%2E+arXiv+preprint+arXiv%3A1808%2E06226%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
29. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+Rewardbench%3A+Evaluating+reward+models+for+langua)
30. Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023. [find](https://scholar.google.com/scholar?q=Ariel+N+Lee%2C+Cole+J+Hunter%2C+and+Nataniel+Ruiz%2E+Platypus%3A+Quick%2C+cheap%2C+and+powerful+refinement+of+llms%2E+arXiv+preprint+arXiv%3A2308%2E07317%2C+2023%2E)
31. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for ”mind” exploration of large language model society, 2023. [find](https://scholar.google.com/scholar?q=Guohao+Li%2C+Hasan+Abed+Al+Kader+Hammoud%2C+Hani+Itani%2C+Dmitrii+Khizbullin%2C+and+Bernard+Ghanem%2E+Camel%3A+Communicative+agents+for+%E2%80%9Dmind%E2%80%9D+exploration+of+large+language+model+society%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/)
32. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, april 2024. URL [https://lmsys.org/blog/2024-04-19-arena-hard](https://lmsys.org/blog/2024-04-19-arena-hard). [↗](https://lmsys.org/blog/2024-04-19-arena-hard)
33. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. [find](https://scholar.google.com/scholar?q=Hunter+Lightman%2C+Vineet+Kosaraju%2C+Yura+Burda%2C+Harri+Edwards%2C+Bowen+Baker%2C+Teddy+Lee%2C+Jan+Leike%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Karl+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2E+arXiv+preprint+arXiv%3A2305%2E2005) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
34. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=1qvx61… [↗](https://openreview.net/forum?) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
35. Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, M Shoeybi, and B Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.10225, 2024. [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Wei+Ping%2C+Rajarshi+Roy%2C+Peng+Xu%2C+M+Shoeybi%2C+and+B+Catanzaro%2E+Chatqa%3A+Surpassing+gpt%2D4+on+conversational+qa+and+rag%2E+arXiv+preprint+arXiv%3A2401%2E10225%2C+2024%2E)
36. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+arXiv+p)
37. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024. [find](https://scholar.google.com/scholar?q=Pratyush+Maini%2C+Skyler+Seto%2C+He+Bai%2C+David+Grangier%2C+Yizhe+Zhang%2C+and+Navdeep+Jaitly%2E+Rephrasing+the+web%3A+A+recipe+for+compute+and+data%2Defficient+language+modeling%2C+2024%2E)
38. MetaAI. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/), 2024. [↗](https://ai.meta.com/blog/meta-llama-3/)
39. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2C+2018%2E)
40. Mistral-AI-Team. Mistral large. [https://mistral.ai/news/mistral-large](https://mistral.ai/news/mistral-large), 2024a. [↗](https://mistral.ai/news/mistral-large)
41. Mistral-AI-Team. Mistral 8x22b. [https://mistral.ai/news/mixtral-8x22b](https://mistral.ai/news/mixtral-8x22b), 2024b. [↗](https://mistral.ai/news/mixtral-8x22b)
42. Eric Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2E+A+note+on+dpo+with+noisy+preferences+%26+relationship+to+ipo%2C+2023%2E)
43. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+Efficient+large%2D)
44. NVIDIA. H100 Tensor Core GPU Architecture Overview, 2022. [find](https://scholar.google.com/scholar?q=NVIDIA%2E+H100+Tensor+Core+GPU+Architecture+Overview%2C+2022%2E)
45. OpenAI. Gpt-4-1106-preview. gpt-4-turbo-and-gpt-4, 2023. [https://platform.openai.com/docs/models/](https://platform.openai.com/docs/models/) [↗](https://platform.openai.com/docs/models/)
46. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
47. Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, and Ramón Fernandez Astudillo. Brain: Bayesian reward-conditioned amortized inference for natural language generation from feedback. arXiv preprint arXiv:2402.02479, 2024. — [arXiv:2402.02479](https://arxiv.org/abs/2402.02479) [find](https://scholar.google.com/scholar?q=Gaurav+Pandey%2C+Yatin+Nandwani%2C+Tahira+Naseem%2C+Mayank+Mishra%2C+Guangxuan+Xu%2C+Dinesh+Raghu%2C+Sachindra+Joshi%2C+Asim+Munawar%2C+and+Ramo%CC%81n+Fernandez+Astudillo%2E+Brain%3A+Bayesian+reward%2Dconditioned+amortized+inf)
48. Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu,… [find](https://scholar.google.com/scholar?q=Jupinder+Parmar%2C+Shrimai+Prabhumoye%2C+Joseph+Jennings%2C+Mostofa+Patwary%2C+Sandeep+Subramanian%2C+Dan+Su%2C+Chen+Zhu%2C+Deepak+Narayanan%2C+Aastha+Jhunjhunwala%2C+Ayush+Dattagupta%2C+Vibhu+Jawa%2C+Jiwei+Liu%2C+Ameya+Maha)
49. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015. [find](https://scholar.google.com/scholar?q=Panupong+Pasupat+and+Percy+Liang%2E+Compositional+semantic+parsing+on+semi%2Dstructured+tables%2E+arXiv+preprint+arXiv%3A1508%2E00305%2C+2015%2E)
50. Qwen-Team. Hello qwen2. [https://qwenlm.github.io/blog/qwen2](https://qwenlm.github.io/blog/qwen2), 2024. [↗](https://qwenlm.github.io/blog/qwen2)
51. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Archit+Sharma%2C+Eric+Mitchell%2C+Christopher+D+Manning%2C+Stefano+Ermon%2C+and+Chelsea+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+Advances+in+Neura)
52. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text%2Dto%2DText+Trans)
53. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. [find](https://scholar.google.com/scholar?q=Corby+Rosset%2C+Ching%2DAn+Cheng%2C+Arindam+Mitra%2C+Michael+Santacroce%2C+Ahmed+Awadallah%2C+and+Tengyang+Xie%2E+Direct+nash+optimization%3A+Teaching+language+models+to+self%2Dimprove+with+general+preferences%2E+arXiv+p)
54. RyokoAI. RyokoAI/ShareGPT52K. [https://huggingface.co/datasets/RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K), 2023. [↗](https://huggingface.co/datasets/RyokoAI/ShareGPT52K)
55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale. In AAAI, 2020. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+WINOGRANDE%3A+An+Adversarial+Winograd+Schema+Challenge+at+Scale%2E+In+AAAI%2C+2020%2E)
56. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692, 2023. [find](https://scholar.google.com/scholar?q=Tomohiro+Sawada%2C+Daniel+Paleka%2C+Alexander+Havrilla%2C+Pranav+Tadepalli%2C+Paula+Vidas%2C+Alexander+Kranias%2C+John+J+Nay%2C+Kshitij+Gupta%2C+and+Aran+Komatsuzaki%2E+Arb%3A+Advanced+reasoning+benchmark+for+large+langu)
57. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models using Model Parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2DLM%3A+Training+Multi%2DBillion+Parameter+Language+Models+using+Model+Parallelism%2E+arXiv+preprin)
58. Makesh Narsimhan Sreedhar, Traian Rebedea, Shaona Ghosh, and Christopher Parisien. Canttalkaboutthis: Aligning language models to stay on topic in dialogues, 2024. [find](https://scholar.google.com/scholar?q=Makesh+Narsimhan+Sreedhar%2C+Traian+Rebedea%2C+Shaona+Ghosh%2C+and+Christopher+Parisien%2E+Canttalkaboutthis%3A+Aligning+language+models+to+stay+on+topic+in+dialogues%2C+2024%2E)
59. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864, 2021. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Ahmed+Murtadha%2C+Bo+Wen%2C+and+Yunfeng+Liu%2E+Roformer%3A+Enhanced+Transformer+with+Rotary+Position+Embedding%2E+arXiv+preprint+arXiv%3A2104%2E09864%2C+2021%2E)
60. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Nathan+Scales%2C+Nathanael+Scha%CC%88rli%2C+Sebastian+Gehrmann%2C+Yi+Tay%2C+Hyung+Won+Chung%2C+Aakanksha+Chowdhery%2C+Quoc+V%2E+Le%2C+Ed+H%2E+Chi%2C+Denny+Zhou%2C+and+Jason+Wei%2E+Challenging+big%2Dbench+tasks+and+whe) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
61. Evan Frick Lisa Dunlap Banghua Zhu Joseph E. Gonzalez Ion Stoica Tianle Li*, Wei-Lin Chiang*. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL [https://lmsys.org/](https://lmsys.org/) blog/2024-04-19-arena-hard/. [↗](https://lmsys.org/)
62. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+Foundation+and+Fine%2Dtun)
63. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762. [↗](http://arxiv.org/abs/1706.03762)
64. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023a. [find](https://scholar.google.com/scholar?q=Xiaoxuan+Wang%2C+Ziniu+Hu%2C+Pan+Lu%2C+Yanqiao+Zhu%2C+Jieyu+Zhang%2C+Satyen+Subramaniam%2C+Arjun+R+Loomba%2C+Shichang+Zhang%2C+Yizhou+Sun%2C+and+Wei+Wang%2E+Scibench%3A+Evaluating+college%2Dlevel+scientific+problem%2Dsolving+a) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/)
65. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+preprint)
66. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023b. [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Yi+Dong%2C+Jiaqi+Zeng%2C+Virginia+Adams%2C+Makesh+Narsimhan+Sreedhar%2C+Daniel+Egert%2C+Olivier+Delalleau%2C+Jane+Polak+Scowcroft%2C+Neel+Kant%2C+Aidan+Swope%2C+et+al%2E+Helpsteer%3A+Multi%2Dattribute+helpfulnes)
67. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024. [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Yi+Dong%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Gerald+Shen%2C+Daniel+Egert%2C+Jimmy+J%2E+Zhang%2C+Makesh+Narsimhan+Sreedhar%2C+and+Oleksii+Kuchaiev%2E+Helpsteer2%3A+Open%2Dsource+dataset+for+training+top%2Dperfor)
68. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+HellaSwag%3A+Can+a+Machine+Really+Finish+Your+Sentence%3F+In+ACL%2C+2019%2E)
69. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Tianle+Li%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Eric+Xing%2C+et+al%2E+Lmsys%2Dchat%2D1m%3A+A+large%2Dscale+real%2Dworld+llm+conversation+dataset) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/)
70. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advance)
71. Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658, 2024b. [find](https://scholar.google.com/scholar?q=Tianyu+Zheng%2C+Ge+Zhang%2C+Tianhao+Shen%2C+Xueling+Liu%2C+Bill+Yuchen+Lin%2C+Jie+Fu%2C+Wenhu+Chen%2C+and+Xiang+Yue%2E+Opencodeinterpreter%3A+Integrating+code+generation+with+execution+and+refinement%2E+arXiv+preprint+ar)
72. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. [find](https://scholar.google.com/scholar?q=Jeffrey+Zhou%2C+Tianjian+Lu%2C+Swaroop+Mishra%2C+Siddhartha+Brahma%2C+Sujoy+Basu%2C+Yi+Luan%2C+Denny+Zhou%2C+and+Le+Hou%2E+Instruction%2Dfollowing+evaluation+for+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E07911%2C+) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)