---
title: "Instruction Pre-Training — paper by Cheng et al. (2024)"
description: "The authors propose Instruction Pre-Training. 109 references, cited by 40 papers here. Among them: 19."
authors: ["Daixuan Cheng", "Yuxian Gu", "Shaohan Huang", "Junyu Bi", "Minlie Huang", "Furu Wei"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.148/"
doi: "10.18653/v1/2024.emnlp-main.148"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cheng-2024-instruction-pre-training-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cheng-2024-instruction-pre-training-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Instruction Pre-Training: Language Models are Supervised Multitask Learners

# Instruction Pre-Training: Language Models are Supervised Multitask Learners

**Authors:** [Daixuan Cheng](https://md.research.elamur.ai/free/en/authors/daixuan-cheng/), [Yuxian Gu](https://md.research.elamur.ai/free/en/authors/yuxian-gu/), [Shaohan Huang](https://md.research.elamur.ai/free/en/authors/shaohan-huang/), Junyu Bi, [Minlie Huang](https://md.research.elamur.ai/free/en/authors/minlie-huang/) and [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.148/) · [PDF](https://aclanthology.org/2024.emnlp-main.148.pdf) · [DOI 10.18653/v1/2024.emnlp-main.148](https://doi.org/10.18653/v1/2024.emnlp-main.148) · [OpenAlex](https://openalex.org/W4404782759)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/cheng-2024-instruction-pre-training-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Modern language models are usually pre-trained on raw text with next-token prediction, while supervised multitask learning is saved for post-training. This paper asks whether supervised signals can help during pre-training itself. The authors propose Instruction Pre-Training. Each raw document is enriched with synthetic instruction-response pairs grounded in its content, and the model is pre-trained on this augmented text. All other training settings stay unchanged.

The pairs come from an instruction synthesizer. It is a fine-tuned open-source model of about 7B parameters, trained on many reformatted context-based task datasets. Multi-round inference lets it produce few-shot style examples. Using it, the team generated 200M pairs spanning more than 40 task categories.

The method was tested in two settings: pre-training from scratch and domain-adaptive continual pre-training. In both, it beat standard pre-training. Models trained this way also gained more from later instruction tuning. The work matters because it shows a cheap, scalable way to inject supervised multitask data into pre-training without relying on closed-source models.

## Key points
- **Data scale:** the synthesizer produced 200M instruction-response pairs. For the general corpus, they cover 49 task categories grouped into 9 scenarios, with commonsense reasoning the largest share at 23%.
- **Pre-training from scratch:** a 500M model trained on 100B tokens matched a 1B model trained on 300B tokens.
- **Continual pre-training:** Llama3-8B was adapted to finance and biomedicine and became comparable to, or better than, Llama3-70B.
- **Quality of synthetic pairs (GPT-4 evaluation):**

| Corpus | Response accuracy | Context relevance |
|---|---|---|
| General | 77.5% | 92.9% |
| BioMed | 86.2% | 99.4% |
| Finance | 69.8% | 85.8% |

  The authors flag possible hallucinations as a limitation, and their experiments stay at the scale of billions of tokens.
- **Resources:** the model, code and data are released via github.com/microsoft/LMOps and huggingface.co/instruction-pretrain.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [State Key Laboratory of Intelligent Technology and Systems](https://md.research.elamur.ai/free/en/labs/state-key-laboratory-of-intelligent-technology-and-systems/) — Research institute, Beijing, China · 24 works · 9,850 citations
- [Beijing National Research Center for Information Science and Technology](https://md.research.elamur.ai/free/en/labs/beijing-national-research-center-for-information-science-and-technology/) — Research institute, Beijing, China · 33 works · 6,706 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang and Furu Wei. 2024. Instruction Pre-Training: Language Models are Supervised Multitask Learners. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cheng2024instruction,
  title = {{Instruction Pre-Training: Language Models are Supervised Multitask Learners}},
  author = {Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang and Furu Wei},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.148},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.148}
}
```

- [← previous BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/)
- [next → InternLM2.5-StepProver: Advancing Automated Theorem Proving via Critic-Guided Search](https://md.research.elamur.ai/free/en/papers/wu-2024-internlm2-5-stepprover-advancing-automated/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

109 references, 60 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. 2022. [Topiocqa: Open-domain conversational question answering with topic switching](https://md.research.elamur.ai/free/en/papers/adlakha-2022-topiocqa-open-domain-conversational-question/). Transactions of the Association for Computational Linguistics, 10:468–483.
2. Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. [Explanations for commonsenseqa: New dataset and models](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe…
3. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/). arXiv preprint arXiv:2402.16827.
4. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867.
5. Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 84–90. [find](https://scholar.google.com/scholar?q=Julio+Cesar+Salinas+Alvarado%2C+Karin+Verspoor%2C+and+Timothy+Baldwin%2E+2015%2E+Domain+adaption+of+named+entity+recognition+to+support+credit+risk+assessment%2E+In+Proceedings+of+the+Australasian+Language+Tech)
6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
7. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
8. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
10. Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Milan Deriu, Mark Cieliebak, and Eneko Agirre. 2020. Doqa-accessing domain-specific faqs via conversational qa. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7302–7314. [find](https://scholar.google.com/scholar?q=Jon+Ander+Campos%2C+Arantxa+Otegi%2C+Aitor+Soroa%2C+Jan+Milan+Deriu%2C+Mark+Cieliebak%2C+and+Eneko+Agirre%2E+2020%2E+Doqa%2Daccessing+domain%2Dspecific+faqs+via+conversational+qa%2E+In+Proceedings+of+the+58th+Annual+Meet)
11. Rich Caruana. 1997. Multitask learning. Machine learning, 28:41–75. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1997%2E+Multitask+learning%2E+Machine+learning%2C+28%3A41%E2%80%9375%2E)
12. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. 2024. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Mayee+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+R%C3%A9%2E+2024%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2E+Advance)
13. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. [Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering](https://md.research.elamur.ai/free/en/papers/chen-2022-convfinqa-exploring-chain-numerical-reasoning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279– 6…
14. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Daixuan+Cheng%2C+Shaohan+Huang%2C+and+Furu+Wei%2E+2023%2E+Adapting+large+language+models+via+reading+comprehension%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
15. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Aviya+Skowron%2C+Lintang+Sutawika%2C+Eric+Tang%2C+Anish+Thite%2C+Ben+Wang%2C+Kevin+Wang%2C+and+Andy+Zou%2E+2023%2E+A+framework+for+few%2Dshot+language+model+evaluation%2E)
16. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. [Quac: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184.
17. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Pre-training to learn in context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4849–4870. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2023%2E+Pre%2Dtraining+to+learn+in+context%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Paper)
18. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
19. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2024%2E+Scaling+instruction%2Dfinetuned+language+models%2E+J)
20. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
21. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
22. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang…
23. Franck Dernoncourt and Ji Young Lee. 2017. [Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts](https://md.research.elamur.ai/free/en/papers/dernoncourt-2017-pubmed-200k-rct-dataset-sequential/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 308–313.
24. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
25. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huan… — [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/)
26. Yuxian Gu, Pei Ke, Xiaoyan Zhu, and Minlie Huang. 2022b. Learning instructions with unlabeled data for zero-shot cross-task generalization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1617–1634. [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Pei+Ke%2C+Xiaoyan+Zhu%2C+and+Minlie+Huang%2E+2022b%2E+Learning+instructions+with+unlabeled+data+for+zero%2Dshot+cross%2Dtask+generalization%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+i)
27. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
28. Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2022. Wikiwhy: Answering and explaining cause-and-effect questions. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Matthew+Ho%2C+Aditya+Sharma%2C+Justin+Chang%2C+Michael+Saxon%2C+Sharon+Levy%2C+Yujie+Lu%2C+and+William+Yang+Wang%2E+2022%2E+Wikiwhy%3A+Answering+and+explaining+cause%2Dand%2Deffect+questions%2E+In+The+Eleventh+International+)
29. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428.
30. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. [Cosmos qa: Machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
31. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
32. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024a. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024a%2E+Mixtral+)
33. Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. 2024b. Improving domain adaptation through extended-text reading comprehension. arXiv preprint arXiv:2401.07284. 2016. Chemprot-3.0: a global chemical biology diseases… — [arXiv:2401.07284](https://arxiv.org/abs/2401.07284) [find](https://scholar.google.com/scholar?q=Ting+Jiang%2C+Shaohan+Huang%2C+Shengyue+Luo%2C+Zihan+Zhang%2C+Haizhen+Huang%2C+Furu+Wei%2C+Weiwei+Deng%2C+Feng+Sun%2C+Qi+Zhang%2C+Deqing+Wang%2C+et+al%2E+2024b%2E+Improving+domain+adaptation+through+extended%2Dtext+reading+com)
34. Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. 2024c. Instruction-tuned language models are better knowledge learners. arXiv preprint arXiv:2402.12847. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Zhiqing+Sun%2C+Weijia+Shi%2C+Pedro+Rodriguez%2C+Chunting+Zhou%2C+Graham+Neubig%2C+Xi+Victoria+Lin%2C+Wen%2Dtau+Yih%2C+and+Srinivasan+Iyer%2E+2024c%2E+Instruction%2Dtuned+language+models+are+better+knowledge)
35. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
36. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
37. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
38. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
39. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. [Fasttext](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/). zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
40. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pages 4… [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Minjoon+Seo%2C+Dustin+Schwenk%2C+Jonghyun+Choi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comp)
41. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
42. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Peter+Clark%2C+Michal+Guerquin%2C+Peter+Jansen%2C+and+Ashish+Sabharwal%2E+2020%2E+Qasc%3A+A+dataset+for+question+answering+via+sentence+composition%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificia)
43. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328.
44. Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. [find](https://scholar.google.com/scholar?q=Jens+Kringelum%2C+Sonny+Kim+Kjaerulff%2C+S%C3%B8ren+Brunak%2C+Ole+Lund%2C+Tudor+I+Oprea%2C+and+Olivier+Taboureau%2E)
45. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794.
46. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. [Qed: A framework and dataset for explanations in question answering](https://md.research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/). Transactions of the Association for computational Linguistics, 9:790–806.
47. Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anu-manchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2024. Llm2llm: Boosting llms with novel iterative data enhancement. arXiv preprint arXiv:2403.15042. [find](https://scholar.google.com/scholar?q=Nicholas+Lee%2C+Thanakul+Wattanawong%2C+Sehoon+Kim%2C+Karttikeya+Mangalam%2C+Sheng+Shen%2C+Gopala+Anu%2Dmanchipali%2C+Michael+W+Mahoney%2C+Kurt+Keutzer%2C+and+Amir+Gholami%2E+2024%2E+Llm2llm%3A+Boosting+llms+with+novel+itera)
48. Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. 2022. xformers: A modular and hackable transformer modelling library… [find](https://scholar.google.com/scholar?q=Benjamin+Lefaudeux%2C+Francisco+Massa%2C+Diana+Liskovich%2C+Wenhan+Xiong%2C+Vittorio+Caggiano%2C+Sean+Naren%2C+Min+Xu%2C+Jieru+Hu%2C+Marta+Tintore%2C+Susan+Zhang%2C+Patrick+Labatut%2C+Daniel+Haziza%2C+Luca+Wehrstedt%2C+Jeremy+)
49. Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. 2024. [Synthetic data (almost) from scratch: Generalized instruction tuning for language models](https://md.research.elamur.ai/free/en/papers/li-2024-synthetic-data-almost-scratch-generalized/). arXiv preprint arXiv:2402.13064.
50. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2023a. Self-alignment with instruction back-translation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xian+Li%2C+Ping+Yu%2C+Chunting+Zhou%2C+Timo+Schick%2C+Omer+Levy%2C+Luke+Zettlemoyer%2C+Jason+E+Weston%2C+and+Mike+Lewis%2E+2023a%2E+Self%2Dalignment+with+instruction+back%2Dtranslation%2E+In+The+Twelfth+International+Confere)
51. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463.
52. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%22Teknium%22%2E+2023%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca)
53. Yichan Liang, Jianheng Li, and Jian Yin. 2019. A new multi-choice reading comprehension dataset for curriculum learning. In Asian Conference on Machine Learning, pages 742–757. PMLR. [find](https://scholar.google.com/scholar?q=Yichan+Liang%2C+Jianheng+Li%2C+and+Jian+Yin%2E+2019%2E+A+new+multi%2Dchoice+reading+comprehension+dataset+for+curriculum+learning%2E+In+Asian+Conference+on+Machine+Learning%2C+pages+742%E2%80%93757%2E+PMLR%2E)
54. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. [Reasoning over paragraph effects in situations](https://md.research.elamur.ai/free/en/papers/lin-2019-reasoning-over-paragraph-effects-situations/). In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58– 62.
55. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
56. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. [Logiqa: a challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3622…
57. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. [Best practices and lessons learned on synthetic data](https://md.research.elamur.ai/free/en/papers/liu-2024-best-practices-lessons-learned-synthetic/) for language models. arXiv preprint arXiv:2404.07503.
58. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). In International Conference on Machine Learning, pages 22631–22648. PMLR.
59. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942. [find](https://scholar.google.com/scholar?q=Macedo+Maia%2C+Siegfried+Handschuh%2C+Andr%C3%A9+Freitas%2C+Brian+Davis%2C+Ross+McDermott%2C+Manel+Zarrouk%2C+and+Alexandra+Balahur%2E+2018%2E+Www%E2%80%9918+open+challenge%3A+financial+opinion+mining+and+question+answering%2E+In+Com)
60. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. [Rephrasing the web: A recipe for compute and data-efficient language modeling](https://md.research.elamur.ai/free/en/papers/maini-2024-rephrasing-web-recipe-compute-data/). arXiv preprint arXiv:2401.16380.
61. Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782–796. [find](https://scholar.google.com/scholar?q=Pekka+Malo%2C+Ankur+Sinha%2C+Pekka+Korhonen%2C+Jyrki+Wallenius%2C+and+Pyry+Takala%2E+2014%2E+Good+debt+or+bad+debt%3A+Detecting+semantic+orientations+in+economic+texts%2E+Journal+of+the+Association+for+Information+Sc)
62. Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Effective transfer learning for identifying similar questions: matching user questions to covid-19 faqs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,… [find](https://scholar.google.com/scholar?q=Clara+H+McCreery%2C+Namit+Katariya%2C+Anitha+Kannan%2C+Manish+Chablani%2C+and+Xavier+Amatriain%2E+2020%2E+Effective+transfer+learning+for+identifying+similar+questions%3A+matching+user+questions+to+covid%2D19+faqs%2E+I)
63. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+)
64. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Metaicl%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+fo)
65. Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. [Spartqa: A textual question answering benchmark for spatial reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie…
66. Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020. Covid-qa: A question answering dataset for covid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020. [find](https://scholar.google.com/scholar?q=Timo+M%C3%B6ller%2C+Anthony+Reina%2C+Raghavan+Jayakumar%2C+and+Malte+Pietsch%2E+2020%2E+Covid%2Dqa%3A+A+question+answering+dataset+for+covid%2D19%2E+In+Proceedings+of+the+1st+Workshop+on+NLP+for+COVID%2D19+at+ACL+2020%2E)
67. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
68. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
69. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–…
70. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
71. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
72. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
73. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
74. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
75. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
76. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
77. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. [Mctest: A challenge dataset for the open-domain machine comprehension of text](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 193– 203.
78. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. ACM Computing Surveys, 55(10):1–45. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Matt+Gardner%2C+and+Isabelle+Augenstein%2E+2023%2E+Qa+dataset+explosion%3A+A+taxonomy+of+nlp+resources+for+question+answering+and+reading+comprehension%2E+ACM+Computing+Surveys%2C+55%2810%29%3A1%E2%80%9345%2E)
79. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to ai complete question answering: A set of prerequisite real tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8722–8731. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+Matthew+Downey%2C+and+Anna+Rumshisky%2E+2020%2E+Getting+closer+to+ai+complete+question+answering%3A+A+set+of+prerequisite+real+tasks%2E+In+Proceedings+of+the+AAAI+conference+on+artif)
80. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
81. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). In International Conference on Learning Representations.
82. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2019%2E+Social+iqa%3A+Commonsense+reasoning+about+social+interactions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in)
83. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. 2023. [In-context pretraining: Language modeling beyond document boundaries](https://md.research.elamur.ai/free/en/papers/shi-2023-context-pretraining-language-modeling-document/). In The Twelfth International Conference on Learning Representations.
84. Ankur Sinha and Tanmay Khandait. 2021. Impact of news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 589–601. Springer. [find](https://scholar.google.com/scholar?q=Ankur+Sinha+and+Tanmay+Khandait%2E+2021%2E+Impact+of+news+on+the+commodity+market%3A+Dataset+and+results%2E+In+Advances+in+Information+and+Communication%3A+Proceedings+of+the+2021+Future+of+Information+and+Comm)
85. Amir Soleimani, Christof Monz, and Marcel Worring. 2021. Nlquad: A non-factoid long question answering data set. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1245– 1255. [find](https://scholar.google.com/scholar?q=Amir+Soleimani%2C+Christof+Monz%2C+and+Marcel+Worring%2E+2021%2E+Nlquad%3A+A+non%2Dfactoid+long+question+answering+data+set%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Association+for+Co)
86. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. [Dream: A challenge data set and models for dialogue-based reading comprehension](https://md.research.elamur.ai/free/en/papers/sun-2019-dream-challenge-data-set-models/). Transactions of the Association for Computational Linguistics, 7:217–231.
87. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. [Quartz: An open-domain dataset of qualitative relationship questions](https://md.research.elamur.ai/free/en/papers/tafjord-2019-quartz-open-domain-dataset-qualitative/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E…
88. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
89. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
90. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. [Newsqa: A machine comprehension dataset](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/). In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200.
91. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Proce)
92. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+et+al%2E+2022%2E+Super%2Dnaturalinst)
93. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations.
94. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
95. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. 2020. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003– 4012.
96. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698.
97. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dpara)
98. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+2024%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretrai)
99. Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Tweetqa: A social media focused question answering dataset. arXiv preprint arXiv:1907.06292. [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Jiawei+Wu%2C+Hong+Wang%2C+Vivek+Kulkarni%2C+Mo+Yu%2C+Shiyu+Chang%2C+Xiaoxiao+Guo%2C+and+William+Yang+Wang%2E+2019%2E+Tweetqa%3A+A+social+media+focused+question+answering+dataset%2E+arXiv+preprint+arXiv%3A1907)
100. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–…
101. Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. 2024. Genie: Achieving human parity in content-grounded datasets generation. arXiv preprint arXiv:2401.14367. — [arXiv:2401.14367](https://arxiv.org/abs/2401.14367) [find](https://scholar.google.com/scholar?q=Asaf+Yehudai%2C+Boaz+Carmeli%2C+Yosi+Mass%2C+Ofir+Arviv%2C+Nathaniel+Mills%2C+Assaf+Toledo%2C+Eyal+Shnarch%2C+and+Leshem+Choshen%2E+2024%2E+Genie%3A+Achieving+human+parity+in+content%2Dgrounded+datasets+generation%2E+arXiv+p)
102. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2019. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). In International Conference on Learning Representations.
103. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. [Mammoth2: Scaling instructions from the web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/). arXiv preprint arXiv:2405.03548.
104. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
105. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
106. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2024%2E+Lima%3A+Less+is+more+for+alignment%2E+Advances+in+Neural+Information+Proc)
107. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2023. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations.
108. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, et al. 2022. [Fantastic questions and where to find them: Fairytaleqa–an authentic dataset for narrative comprehension](https://md.research.elamur.ai/free/en/papers/xu-2022-fantastic-questions-where-find-them/). In Proceedings of the 60th Annual Meeting of the Associati…
109. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. [Fingpt: Open-source financial large language models](https://md.research.elamur.ai/free/en/papers/yang-2023-fingpt-open-source-financial-large/). FinLLM at IJCAI.

*Updated 2 November 2026*