---
title: "HSS-Synth — paper by Peng et al. (2026) — summary and ideas"
description: "They define 14 HSS fields, including Law, Economics, Geography, History and Psychology, and build HSS-Synth, a… 67 references."
authors: ["Ru Peng", "Tianyu Zhao", "Xijun Gu", "Zhiting Fan", "Haokai Xu", "Jinyang Zhang", "Yawen Zeng", "Yihong Zhuang", "Kexin Yang", "Junyang Lin", "Dayiheng Liu", "Junbo Zhao"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1880/"
doi: "10.18653/v1/2026.findings-acl.1880"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2026-hss-synth-humanities-social-sciences/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2026-hss-synth-humanities-social-sciences/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

# HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

**Authors:** Ru Peng, [Tianyu Zhao](https://md.research.elamur.ai/free/en/authors/tianyu-zhao/), Xijun Gu, Zhiting Fan, Haokai Xu, [Jinyang Zhang](https://md.research.elamur.ai/free/en/authors/jinyang-zhang/), [Yawen Zeng](https://md.research.elamur.ai/free/en/authors/yawen-zeng/), Yihong Zhuang, Kexin Yang, [Junyang Lin](https://md.research.elamur.ai/free/en/authors/junyang-lin/), [Dayiheng Liu](https://md.research.elamur.ai/free/en/authors/dayiheng-liu/) and [Junbo Zhao](https://md.research.elamur.ai/free/en/authors/junbo-zhao/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1880/) · [PDF](https://aclanthology.org/2026.findings-acl.1880.pdf) · [DOI 10.18653/v1/2026.findings-acl.1880](https://doi.org/10.18653/v1/2026.findings-acl.1880) · [OpenAlex](https://openalex.org/W7166813621)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Synthetic training data for LLMs has worked well in domains with checkable answers, such as math and code. The humanities and social sciences (HSS) have received little attention because their open-ended questions need judgment rather than a single correct answer. The authors organize the work by subject rather than by skill. They define 14 HSS fields, including Law, Economics, Geography, History and Psychology, and build HSS-Synth, a three-stage pipeline:

1. **Seed documents.** Web texts from SlimPajama are filtered, classified by domain, scored against quality rubrics, and cleaned by an LLM. A judge model then checks the cleaned versions.
2. **Instructions.** Each seed document is turned into an instruction using explicit requirements plus a persona. A consistency check confirms the question matches its source.
3. **Teacher-forced answering (TeachForceA).** The model sees the seed document while writing the answer, which keeps facts and tone grounded.

The resulting 237k-sample dataset beats 14 synthesis baselines across 16 benchmarks. A fine-tuned Qwen3-8B-Base comes close to the official instruction-tuned Qwen3-8B. It gains on human-preference alignment without sacrificing knowledge scores.

## Key points

- **Seed sources and filtering:** Seeds come from the 627B-token SlimPajama corpus, using the FineWeb toolkit (fastText, Gopher/C4/FineWeb filters, MinHash). Twelve rubrics in three tiers (readability, applicability, human touch) decide what is kept.
- **Evaluation and transfer:** The 237k samples were tested across eight capability areas. Gains carried over to Llama3.1-8B and Qwen2.5-14B.
- **Grounding of answers:** Teacher-forced answers scored best on rubric scores and MTLD, and they reused more seed-document text than reverse answers did.
- **Rubric validation:** Over 400 texts were scored by 20 human raters. The synthesis model agreed with the human majority in 86% of cases and with GPT-4.1 in 91%.
- **Limitations:** Anthropology and religious studies are not covered. Gains in commonsense reasoning, long-context understanding and reading comprehension were marginal. Code is available at github.com/pengr/HSS-Synth.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Ru Peng, Tianyu Zhao, Xijun Gu, Zhiting Fan, Haokai Xu, Jinyang Zhang, Yawen Zeng, Yihong Zhuang, Kexin Yang, Junyang Lin, Dayiheng Liu and Junbo Zhao. 2026. HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peng2026hss,
  title = {{HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs}},
  author = {Ru Peng, Tianyu Zhao, Xijun Gu, Zhiting Fan, Haokai Xu, Jinyang Zhang, Yawen Zeng, Yihong Zhuang, Kexin Yang, Junyang Lin, Dayiheng Liu and Junbo Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1880},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1880}
}
```

- [← previous HowToNarrate: A General-Domain Benchmark for Synchronized Video Narration with External Knowledge](https://md.research.elamur.ai/free/en/papers/wang-2026-howtonarrate-general-domain-benchmark-synchronized/)
- [next → HTMR: Hybrid Token Masking Reinforcement Learning with Verifiable Rewards for Event Argument Extraction with Multi-Perspective Reasoning](https://md.research.elamur.ai/free/en/papers/luo-2026-htmr-hybrid-token-masking-reinforcement/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

67 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905.
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
3. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. [Longwriter: Unleashing 10,000+ word generation from long context llms](https://md.research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/). arXiv preprint arXiv:2408.07055.
4. Loubna Ben, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. Cosmopedia. [find](https://scholar.google.com/scholar?q=Loubna+Ben%2C+Anton+Lozhkov%2C+Guilherme+Penedo%2C+Thomas+Wolf%2C+and+Leandro+von+Werra%2E+2024%2E+Cosmopedia%2E)
5. Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE. [find](https://scholar.google.com/scholar?q=Andrei+Z+Broder%2E+1997%2E+On+the+resemblance+and+containment+of+documents%2E+In+Proceedings%2E+Compression+and+Complexity+of+SEQUENCES+1997+%28Cat%2E+No%2E+97TB100171%29%2C+pages+21%E2%80%9329%2E+IEEE%2E)
6. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2023. [Large language models as tool makers](https://md.research.elamur.ai/free/en/papers/cai-2023-large-language-models-tool-makers/). arXiv preprint arXiv:2305.17126.
7. Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, and Kai Chen. 2025. [Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement](https://md.research.elamur.ai/free/en/papers/cao-2025-condor-enhance-llm-alignment-knowledge/). arXiv preprint arXiv:2501.12273.
8. Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. 2024a. DoG-instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… [find](https://scholar.google.com/scholar?q=Yongrui+Chen%2C+Haiyun+Jiang%2C+Xinting+Huang%2C+Shuming+Shi%2C+and+Guilin+Qi%2E+2024a%2E+DoG%2Dinstruct%3A+Towards+premium+instruction%2Dtuning+data+via+text%2Dgrounded+instruction+wrapping%2E+In+Proceedings+of+the+2024+C)
9. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and 1 others. 2024b. [Tombench: Benchmarking theory of mind in large language models](https://md.research.elamur.ai/free/en/papers/chen-2024-tombench-benchmarking-theory-mind-large/). arXiv preprint arXiv:2402.15052.
10. Dan Dan Friedman and Adji Bousso Dieng. 2023. The vendi score: A diversity evaluation metric for machine learning. Transactions on machine learning research. [find](https://scholar.google.com/scholar?q=Dan+Dan+Friedman+and+Adji+Bousso+Dieng%2E+2023%2E+The+vendi+score%3A+A+diversity+evaluation+metric+for+machine+learning%2E+Transactions+on+machine+learning+research%2E)
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
12. Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29. [find](https://scholar.google.com/scholar?q=Alex+M+Lamb%2C+Anirudh+Goyal+ALIAS+PARTH+GOYAL%2C+Ying+Zhang%2C+Saizheng+Zhang%2C+Aaron+C+Courville%2C+and+Yoshua+Bengio%2E+2016%2E+Professor+forcing%3A+A+new+algorithm+for+training+recurrent+networks%2E+Advances+in+ne)
13. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2E)
14. Jing Yang Lee, Hamed Bonab, Nasser Zalmout, Ming Zeng, Sanket Lokegaonkar, Colin Lockard, Binxuan Huang, Ritesh Sarkhel, and Haodong Wang. 2025. Doctalk: Scalable graph-based dialogue synthesis for enhancing llm conversational capabilities. arXiv preprint arXiv:2507.05750. [find](https://scholar.google.com/scholar?q=Jing+Yang+Lee%2C+Hamed+Bonab%2C+Nasser+Zalmout%2C+Ming+Zeng%2C+Sanket+Lokegaonkar%2C+Colin+Lockard%2C+Binxuan+Huang%2C+Ritesh+Sarkhel%2C+and+Haodong+Wang%2E+2025%2E+Doctalk%3A+Scalable+graph%2Dbased+dialogue+synthesis+for+en)
15. Ronen Eldan and Yuanzhi Li. 2023. [Tinystories: How small can language models be and still speak coherent english?](https://md.research.elamur.ai/free/en/papers/eldan-2023-tinystories-small-language-models-still/) arXiv preprint arXiv:2305.07759.
16. Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. [Scaling synthetic data creation with 1,000,000,000 personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/). arXiv preprint arXiv:2406.20094.
17. Xintong Hao, Ruijie Zhu, Ge Zhang, Ke Shen, and Chenggang Li. 2025. Reformulation for pretraining data augmentation. arXiv preprint arXiv:2502.04235. — [arXiv:2502.04235](https://arxiv.org/abs/2502.04235) [find](https://scholar.google.com/scholar?q=Xintong+Hao%2C+Ruijie+Zhu%2C+Ge+Zhang%2C+Ke+Shen%2C+and+Chenggang+Li%2E+2025%2E+Reformulation+for+pretraining+data+augmentation%2E+arXiv+preprint+arXiv%3A2502%2E04235%2E+%E2%80%94+%5BarXiv%3A2502%2E04235%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E04)
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
19. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). arXiv preprint arXiv:2210.11610.
20. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436.
21. Yuxin Jiang, Yufei Wang, Chuhan Wu, Xinyi Dai, Yan Xu, Weinan Gan, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, and 1 others. 2025. Instruction-tuning data synthesis from scratch via web reconstruction. arXiv preprint arXiv:2504.15573. [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Yufei+Wang%2C+Chuhan+Wu%2C+Xinyi+Dai%2C+Yan+Xu%2C+Weinan+Gan%2C+Yasheng+Wang%2C+Xin+Jiang%2C+Lifeng+Shang%2C+Ruiming+Tang%2C+and+1+others%2E+2025%2E+Instruction%2Dtuning+data+synthesis+from+scratch+via+web+recon)
22. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. [Fasttext](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/). zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
23. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328.
24. Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. 2023. Longform: Effective instruction tuning with reverse instructions. arXiv preprint arXiv:2304.08460. [find](https://scholar.google.com/scholar?q=Abdullatif+K%C3%B6ksal%2C+Timo+Schick%2C+Anna+Korhonen%2C+and+Hinrich+Sch%C3%BCtze%2E+2023%2E+Longform%3A+Effective+instruction+tuning+with+reverse+instructions%2E+arXiv+preprint+arXiv%3A2304%2E08460%2E)
25. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. 2023a. [Self-alignment with instruction back-translation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/). arXiv preprint arXiv:2308.06259.
26. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024. Self-alignment with instruction back-translation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xian+Li%2C+Ping+Yu%2C+Chunting+Zhou%2C+Timo+Schick%2C+Omer+Levy%2C+Luke+Zettlemoyer%2C+Jason+E+Weston%2C+and+Mike+Lewis%2E+2024%2E+Self%2Dalignment+with+instruction+back%2Dtranslation%2E+In+The+Twelfth+International+Conferen)
27. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/). arXiv preprint arXiv:2308.03281.
28. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. 2024. [Best practices and lessons learned on synthetic data](https://md.research.elamur.ai/free/en/papers/liu-2024-best-practices-lessons-learned-synthetic/). In First Conference on Language Modeling.
29. Yen-Ju Lu, Ting-Yao Hu, Hema Swetha Koppula, Hadi Pouransari, Jen-Hao Rick Chang, Yin Xia, Xiang Kong, Qi Zhu, Simon Wang, Oncel Tuzel, and 1 others. 2025. Mutual reinforcement of llm dialogue synthesis and summarization capabilities for few-shot dialogue summarization. arXiv preprint arXiv:2502.173… [find](https://scholar.google.com/scholar?q=Yen%2DJu+Lu%2C+Ting%2DYao+Hu%2C+Hema+Swetha+Koppula%2C+Hadi+Pouransari%2C+Jen%2DHao+Rick+Chang%2C+Yin+Xia%2C+Xiang+Kong%2C+Qi+Zhu%2C+Simon+Wang%2C+Oncel+Tuzel%2C+and+1+others%2E+2025%2E+Mutual+reinforcement+of+llm+dialogue+synthes)
30. Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. [Rephrasing the web: A recipe for compute and data-efficient language modeling](https://md.research.elamur.ai/free/en/papers/maini-2024-rephrasing-web-recipe-compute-data/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14…
31. Philip M McCarthy and Scott Jarvis. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392. [find](https://scholar.google.com/scholar?q=Philip+M+McCarthy+and+Scott+Jarvis%2E+2010%2E+Mtld%2C+vocd%2Dd%2C+and+hd%2Dd%3A+A+validation+study+of+sophisticated+approaches+to+lexical+diversity+assessment%2E+Behavior+research+methods%2C+42%282%29%3A381%E2%80%93392%2E)
32. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. [find](https://scholar.google.com/scholar?q=Mary+L+McHugh%2E+2012%2E+Interrater+reliability%3A+the+kappa+statistic%2E+Biochemia+medica%2C+22%283%29%3A276%E2%80%93282%2E)
33. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+evaluation+framework+for+deeper+understanding+of+c)
34. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Nouamane+Tazi%2C+Aleksandra+Piktus%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+A+Raffel%2E+2023%2E+Scaling+data%2Dconstrained+language+models%2E+Advance)
35. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
36. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2025. [A comprehensive overview of large language models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 16(5):1–72.
37. Nihal V Nayak, Yiyang Nan, Avi Trost, and Stephen H Bach. 2024. Learning to generate instruction tuning datasets for zero-shot task adaptation. arXiv preprint arXiv:2402.18334. [find](https://scholar.google.com/scholar?q=Nihal+V+Nayak%2C+Yiyang+Nan%2C+Avi+Trost%2C+and+Stephen+H+Bach%2E+2024%2E+Learning+to+generate+instruction+tuning+datasets+for+zero%2Dshot+task+adaptation%2E+arXiv+preprint+arXiv%3A2402%2E18334%2E)
38. Samuel J Paech. 2023. [Eq-bench: An emotional intelligence benchmark for large language models](https://md.research.elamur.ai/free/en/papers/paech-2023-eq-bench-emotional-intelligence-benchmark/). arXiv preprint arXiv:2312.06281.
39. Samuel J. Paech. 2025. Eq-bench 3: Emotional intelligence benchmark. [https://github.com/](https://github.com/) EQ-bench/eqbench3. Commit \\<hash\> or release \\<tag\>. [↗](https://github.com/)
40. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Advances in Neural Information Processing Systems, 37:30811–30849.
41. Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. 2025. Dataman: Data manager for pre-training large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ru+Peng%2C+Kexin+Yang%2C+Yawen+Zeng%2C+Junyang+Lin%2C+Dayiheng+Liu%2C+and+Junbo+Zhao%2E+2025%2E+Dataman%3A+Data+manager+for+pre%2Dtraining+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Learning+R)
42. Kun Qian, Maximillian Chen, Siyan Li, Arpit Sharma, and Zhou Yu. 2025. Bottom-up synthesis of knowledge-grounded task-oriented dialogues with iteratively self-refined prompts. arXiv preprint arXiv:2504.14375. [find](https://scholar.google.com/scholar?q=Kun+Qian%2C+Maximillian+Chen%2C+Siyan+Li%2C+Arpit+Sharma%2C+and+Zhou+Yu%2E+2025%2E+Bottom%2Dup+synthesis+of+knowledge%2Dgrounded+task%2Doriented+dialogues+with+iteratively+self%2Drefined+prompts%2E+arXiv+preprint+arXiv%3A250)
43. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
44. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+and+1+others%2E+2021%2E+Scaling+language+models%3A+Met)
45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
46. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
47. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
48. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama](https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama). [↗](https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama)
49. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. [Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset](https://md.research.elamur.ai/free/en/papers/su-2025-nemotron-cc-transforming-common-crawl/). arXiv preprint arXiv:2412.02595.
50. Teknium. 2023. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants. [find](https://scholar.google.com/scholar?q=Teknium%2E+2023%2E+Openhermes+2%2E5%3A+An+open+dataset+of+synthetic+data+for+generalist+llm+assistants%2E)
51. ValueByte-AI. 2025. Awesome-llm-in-social-science. [https://github.com/ValueByte-AI/](https://github.com/ValueByte-AI/) Awesome-LLM-in-Social-Science. [↗](https://github.com/ValueByte-AI/)
52. Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, and 1 others. 2024. [Weaver: Foundation models for creative writing](https://md.research.elamur.ai/free/en/papers/wang-2024-weaver-foundation-models-creative-writing/). arXiv preprint arXiv:2401.17268.
53. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Proc)
54. Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, and 1 others. 2024. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/). Advances in neural information processing systems, 37:116462–116492.
55. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and 1 others. 2025. Writingbench: A comprehensive benchmark for generative writing. arXiv preprint arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) [find](https://scholar.google.com/scholar?q=Yuning+Wu%2C+Jiahao+Mei%2C+Ming+Yan%2C+Chenliang+Li%2C+Shaopeng+Lai%2C+Yuran+Ren%2C+Zijia+Wang%2C+Ji+Zhang%2C+Mengyue+Wu%2C+Qin+Jin%2C+and+1+others%2E+2025%2E+Writingbench%3A+A+comprehensive+benchmark+for+generative+writing%2E+a)
56. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024a. [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations.
57. Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao. 2025. From real to synthetic: Synthesizing millions of diversified and complicated user instructions with attributed grounding. arXiv preprint arXiv:2506.03968. [find](https://scholar.google.com/scholar?q=Chiwei+Zhu%2C+Benfeng+Xu%2C+Xiaorui+Wang%2C+and+Zhendong+Mao%2E+2025%2E+From+real+to+synthetic%3A+Synthesizing+millions+of+diversified+and+complicated+user+instructions+with+attributed+grounding%2E+arXiv+preprint+a)
58. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. [Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/). arXiv preprint arXiv:2406.08464.
59. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
60. Shunyu Yao, Howard Chen, Austin W Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023. [Collie: Systematic construction of constrained text generation tasks](https://md.research.elamur.ai/free/en/papers/yao-2023-collie-systematic-construction-constrained-text/). arXiv preprint arXiv:2307.08689.
61. Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In The Twelfth International Conference on Learning Representations.
62. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. [Mammoth2: Scaling instructions from the web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/). Advances in Neural Information Processing Systems.
63. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
64. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. [Wildchat: 1m chatGPT interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). In The Twelfth International Conference on Learning Representations.
65. Zilong Zhao, Robert Birke, and Lydia Y Chen. 2025. Tabula: Harnessing language models for tabular data synthesis. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 247–259. Springer. [find](https://scholar.google.com/scholar?q=Zilong+Zhao%2C+Robert+Birke%2C+and+Lydia+Y+Chen%2E+2025%2E+Tabula%3A+Harnessing+language+models+for+tabular+data+synthesis%2E+In+Pacific%2DAsia+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+pages+247%E2%80%93259%2E+Spri)
66. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. [LlamaFactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thail…
67. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911.

*Updated 2 November 2026*