---
title: "Comparing Specialised Small and General Large Language"
description: "They define two \"break-even points.\" The first is where specialised models win on average. 58 references, cited by 9 papers here."
authors: ["Branislav Pecher", "Ivan Srba", "Maria Bielikova"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.9/"
doi: "10.18653/v1/2025.emnlp-main.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pecher-2025-comparing-specialised-small-general-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pecher-2025-comparing-specialised-small-general-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance

# Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance

**Authors:** [Branislav Pecher](https://md.research.elamur.ai/free/en/authors/branislav-pecher/), [Ivan Srba](https://md.research.elamur.ai/free/en/authors/ivan-srba/) and [Maria Bielikova](https://md.research.elamur.ai/free/en/authors/maria-bielikova/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.9/) · [PDF](https://aclanthology.org/2025.emnlp-main.9.pdf) · [DOI 10.18653/v1/2025.emnlp-main.9](https://doi.org/10.18653/v1/2025.emnlp-main.9) · [OpenAlex](https://openalex.org/W4416037315)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/pecher-2025-comparing-specialised-small-general-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how much labelled data a small, task-specialised language model needs before it beats a large general-purpose model used without parameter updates. The authors compare two families. Specialised models are built by fine-tuning or instruction-tuning. General LLMs are used through zero-shot prompting or few-shot in-context learning. They test 8 language models on 8 English text classification datasets that differ in number of classes, text length and task type. Training data grows exponentially from 10 samples up to the full dataset, and every setting is repeated over many runs to measure sensitivity to randomness.

They define two "break-even points." The first is where specialised models win on average. The second is where they still win after accounting for variance. On average, about 100 labelled samples are enough for specialised models to match or exceed general ones. Accounting for variance roughly doubles or triples that number.

These results give practitioners a concrete basis for deciding how much to annotate and which approach to choose under limited annotation and compute budgets.

## Key points
- **Setup:** Datasets are SST2, AG News, DB Pedia, MRPC, BoolQ, CoLA, SNIPS and TREC. General models include LLaMA-2, LLaMA-3, Mistral-7B, Zephyr-7B and GPT-4. Evaluation uses mean F1 macro and standard deviation on 1,000 randomly sampled test instances, with a 60:20:20 split. GPT-4 was run 10 times rather than 20 because of cost.
- **First break-even point:** Specialised models need roughly 100 labelled samples on average. Fine-tuning on binary datasets and on tasks needing deeper understanding, such as question answering, needs considerably more.
- **Effect of variance:** Including variance raises the required labels by 100–200% on average and by up to 500–3000% in some cases. Most of this variance comes from in-context learning and few-sample fine-tuning.
- **Model size and quantisation:** Bigger general models did not reliably give better performance, data-efficiency or stability. 4-bit quantisation barely changed their behaviour.
- **Limitations and code:** The study covers only English classification, uses a single prompt tuned on Mistral-7B, and may be affected by pre-training data contamination. Code is available at github.com/kinit-sk/L3D-sensitivity-investigation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Brno University of Technology](https://md.research.elamur.ai/free/en/universities/brno-university-of-technology/) — University, Brno, Czechia · 11 works · 32,326 citations
- [Kempelen Institute of Intelligent Technologies](https://md.research.elamur.ai/free/en/ngos/kempelen-institute-of-intelligent-technologies/) — Nonprofit, Bratislava, Slovakia · 11 works · 82 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Branislav Pecher, Ivan Srba and Maria Bielikova. 2025. Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pecher2025comparing,
  title = {{Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance}},
  author = {Branislav Pecher, Ivan Srba and Maria Bielikova},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.9},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.9}
}
```

- [← previous ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation](https://md.research.elamur.ai/free/en/papers/zhao-2025-chartcoder-advancing-multimodal-large-language/)
- [next → Defense Against Prompt Injection Attack by Leveraging Attack Techniques](https://md.research.elamur.ai/free/en/papers/chen-2025-defense-against-prompt-injection-attack/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

58 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. [In-context examples selection for machine translation](https://md.research.elamur.ai/free/en/papers/agrawal-2023-context-examples-selection-machine-translation/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems, 33:1877–1901.
3. Ting-Yun Chang and Robin Jia. 2023. Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8123–8144. ACL. [find](https://scholar.google.com/scholar?q=Ting%2DYun+Chang+and+Robin+Jia%2E+2023%2E+Data+curation+alone+can+stabilize+in%2Dcontext+learning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pa)
4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2024%2E+Scaling+instruction%2Dfinetuned+language+models%2E+J)
5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
6. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. [Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/). arXiv pr…
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). Preprint, arXiv:1810.04805.
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
9. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+arX)
10. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing.
11. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. [A survey for in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv preprint arXiv:2301.00234.
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
14. Odd Erik Gundersen, Kevin Coakley, and Christine Kirkpatrick. 2022a. Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610. [find](https://scholar.google.com/scholar?q=Odd+Erik+Gundersen%2C+Kevin+Coakley%2C+and+Christine+Kirkpatrick%2E+2022a%2E+Sources+of+irreproducibility+in+machine+learning%3A+A+review%2E+arXiv+preprint+arXiv%3A2204%2E07610%2E)
15. Odd Erik Gundersen, Kevin Coakley, Christine Kirkpatrick, and Yolanda Gil. 2022b. Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610. [find](https://scholar.google.com/scholar?q=Odd+Erik+Gundersen%2C+Kevin+Coakley%2C+Christine+Kirkpatrick%2C+and+Yolanda+Gil%2E+2022b%2E+Sources+of+irreproducibility+in+machine+learning%3A+A+review%2E+arXiv+preprint+arXiv%3A2204%2E07610%2E)
16. Odd Erik Gundersen, Saeid Shamsaliei, Håkon Slet-ten Kjærnli, and Helge Langseth. 2023. On reporting robust and trustworthy conclusions from model comparison studies involving neural networks and randomness. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, pages 37–61. [find](https://scholar.google.com/scholar?q=Odd+Erik+Gundersen%2C+Saeid+Shamsaliei%2C+H%C3%A5kon+Slet%2Dten+Kj%C3%A6rnli%2C+and+Helge+Langseth%2E+2023%2E+On+reporting+robust+and+trustworthy+conclusions+from+model+comparison+studies+involving+neural+networks+and+rand)
17. Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, Santosh Mashetty, and Chitta Baral. 2023. Instruction tuned models are quick learners. arXiv preprint arXiv:2306.05539. — [arXiv:2306.05539](https://arxiv.org/abs/2306.05539) [find](https://scholar.google.com/scholar?q=Himanshu+Gupta%2C+Saurabh+Arjun+Sawant%2C+Swaroop+Mishra%2C+Mutsumi+Nakamura%2C+Arindam+Mitra%2C+Santosh+Mashetty%2C+and+Chitta+Baral%2E+2023%2E+Instruction+tuned+models+are+quick+learners%2E+arXiv+preprint+arXiv%3A2306%2E)
18. Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, Emily Alsentzer, et al. 2023. Do we still need clinical language models? In Conference on Health, Inference, and Learning, pages 578–597. PMLR. [find](https://scholar.google.com/scholar?q=Evan+Hernandez%2C+Diwakar+Mahajan%2C+Jonas+Wulff%2C+Micah+J+Smith%2C+Zachary+Ziegler%2C+Daniel+Nadler%2C+Peter+Szolovits%2C+Alistair+Johnson%2C+Emily+Alsentzer%2C+et+al%2E+2023%2E+Do+we+still+need+clinical+language+models%3F)
19. SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=SU+Hongjin%2C+Jungo+Kasai%2C+Chen+Henry+Wu%2C+Weijia+Shi%2C+Tianlu+Wang%2C+Jiayi+Xin%2C+Rui+Zhang%2C+Mari+Ostendorf%2C+Luke+Zettlemoyer%2C+Noah+A+Smith%2C+et+al%2E+2022%2E+Selective+annotation+makes+language+models+better+fe)
20. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
21. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
22. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
23. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12186–12215, Bangkok, Thailand and virtual meeting.… [find](https://scholar.google.com/scholar?q=Renren+Jin%2C+Jiangcun+Du%2C+Wuwei+Huang%2C+Wei+Liu%2C+Jian+Luan%2C+Bin+Wang%2C+and+Deyi+Xiong%2E+2024%2E+A+comprehensive+evaluation+of+quantization+strategies+for+large+language+models%2E+In+Findings+of+the+Associatio)
24. Abdullatif Köksal, Timo Schick, and Hinrich Schuetze. 2023. MEAL: Stable and active learning for few-shot prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 506–517, Singapore. ACL. [find](https://scholar.google.com/scholar?q=Abdullatif+K%C3%B6ksal%2C+Timo+Schick%2C+and+Hinrich+Schuetze%2E+2023%2E+MEAL%3A+Stable+and+active+learning+for+few%2Dshot+prompting%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+pages+506)
25. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910.09700.
26. Teven Le Scao and Alexander Rush. 2021. [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. ACL.
27. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015. [DBpedia – a large-scale, multilingual knowledge base extracted from wikipedia](https://md.research.elamur.ai/free/en/papers/lehmann-2015-dbpedia-large-scale-multilingual-knowledge/). Semant. Web, 6(2):167–195.
28. Changmao Li and Jeffrey Flanigan. 2024. Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18471– 18480. [find](https://scholar.google.com/scholar?q=Changmao+Li+and+Jeffrey+Flanigan%2E+2024%2E+Task+contamination%3A+Language+models+may+not+be+few%2Dshot+anymore%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+38%2C+pages+18471%E2%80%93+18480)
29. Xiaonan Li and Xipeng Qiu. 2023. [Finding support examples for in-context learning](https://md.research.elamur.ai/free/en/papers/li-2023-finding-support-examples-context-learning/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6219–6235, Singapore. ACL.
30. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35:1950–1965.
31. Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2024. Do emergent abilities exist in quantized large language models: An empirical study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources… [find](https://scholar.google.com/scholar?q=Peiyu+Liu%2C+Zikang+Liu%2C+Ze%2DFeng+Gao%2C+Dawei+Gao%2C+Wayne+Xin+Zhao%2C+Yaliang+Li%2C+Bolin+Ding%2C+and+Ji%2DRong+Wen%2E+2024%2E+Do+emergent+abilities+exist+in+quantized+large+language+models%3A+An+empirical+study%2E+In+Pro)
32. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Comput. Surv., 55(9).
33. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
34. Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. [Cutting down on prompts and parameters: Simple few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 2824–2835, Dublin, Ireland. AC…
35. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
36. Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. [Large language model is not a good few-shot information extractor, but a good reranker for hard samples!](https://md.research.elamur.ai/free/en/papers/ma-2023-large-language-model-is-not/) In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10572–10601, Singapore. ACL.
37. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Marius+Mosbach%2C+Maksym+Andriushchenko%2C+and+Dietrich+Klakow%2E+2021%2E+On+the+Stability+of+Fine%2Dtuning+BERT%3A+Misconceptions%2C+Explanations%2C+and+Strong+Baselines%2E+In+International+Conference+on+Learning+Repr)
38. Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. [Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation](https://md.research.elamur.ai/free/en/papers/mosbach-2023-few-shot-fine-tuning-vs/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 12284– 12314, Toronto, Canada. ACL.
39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
40. Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024a. On sensitivity of learning with limited labelled data to the effects of randomness: Impact of interactions and systematic choices. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 522–556, Miami,… [find](https://scholar.google.com/scholar?q=Branislav+Pecher%2C+Ivan+Srba%2C+and+Maria+Bielikova%2E+2024a%2E+On+sensitivity+of+learning+with+limited+labelled+data+to+the+effects+of+randomness%3A+Impact+of+interactions+and+systematic+choices%2E+In+Proceedin)
41. Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024b. A survey on stability of learning with limited labelled data and its sensitivity to the effects of randomness. ACM Computing Surveys, 57(1). [find](https://scholar.google.com/scholar?q=Branislav+Pecher%2C+Ivan+Srba%2C+and+Maria+Bielikova%2E+2024b%2E+A+survey+on+stability+of+learning+with+limited+labelled+data+and+its+sensitivity+to+the+effects+of+randomness%2E+ACM+Computing+Surveys%2C+57%281%29%2E)
42. Branislav Pecher, Ivan Srba, Maria Bielikova, and Joaquin Vanschoren. 2024c. [Automatic combination of sample selection strategies for few-shot learning](https://md.research.elamur.ai/free/en/papers/pecher-2026-automatic-combination-sample-selection-strategies/). arXiv preprint arXiv:2402.03038.
43. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is ChatGPT a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1339–1384, Singapore. ACL.
44. Timo Schick and Hinrich Schütze. 2021. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. ACL.
45. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+Language+Models%E2%80%99+Sensitivity+to+Spurious+Features+in+Prompt+Design+or%3A+How+I+learned+to+start+worrying+about+prompt+formatt)
46. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2024%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+i+learned+to+start+worrying+about+prompt+formatt)
47. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–…
48. Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang, Tianwei Zhang, Jiwei Li, Fei Cheng, Lingjuan Lyu, Fei Wu, et al. 2023. [Pushing the limits of chatgpt on nlp tasks](https://md.research.elamur.ai/free/en/papers/sun-2023-pushing-limits-chatgpt-nlp-tasks/). arXiv preprint arXiv:2306.09719.
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
50. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+2023%2E+Zephyr%3A+Direct+distilla)
51. Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’00, page 200–207, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees+and+Dawn+M%2E+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+Proceedings+of+the+23rd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information)
52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in Neural Information Processing Systems, 32.
53. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
54. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
55. Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. [Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning](https://md.research.elamur.ai/free/en/papers/weber-2023-mind-instructions-holistic-evaluation-consistency/). In Proc. of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 294–313, Singapore. Association for Computational Lingui…
56. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
57. Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. [Active example selection for in-context learning](https://md.research.elamur.ai/free/en/papers/zhang-2022-active-example-selection-context-learning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134– 9148, Abu Dhabi, United Arab Emirates. ACL.
58. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International Conference on Machine Learning, pages 12697–12706. PMLR.

*Updated 31 October 2026*