---
title: "Large Language Model Routing with Benchmark Datasets (2023)"
description: "To rank models on a new task, the paper proposes three scores. 47 references, cited by 52 papers here."
authors: ["Tal Shnitzer", "Anthony Ou", "Mirian Silva", "Kate Soule", "Yuekai Sun", "Justin Solomon", "Neil Thompson", "Mikhail Yurochkin"]
year: 2023
venue: "arXiv:2309.15789"
source: "https://arxiv.org/abs/2309.15789"
doi: "10.48550/arxiv.2309.15789"
arxiv: "2309.15789"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shnitzer-2023-large-language-model-routing-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shnitzer-2023-large-language-model-routing-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Large Language Model Routing with Benchmark Datasets

# Large Language Model Routing with Benchmark Datasets

**Authors:** Tal Shnitzer, Anthony Ou, Mirian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson and Mikhail Yurochkin · **Year:** 2023 · **Venue:** arXiv:2309.15789 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.15789) · [PDF](https://arxiv.org/pdf/2309.15789) · [DOI 10.48550/arxiv.2309.15789](https://doi.org/10.48550/arxiv.2309.15789) · [arXiv 2309.15789](https://arxiv.org/abs/2309.15789) · [OpenAlex](https://openalex.org/W4387163503)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · 52 in our library, of the 141,840 papers whose bibliographies we read

## Summary
No single open-source LLM wins on every task, so practitioners need a way to pick a good model for their own new task without testing every candidate. The authors treat this as a routing problem and train the router on per-sample results that benchmarks already produce. For each candidate LLM, a binary classifier predicts whether that model will answer a given input "correctly." Correctness is the metric itself for classification and QA, and a thresholded metric for generation tasks. The classifier is a k-nearest neighbors model over sentence-transformer embeddings of the inputs.

To rank models on a new task, the paper proposes three scores. One of them corrects for the likely weaker accuracy of these predictors on out-of-distribution data. The authors also link the approach to meta-learning.

Experiments use HELM and MixInstruct. Routing consistently beats always using the single best model. At test time it is cheap, because only the chosen LLM runs. The main remaining obstacle is generalizing to tasks far from the benchmark data.

## Key points
- The evaluation covers 29 HELM datasets (QA, text classification, knowledge, reasoning) plus MixInstruct, which tests instruction following.
- Unlike rankers such as the one in Jiang et al. (2023), or FrugalGPT's sequential calls, the router uses only the input. It never needs outputs from every candidate LLM.
- Routing among models of at most 13B parameters can match Llama 2 70B on HELM. This requires scores S1/S2 plus a few labeled target samples (α = 0.04, about 2–40 examples).
- Routing quality, measured as the Pearson correlation between S3 scores and true accuracies, nears 1 when a new task is close to the benchmark data. It drops for distant tasks.
- On MixInstruct, selections get closer to the oracle as test points get closer to the reference set. This suggests that adding more benchmarks should improve routing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations
- [MIT Computer Science and Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/mit-computer-science-and-artificial-intelligence-laboratory/) — Research institute, Cambridge, United States · 54 works · 29,441 citations
- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 36 works · 12,181 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Tal Shnitzer, Anthony Ou, Mirian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson and Mikhail Yurochkin. 2023. Large Language Model Routing with Benchmark Datasets. arXiv:2309.15789.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shnitzer2023large,
  title = {{Large Language Model Routing with Benchmark Datasets}},
  author = {Tal Shnitzer, Anthony Ou, Mirian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson and Mikhail Yurochkin},
  year = {2023},
  doi = {10.48550/arxiv.2309.15789},
  eprint = {2309.15789},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.15789}
}
```

- [← previous FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models](https://md.research.elamur.ai/free/en/papers/dobler-2023-focus-effective-embedding-initialization-monolingual/)
- [next → TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

47 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. (2019). [Invariant risk minimization](https://md.research.elamur.ai/free/en/papers/arjovsky-2019-invariant-risk-minimization/). arXiv preprint arXiv:1907.02893.
2. Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. (2023). Open llm leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). [↗](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
3. Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning, volume 4. Springer. [find](https://scholar.google.com/scholar?q=Bishop%2C+C%2E+M%2E+and+Nasrabadi%2C+N%2E+M%2E+%282006%29%2E+Pattern+recognition+and+machine+learning%2C+volume+4%2E+Springer%2E)
4. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. (2014). [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the ninth workshop on statistical machine translation, pages 12–58.
5. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021). [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258.
6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
7. Chen, J., Liu, F., Avci, B., Wu, X., Liang, Y., and Jha, S. (2021a). Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34:14980– 14992. [find](https://scholar.google.com/scholar?q=Chen%2C+J%2E%2C+Liu%2C+F%2E%2C+Avci%2C+B%2E%2C+Wu%2C+X%2E%2C+Liang%2C+Y%2E%2C+and+Jha%2C+S%2E+%282021a%29%2E+Detecting+errors+and+estimating+accuracy+on+unlabeled+data+with+self%2Dtraining+ensembles%2E+Advances+in+Neural+Information+Processing+)
8. Chen, L., Zaharia, M., and Zou, J. (2023). [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/). arXiv preprint arXiv:2305.05176.
9. Chen, L., Zaharia, M., and Zou, J. Y. (2020). Frugalml: How to use ml prediction apis more accurately and cheaply. Advances in neural information processing systems, 33:10685–10696. [find](https://scholar.google.com/scholar?q=Chen%2C+L%2E%2C+Zaharia%2C+M%2E%2C+and+Zou%2C+J%2E+Y%2E+%282020%29%2E+Frugalml%3A+How+to+use+ml+prediction+apis+more+accurately+and+cheaply%2E+Advances+in+neural+information+processing+systems%2C+33%3A10685%E2%80%9310696%2E)
10. Chen, M., Goel, K., Sohoni, N. S., Poms, F., Fatahalian, K., and Ré, C. (2021b). Mandoline: Model evaluation under distribution shift. In International conference on machine learning, pages 1617–1629. PMLR. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Goel%2C+K%2E%2C+Sohoni%2C+N%2E+S%2E%2C+Poms%2C+F%2E%2C+Fatahalian%2C+K%2E%2C+and+Re%CC%81%2C+C%2E+%282021b%29%2E+Mandoline%3A+Model+evaluation+under+distribution+shift%2E+In+International+conference+on+machine+learning%2C+pages+1617%E2%80%931629)
11. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. (2023). Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%2C+Li%2C+Z%2E%2C+Lin%2C+Z%2E%2C+Sheng%2C+Y%2E%2C+Wu%2C+Z%2E%2C+Zhang%2C+H%2E%2C+Zheng%2C+L%2E%2C+Zhuang%2C+S%2E%2C+Zhuang%2C+Y%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+Stoica%2C+I%2E%2C+and+Xing%2C+E%2E+P%2E+%282023%29%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+wi)
12. Cover, T. and Hart, P. (1967). Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21–27. [find](https://scholar.google.com/scholar?q=Cover%2C+T%2E+and+Hart%2C+P%2E+%281967%29%2E+Nearest+neighbor+pattern+classification%2E+IEEE+transactions+on+information+theory%2C+13%281%29%3A21%E2%80%9327%2E)
13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
14. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2021). A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Tang%2C+E%2E%2C+Thite%2C+A%2E%2C+Wang%2C+B%2E%2C+Wang%2C+K%2E%2C+and+Zou%2C+A%2E+%282)
15. Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. (2022). Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Garg%2C+S%2E%2C+Balakrishnan%2C+S%2E%2C+Lipton%2C+Z%2E+C%2E%2C+Neyshabur%2C+B%2E%2C+and+Sedghi%2C+H%2E+%282022%29%2E+Leveraging+unlabeled+data+to+predict+out%2Dof%2Ddistribution+performance%2E+In+International+Conference+on+Learning+Represent)
16. Goyal, P., Mahajan, D., Gupta, A., and Misra, I. (2019). Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the ieee/cvf International Conference on computer vision, pages 6391–6400. [find](https://scholar.google.com/scholar?q=Goyal%2C+P%2E%2C+Mahajan%2C+D%2E%2C+Gupta%2C+A%2E%2C+and+Misra%2C+I%2E+%282019%29%2E+Scaling+and+benchmarking+self%2Dsupervised+visual+representation+learning%2E+In+Proceedings+of+the+ieee%2Fcvf+International+Conference+on+computer+vi)
17. Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. (2021). Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1134–1144. [find](https://scholar.google.com/scholar?q=Guillory%2C+D%2E%2C+Shankar%2C+V%2E%2C+Ebrahimi%2C+S%2E%2C+Darrell%2C+T%2E%2C+and+Schmidt%2C+L%2E+%282021%29%2E+Predicting+with+confidence+on+unseen+distributions%2E+In+Proceedings+of+the+IEEE%2FCVF+international+conference+on+computer+vi)
18. Gulrajani, I. and Lopez-Paz, D. (2021). In search of lost domain generalization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gulrajani%2C+I%2E+and+Lopez%2DPaz%2C+D%2E+%282021%29%2E+In+search+of+lost+domain+generalization%2E+In+International+Conference+on+Learning+Representations%2E)
19. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Pleiss%2C+G%2E%2C+Sun%2C+Y%2E%2C+and+Weinberger%2C+K%2E+Q%2E+%282017%29%2E+On+calibration+of+modern+neural+networks%2E+In+International+conference+on+machine+learning%2C+pages+1321%E2%80%931330%2E+PMLR%2E)
20. Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer. [find](https://scholar.google.com/scholar?q=Hastie%2C+T%2E%2C+Tibshirani%2C+R%2E%2C+Friedman%2C+J%2E+H%2E%2C+and+Friedman%2C+J%2E+H%2E+%282009%29%2E+The+elements+of+statistical+learning%3A+data+mining%2C+inference%2C+and+prediction%2C+volume+2%2E+Springer%2E)
21. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
22. Jiang, D., Ren, X., and Lin, B. Y. (2023). [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada. Association fo…
23. Jiang, Y., Nagarajan, V., Baek, C., and Kolter, J. Z. (2021). Assessing Generalization of SGD via Disagreement. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiang%2C+Y%2E%2C+Nagarajan%2C+V%2E%2C+Baek%2C+C%2E%2C+and+Kolter%2C+J%2E+Z%2E+%282021%29%2E+Assessing+Generalization+of+SGD+via+Disagreement%2E+In+International+Conference+on+Learning+Representations%2E)
24. Kelly, M., Longjohn, R., and Nottingham, K. (2023). The UCI Machine Learning Repository. [find](https://scholar.google.com/scholar?q=Kelly%2C+M%2E%2C+Longjohn%2C+R%2E%2C+and+Nottingham%2C+K%2E+%282023%29%2E+The+UCI+Machine+Learning+Repository%2E)
25. Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2021). Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637–5664. PMLR. [find](https://scholar.google.com/scholar?q=Koh%2C+P%2E+W%2E%2C+Sagawa%2C+S%2E%2C+Marklund%2C+H%2E%2C+Xie%2C+S%2E+M%2E%2C+Zhang%2C+M%2E%2C+Balsubramani%2C+A%2E%2C+Hu%2C+W%2E%2C+Yasunaga%2C+M%2E%2C+Phillips%2C+R%2E+L%2E%2C+Gao%2C+I%2E%2C+et+al%2E+%282021%29%2E+Wilds%3A+A+benchmark+of+in%2Dthe%2Dwild+distribution+shifts%2E+In+)
26. LAION-AI (2023). Open assistant. [find](https://scholar.google.com/scholar?q=LAION%2DAI+%282023%29%2E+Open+assistant%2E)
27. Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y. J., et al. (2022). Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35:9287–9301. [find](https://scholar.google.com/scholar?q=Li%2C+C%2E%2C+Liu%2C+H%2E%2C+Li%2C+L%2E%2C+Zhang%2C+P%2E%2C+Aneja%2C+J%2E%2C+Yang%2C+J%2E%2C+Jin%2C+P%2E%2C+Hu%2C+H%2E%2C+Liu%2C+Z%2E%2C+Lee%2C+Y%2E+J%2E%2C+et+al%2E+%282022%29%2E+Elevater%3A+A+benchmark+and+toolkit+for+evaluating+language%2Daugmented+visual+models%2E+Advance)
28. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022). Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E%2C+Bommasani%2C+R%2E%2C+Lee%2C+T%2E%2C+Tsipras%2C+D%2E%2C+Soylu%2C+D%2E%2C+Yasunaga%2C+M%2E%2C+Zhang%2C+Y%2E%2C+Narayanan%2C+D%2E%2C+Wu%2C+Y%2E%2C+Kumar%2C+A%2E%2C+et+al%2E+%282022%29%2E+Holistic+evaluation+of+language+models%2E+arXiv+preprint+arXiv%3A2211%2E0)
29. Liu, Y. and Liu, P. (2021). [Simcls: A simple framework for contrastive learning of abstractive summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-simcls-simple-framework-contrastive-learning/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)…
30. Maity, S., Yurochkin, M., Banerjee, M., and Sun, Y. (2023). Understanding new tasks through the lens of training data via exponential tilting. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Maity%2C+S%2E%2C+Yurochkin%2C+M%2E%2C+Banerjee%2C+M%2E%2C+and+Sun%2C+Y%2E+%282023%29%2E+Understanding+new+tasks+through+the+lens+of+training+data+via+exponential+tilting%2E+In+International+Conference+on+Learning+Representations%2E)
31. Ng, N. H., Hulkund, N., Cho, K., and Ghassemi, M. (2023). Predicting out-of-domain generalization with neighborhood invariance. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Ng%2C+N%2E+H%2E%2C+Hulkund%2C+N%2E%2C+Cho%2C+K%2E%2C+and+Ghassemi%2C+M%2E+%282023%29%2E+Predicting+out%2Dof%2Ddomain+generalization+with+neighborhood+invariance%2E+Transactions+on+Machine+Learning+Research%2E)
32. Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. (2019). Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=Ovadia%2C+Y%2E%2C+Fertig%2C+E%2E%2C+Ren%2C+J%2E%2C+Nado%2C+Z%2E%2C+Sculley%2C+D%2E%2C+Nowozin%2C+S%2E%2C+Dillon%2C+J%2E%2C+Lakshminarayanan%2C+B%2E%2C+and+Snoek%2C+J%2E+%282019%29%2E+Can+you+trust+your+model%E2%80%99s+uncertainty%3F+evaluating+predictive+uncertainty+u)
33. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%2C+et+al%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
34. Raschka, S. (2018). Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808. — [arXiv:1811.12808](https://arxiv.org/abs/1811.12808) [find](https://scholar.google.com/scholar?q=Raschka%2C+S%2E+%282018%29%2E+Model+evaluation%2C+model+selection%2C+and+algorithm+selection+in+machine+learning%2E+arXiv+preprint+arXiv%3A1811%2E12808%2E+%E2%80%94+%5BarXiv%3A1811%2E12808%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E12808%29)
35. Ravaut, M., Joty, S., and Chen, N. (2022). Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4504–4524. [find](https://scholar.google.com/scholar?q=Ravaut%2C+M%2E%2C+Joty%2C+S%2E%2C+and+Chen%2C+N%2E+%282022%29%2E+Summareranker%3A+A+multi%2Dtask+mixture%2Dof%2Dexperts+re%2Dranking+framework+for+abstractive+summarization%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Associati)
36. Reimers, N. and Gurevych, I. (2019). [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
37. Salazar, J., Liang, D., Nguyen, T. Q., and Kirchhoff, K. (2020). [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712.
38. Sellam, T., Das, D., and Parikh, A. (2020). BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892. [find](https://scholar.google.com/scholar?q=Sellam%2C+T%2E%2C+Das%2C+D%2E%2C+and+Parikh%2C+A%2E+%282020%29%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+7881%E2%80%93)
39. Settles, B. (2009). [Active learning literature survey](https://md.research.elamur.ai/free/en/papers/settles-2009-active-learning-literature-survey/).
40. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
41. Sun, B. and Saenko, K. (2016). Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443–450. Springer. [find](https://scholar.google.com/scholar?q=Sun%2C+B%2E+and+Saenko%2C+K%2E+%282016%29%2E+Deep+coral%3A+Correlation+alignment+for+deep+domain+adaptation%2E+In+Computer+Vision%E2%80%93ECCV+2016+Workshops%3A+Amsterdam%2C+The+Netherlands%2C+October+8%2D10+and+15%2D16%2C+2016%2C+Proceedin)
42. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2019). [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32.
43. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
44. Xu, H. and Tibshirani, R. (2022). Estimation of prediction error with known covariate shift. arXiv preprint arXiv:2205.01849. — [arXiv:2205.01849](https://arxiv.org/abs/2205.01849) [find](https://scholar.google.com/scholar?q=Xu%2C+H%2E+and+Tibshirani%2C+R%2E+%282022%29%2E+Estimation+of+prediction+error+with+known+covariate+shift%2E+arXiv+preprint+arXiv%3A2205%2E01849%2E+%E2%80%94+%5BarXiv%3A2205%2E01849%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E01849%29)
45. Yu, Y., Yang, Z., Wei, A., Ma, Y., and Steinhardt, J. (2022). Predicting out-of-distribution error with the projection norm. In International Conference on Machine Learning, pages 25721–25746. PMLR. [find](https://scholar.google.com/scholar?q=Yu%2C+Y%2E%2C+Yang%2C+Z%2E%2C+Wei%2C+A%2E%2C+Ma%2C+Y%2E%2C+and+Steinhardt%2C+J%2E+%282022%29%2E+Predicting+out%2Dof%2Ddistribution+error+with+the+projection+norm%2E+In+International+Conference+on+Machine+Learning%2C+pages+25721%E2%80%9325746%2E+PMLR%2E)
46. Yuan, W., Neubig, G., and Liu, P. (2021). [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277.
47. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhang%2C+T%2E%2C+Kishore%2C+V%2E%2C+Wu%2C+F%2E%2C+Weinberger%2C+K%2E+Q%2E%2C+and+Artzi%2C+Y%2E+%282020%29%2E+BERTScore%3A+Evaluating+Text+Generation+with+BERT%2E+In+International+Conference+on+Learning+Representations%2E)

*Updated 24 October 2026*