---
title: "Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing"
description: "They propose hybrid inference: a learned router sends each query to either a small or a large model. 42 references, cited by 91 papers here."
authors: ["Dujian Ding", "Ankur Mallick", "Chi Wang", "Robert Sim", "Subhabrata Mukherjee", "Victor Ruhle", "Laks V.S. Lakshmanan", "Ahmed Hassan Awadallah"]
year: 2024
venue: "arXiv:2404.14618"
source: "https://arxiv.org/abs/2404.14618"
doi: "10.48550/arxiv.2404.14618"
arxiv: "2404.14618"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ding-2024-hybrid-llm-cost-efficient-quality/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ding-2024-hybrid-llm-cost-efficient-quality/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

# Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

**Authors:** Dujian Ding, Ankur Mallick, [Chi Wang](https://md.research.elamur.ai/free/en/authors/chi-wang/), [Robert Sim](https://md.research.elamur.ai/free/en/authors/robert-sim/), [Subhabrata Mukherjee](https://md.research.elamur.ai/free/en/authors/subhabrata-mukherjee/), [Victor Ruhle](https://md.research.elamur.ai/free/en/authors/victor-ruhle/), Laks V.S. Lakshmanan and [Ahmed Hassan Awadallah](https://md.research.elamur.ai/free/en/authors/ahmed-hassan-awadallah/) · **Year:** 2024 · **Venue:** arXiv:2404.14618 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.14618) · [PDF](https://arxiv.org/pdf/2404.14618) · [DOI 10.48550/arxiv.2404.14618](https://doi.org/10.48550/arxiv.2404.14618) · [arXiv 2404.14618](https://arxiv.org/abs/2404.14618) · [OpenAlex](https://openalex.org/W4395443209)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/ding-2024-hybrid-llm-cost-efficient-quality/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large LLMs answer well but need costly cloud hardware. Smaller models are cheap enough for edge devices but usually give weaker answers. The authors note that many queries are "easy," and a small model handles them about as well as a big one. They propose hybrid inference: a learned router sends each query to either a small or a large model. The decision is based on the predicted quality gap between the two models' responses. A threshold set at test time shifts the balance between cost and quality without retraining.

The paper presents three router variants. One is deterministic, one accounts for randomness in LLM outputs, and one uses a label transformation for cases where the small model is much weaker. Training labels come from BART score. Experiments use the benchmark of real-world queries from Jiang et al. (2023), with Flan-t5, Llama-2 and GPT-3.5-turbo. The router cuts calls to the large model by up to 40% with no loss in quality. This lets providers serve queries more cheaply and lets users answer easy queries locally.

## Key points
- Each query still triggers exactly one LLM call. This differs from ensembles or cascades, which may call several models per query.
- Llama-2 (13b) matches or beats GPT-3.5-turbo on roughly 20% of queries. Routing between these two sends 22% of queries to the small model, with under 1% drop in BART score.
- The routers are named r_det, r_prob and r_trans. Under GPT-4 evaluation scores (1–10, via LangChain), they save 20% of large-model calls with at most 1% quality loss and 40% with at most 2.1% loss. This holds when BART and GPT-4 quality gaps correlate well (r = 0.46).
- Transfer to unseen model pairs works when quality gaps correlate strongly with the training pair (r > 0.7). In that case the routers save 20% at ≤1.6% loss and 40% at ≤4.1% loss.
- The authors list future work: task-aware routing, routing across N models, out-of-distribution generalization, and evaluation metrics closer to human judgment.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 104 works · 37,402 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. arXiv:2404.14618.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ding2024hybrid,
  title = {{Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing}},
  author = {Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan and Ahmed Hassan Awadallah},
  year = {2024},
  doi = {10.48550/arxiv.2404.14618},
  eprint = {2404.14618},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.14618}
}
```

- [← previous World Model on Million-Length Video And Language With Blockwise RingAttention](https://md.research.elamur.ai/free/en/papers/liu-2024-world-model-million-length-video/)
- [next → LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots : can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, Virtual Event Canada, Mar. 2021. ACM. ISBN 978-1-4503-8309-7. d… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) [find](https://scholar.google.com/scholar?q=E%2E+M%2E+Bender%2C+T%2E+Gebru%2C+A%2E+McMillan%2DMajor%2C+and+S%2E+Shmitchell%2E+On+the+dangers+of+stochastic+parrots+%3A+can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountabili)
2. K. Blagec, G. Dorffner, M. Moradi, S. Ott, and M. Samwald. A global analysis of metrics used for measuring performance in natural language processing. arXiv preprint arXiv:2204.11574, 2022. [find](https://scholar.google.com/scholar?q=K%2E+Blagec%2C+G%2E+Dorffner%2C+M%2E+Moradi%2C+S%2E+Ott%2C+and+M%2E+Samwald%2E+A+global+analysis+of+metrics+used+for+measuring+performance+in+natural+language+processing%2E+arXiv+preprint+arXiv%3A2204%2E11574%2C+2022%2E)
3. (c) Low correlation(r=0.06, ρ=0.02) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258, 2021.
4. H. Chase. LangChain, Oct. 2022. URL [https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain). [↗](https://github.com/langchain-ai/langchain)
5. L. Chen, M. Zaharia, and J. Zou. [Frugalgpt: How to use large language models while reducing cost and improving performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/). arXiv preprint arXiv:2305.05176, 2023.
6. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+S%2E+Longpre%2C+B%2E+Zoph%2C+Y%2E+Tay%2C+W%2E+Fedus%2C+E%2E+Li%2C+X%2E+Wang%2C+M%2E+Dehghani%2C+S%2E+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+2022%2E)
7. D. Ding, S. Amer-Yahia, and L. V. Lakshmanan. On efficient approximate queries over machine learning models. arXiv preprint arXiv:2206.02845, 2022. — [arXiv:2206.02845](https://arxiv.org/abs/2206.02845) [find](https://scholar.google.com/scholar?q=D%2E+Ding%2C+S%2E+Amer%2DYahia%2C+and+L%2E+V%2E+Lakshmanan%2E+On+efficient+approximate+queries+over+machine+learning+models%2E+arXiv+preprint+arXiv%3A2206%2E02845%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E02845%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E0284)
8. T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Elsken%2C+J%2E+H%2E+Metzen%2C+and+F%2E+Hutter%2E+Neural+architecture+search%3A+A+survey%2E+The+Journal+of+Machine+Learning+Research%2C+20%281%29%3A1997%E2%80%932017%2C+2019%2E)
9. H. Face. Hugging face inference api. [https://huggingface.co/inference-api](https://huggingface.co/inference-api). B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993. [↗](https://huggingface.co/inference-api)
10. P. He, X. Liu, J. Gao, and W. Chen. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). arXiv preprint arXiv:2006.03654, 2020.
11. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
12. G. Hinton, O. Vinyals, J. Dean, et al. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531, 2(7), 2015.
13. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). arXiv preprint arXiv:1904.09751, 2019.
14. T. Iqbal and S. Qureshi. The survey: Text generation models in deep learning. Journal of King Saud University-Computer and Information Sciences, 34(6):2515–2528, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Iqbal+and+S%2E+Qureshi%2E+The+survey%3A+Text+generation+models+in+deep+learning%2E+Journal+of+King+Saud+University%2DComputer+and+Information+Sciences%2C+34%286%29%3A2515%E2%80%932528%2C+2022%2E)
15. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018. [find](https://scholar.google.com/scholar?q=B%2E+Jacob%2C+S%2E+Kligys%2C+B%2E+Chen%2C+M%2E+Zhu%2C+M%2E+Tang%2C+A%2E+Howard%2C+H%2E+Adam%2C+and+D%2E+Kalenichenko%2E+Quantization+and+training+of+neural+networks+for+efficient+integer%2Darithmetic%2Donly+inference%2E+In+Proceedings+of+)
16. D. Jiang, X. Ren, and B. Y. Lin. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). arXiv preprint arXiv:2306.02561, 2023.
17. A. Kag, I. Fedorov, A. Gangrade, P. Whatmough, and V. Saligrama. Efficient edge inference by selective query. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Kag%2C+I%2E+Fedorov%2C+A%2E+Gangrade%2C+P%2E+Whatmough%2C+and+V%2E+Saligrama%2E+Efficient+edge+inference+by+selective+query%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2022%2E)
18. D. Kang, T. Hashimoto, I. Stoica, and Y. Sun. Scaling up trustless dnn inference with zero-knowledge proofs. arXiv preprint arXiv:2210.08674, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Kang%2C+T%2E+Hashimoto%2C+I%2E+Stoica%2C+and+Y%2E+Sun%2E+Scaling+up+trustless+dnn+inference+with+zero%2Dknowledge+proofs%2E+arXiv+preprint+arXiv%3A2210%2E08674%2C+2022%2E)
19. S. Kim, K. Mangalam, S. Moon, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer. Speculative decoding with big little decoder. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Kim%2C+K%2E+Mangalam%2C+S%2E+Moon%2C+J%2E+Malik%2C+M%2E+W%2E+Mahoney%2C+A%2E+Gholami%2C+and+K%2E+Keutzer%2E+Speculative+decoding+with+big+little+decoder%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2C+)
20. B. Krawczyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4):221–232, 2016. [find](https://scholar.google.com/scholar?q=B%2E+Krawczyk%2E+Learning+from+imbalanced+data%3A+open+challenges+and+future+directions%2E+Progress+in+Artificial+Intelligence%2C+5%284%29%3A221%E2%80%93232%2C+2016%2E)
21. Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+J%2E+Denker%2C+and+S%2E+Solla%2E+Optimal+brain+damage%2E+Advances+in+neural+information+processing+systems%2C+2%2C+1989%2E)
22. Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Leviathan%2C+M%2E+Kalman%2C+and+Y%2E+Matias%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pages+19274%E2%80%9319286%2E+PMLR%2C+2023%2E)
23. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/), may 2023. arXiv preprint arXiv:2303.16634, 2023.
24. M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513–5533, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Masana%2C+X%2E+Liu%2C+B%2E+Twardowski%2C+M%2E+Menta%2C+A%2E+D%2E+Bagdanov%2C+and+J%2E+Van+De+Weijer%2E+Class%2Dincremental+learning%3A+survey+and+performance+evaluation+on+image+classification%2E+IEEE+Transactions+on+Pattern+An)
25. G. Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1–37, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Menghani%2E+Efficient+deep+learning%3A+A+survey+on+making+deep+learning+models+smaller%2C+faster%2C+and+better%2E+ACM+Computing+Surveys%2C+55%2812%29%3A1%E2%80%9337%2C+2023%2E)
26. OpenAI. Openai gpt-3.5-turbo. — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5), a. OpenAI. Chatgpt. https://chat.openai.com/, b. OpenAI. Openai platform. https://platform.openai.com/overview, c. OpenAI. Gpt-4 technical report, 2023. [↗](https://platform.openai.com/docs/models/gpt-3-5)
27. U. Ruby and V. Yendapalli. Binary cross entropy with deep learning technique for image classification. Int. J. Adv. Trends Comput. Sci. Eng, 9(10), 2020. [find](https://scholar.google.com/scholar?q=U%2E+Ruby+and+V%2E+Yendapalli%2E+Binary+cross+entropy+with+deep+learning+technique+for+image+classification%2E+Int%2E+J%2E+Adv%2E+Trends+Comput%2E+Sci%2E+Eng%2C+9%2810%29%2C+2020%2E)
28. L. Skyward. Gpt-4 is non-deterministic and moe is the likely reason why, Aug 2023. URL [https://ai.plainenglish.io/gpt-4-is-non-deterministic-af202847529c](https://ai.plainenglish.io/gpt-4-is-non-deterministic-af202847529c). [↗](https://ai.plainenglish.io/gpt-4-is-non-deterministic-af202847529c)
29. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615, 2022.
30. Z. Sun, Z. Li, H. Wang, D. He, Z. Lin, and Z. Deng. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Sun%2C+Z%2E+Li%2C+H%2E+Wang%2C+D%2E+He%2C+Z%2E+Lin%2C+and+Z%2E+Deng%2E+Fast+structured+decoding+for+sequence+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
31. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
32. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
33. M. Treviso, J.-U. Lee, T. Ji, B. v. Aken, Q. Cao, M. R. Ciosici, M. Hassid, K. Heafield, S. Hooker, C. Raffel, et al. [Efficient methods for natural language processing: A survey](https://md.research.elamur.ai/free/en/papers/treviso-2023-efficient-methods-natural-language-processing/). Transactions of the Association for Computational Linguistics, 11:826–860, 2023.
34. G. Urban, K. J. Geras, S. E. Kahou, O. Aslan, S. Wang, R. Caruana, A. Mohamed, M. Philipose, and M. Richardson. Do deep convolutional nets really need to be deep and convolutional? arXiv preprint arXiv:1603.05691, 2016. — [arXiv:1603.05691](https://arxiv.org/abs/1603.05691) [find](https://scholar.google.com/scholar?q=G%2E+Urban%2C+K%2E+J%2E+Geras%2C+S%2E+E%2E+Kahou%2C+O%2E+Aslan%2C+S%2E+Wang%2C+R%2E+Caruana%2C+A%2E+Mohamed%2C+M%2E+Philipose%2C+and+M%2E+Richardson%2E+Do+deep+convolutional+nets+really+need+to+be+deep+and+convolutional%3F+arXiv+preprint+arXi)
35. V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks on cpus. 2011. [find](https://scholar.google.com/scholar?q=V%2E+Vanhoucke%2C+A%2E+Senior%2C+and+M%2E+Z%2E+Mao%2E+Improving+the+speed+of+neural+networks+on+cpus%2E+2011%2E)
36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
37. R. Yao, G. Lin, S. Xia, J. Zhao, and Y. Zhou. Video object segmentation and tracking: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(4):1–47, 2020. [find](https://scholar.google.com/scholar?q=R%2E+Yao%2C+G%2E+Lin%2C+S%2E+Xia%2C+J%2E+Zhao%2C+and+Y%2E+Zhou%2E+Video+object+segmentation+and+tracking%3A+A+survey%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology+%28TIST%29%2C+11%284%29%3A1%E2%80%9347%2C+2020%2E)
38. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022. — [Orca: A Distributed Serving System for Transformer-Based Generative Models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/)
39. W. Yuan, G. Neubig, and P. Liu. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
40. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 2023.
41. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
42. B. Zoph and Q. V. Le. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). arXiv preprint arXiv:1611.01578, 2016.

*Updated 2 November 2026*