---
title: "Orca — paper by Yu et al. (2022) — summary and ideas"
description: "Serving systems such as Triton paired with engines like FasterTransformer batch whole requests. 67 references, cited by 478 papers here."
authors: ["Gyeong-In Yu", "Joo Seong Jeong", "Geon-Woo Kim", "Soojeong Kim", "Byung-Gon Chun"]
year: 2022
venue: "OSDI 2022"
source: "https://www.usenix.org/conference/osdi22/presentation/yu"
licence: "author copy (usenix.org open access) — read only"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Orca: A Distributed Serving System for Transformer-Based Generative Models

# Orca: A Distributed Serving System for Transformer-Based Generative Models

**Authors:** Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim and Byung-Gon Chun · **Year:** 2022 · **Venue:** OSDI 2022 · **Licence:** author copy (usenix.org open access) — read only

**Links:** [Publisher's page](https://www.usenix.org/conference/osdi22/presentation/yu) · [PDF](https://www.usenix.org/system/files/osdi22-yu.pdf)

**Citations:** [478](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Serving systems such as Triton paired with engines like FasterTransformer batch whole requests. Autoregressive models such as GPT-3, however, produce one token per model run, and requests need different numbers of runs. Under request-level batching, a request that finishes early stays stuck until the slowest request in its batch completes, and new arrivals queue until the batch is done.

Orca addresses this by scheduling one iteration (a single pass through all layers) at a time. After each pass, finished requests leave immediately and new ones can join. Requests in such a batch sit at different token positions, which makes Attention inputs irregular in shape. Orca handles this with selective batching: Attention runs per request, while all other operations are batched.

The system adds intra-layer and inter-layer model parallelism, plus a scheduler that accounts for memory and pipelining across workers. On a GPT-3 175B model, Orca reaches 36.9× the throughput of FasterTransformer at equal latency. This matters for anyone deploying large generative Transformers cost-effectively.

## Key points
- **Iteration-level scheduling.** The engine is invoked for one iteration at a time instead of a full request.
  - A newly arrived request waits at most one model pass before it can be considered.
  - Completed requests are returned right away.
- **Selective batching.** Only the Attention operation is split per request; linear layers, LayerNorm, GeLU and similar operations are batched.
  - The reasoning is that Attention has no model parameters, so batching it saves no parameter memory reads.
- **Scale.** Orca borrows intra-layer and inter-layer parallelism from Transformer training systems.
  - It was evaluated on GPT-3 configurations up to 341B parameters.
- **Main result.** Against FasterTransformer on GPT-3 175B, Orca delivers a 36.9× throughput gain at the same latency level.
  - Orca still wins on traces where all requests have equal length, except at max batch size 1, where it reduces to a plain pipeline.
- **Comparison with BatchMaker.** The earlier system BatchMaker batches identical RNN cells.
  - It fails for Transformers because every token position uses a different set of keys and values.
  - In the experiments, sequence lengths ranged from 33 to 640 tokens, so matching cells are rare.

## Interesting ideas
- **Schedule at the natural unit of repetition.** For any multi-step workload, scheduling at the granularity of one step rather than one job removes head-of-line blocking. This could apply to iterative decoding in machine translation, speech recognition, or diffusion-style generators.
- **Batch only what benefits from batching.** Batch the parameter-heavy operations and run irregular, parameter-free operations individually. This turns a hard "everything must have the same shape" constraint into a mostly batched pipeline with little efficiency loss.
- **Maximize work per parameter read.** For large models, loading weights from GPU memory dominates runtime. Processing every ready token in one weight pass is therefore a useful guiding metric when optimizing inference code.
- **Couple scheduler and engine tightly when the standard interface is too coarse.** Orca integrates the scheduler with the execution engine instead of keeping the usual separation. This is a reminder that abstraction boundaries, such as a serving layer kept apart from the engine, can hide large performance wins.
- **Test with uniform-length traces as well as varied ones.** Evaluating on workloads where all requests have equal length isolates how much of the gain comes from early-leaving requests versus other factors. This kind of ablation design is worth copying in other systems papers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2022orca,
  title = {{Orca: A Distributed Serving System for Transformer-Based Generative Models}},
  author = {Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim and Byung-Gon Chun},
  booktitle = {OSDI 2022},
  year = {2022},
  url = {https://www.usenix.org/conference/osdi22/presentation/yu}
}
```

- [← previous Flow Matching for Generative Modeling](https://md.research.elamur.ai/free/en/papers/lipman-2022-flow-matching-generative-modeling/)
- [next → In-context Learning and Induction Heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

67 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. DeepSpeed. Retrieved Dec 13, 2021 from [https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed). [↗](https://github.com/microsoft/DeepSpeed)
2. gRPC. Retrieved Dec 13, 2021 from [https://grpc.io](https://grpc.io). [↗](https://grpc.io)
3. Megatron-LM. Retrieved Dec 13, 2021 from [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM). [↗](https://github.com/NVIDIA/Megatron-LM)
4. NVIDIA FasterTransformer. Retrieved Dec 13, 2021 from [https://github.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer). [↗](https://github.com/NVIDIA/FasterTransformer)
5. NVIDIA NCCL. Retrieved Dec 13, 2021 from [https://github.com/NVIDIA/nccl](https://github.com/NVIDIA/nccl). [↗](https://github.com/NVIDIA/nccl)
6. NVIDIA TensorRT. Retrieved Dec 13, 2021 from [https://developer.nvidia.com/tensorrt](https://developer.nvidia.com/tensorrt). [↗](https://developer.nvidia.com/tensorrt)
7. NVIDIA Triton Inference Server. Retrieved Dec 13, 2021 from [https://developer.nvidia.com/nvidia-triton-inference-server](https://developer.nvidia.com/nvidia-triton-inference-server). [↗](https://developer.nvidia.com/nvidia-triton-inference-server)
8. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yua… — [TensorFlow: A system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/)
9. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). arXiv preprint arXiv:2001.09977, 2020.
10. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettle… — [Efficient Large Scale Language Modeling with Mixtures of Experts](https://md.research.elamur.ai/free/en/papers/artetxe-2021-efficient-large-scale-language-modeling/)
11. Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. [A Statistical Approach to Machine Translation](https://md.research.elamur.ai/free/en/papers/brown-1990-statistical-approach-machine-translation/). Computational Linguistics, 16(2):79–85, 1990.
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
13. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
14. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/). In Proceedings of the 13th USENIX Conference on Operating…
15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
16. Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A Low-Latency Online Prediction Serving System. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation, pages 613–627, 2017. [find](https://scholar.google.com/scholar?q=Daniel+Crankshaw%2C+Xin+Wang%2C+Giulio+Zhou%2C+Michael+J%2E+Franklin%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Clipper%3A+A+Low%2DLatency+Online+Prediction+Serving+System%2E+In+Proceedings+of+the+14th+USENIX+Symposium+o)
17. Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. A Survey of Multilingual Neural Machine Translation. ACM Computing Surveys, 53(5), 2020. [find](https://scholar.google.com/scholar?q=Raj+Dabre%2C+Chenhui+Chu%2C+and+Anoop+Kunchukuttan%2E+A+Survey+of+Multilingual+Neural+Machine+Translation%2E+ACM+Computing+Surveys%2C+53%285%29%2C+2020%2E)
18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
19. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. [CogView: Mastering Text-to-Image Generation via Transformers](https://md.research.elamur.ai/free/en/papers/ding-2021-cogview-mastering-text-image/). Advances in Neural Information Processing Systems, 2021.
20. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73, 2018. [find](https://scholar.google.com/scholar?q=Lucas+Dixon%2C+John+Li%2C+Jeffrey+Sorensen%2C+Nithum+Thain%2C+and+Lucy+Vasserman%2E+Measuring+and+Mitigating+Unintended+Bias+in+Text+Classification%2E+In+Proceedings+of+the+2018+AAAI%2FACM+Conference+on+AI%2C+Ethics%2C)
21. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Di… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
22. Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. TurboTransformers: An Efficient GPU Serving System for Transformer Models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 389–402, 2021. [find](https://scholar.google.com/scholar?q=Jiarui+Fang%2C+Yang+Yu%2C+Chengduo+Zhao%2C+and+Jie+Zhou%2E+TurboTransformers%3A+An+Efficient+GPU+Serving+System+for+Transformer+Models%2E+In+Proceedings+of+the+26th+ACM+SIGPLAN+Symposium+on+Principles+and+Practic)
23. Pin Gao, Lingfan Yu, Yongwei Wu, and Jinyang Li. Low Latency RNN Inference with Cellular Batching. In Proceedings of the Thirteenth EuroSys Conference, 2018. [find](https://scholar.google.com/scholar?q=Pin+Gao%2C+Lingfan+Yu%2C+Yongwei+Wu%2C+and+Jinyang+Li%2E+Low+Latency+RNN+Inference+with+Cellular+Batching%2E+In+Proceedings+of+the+Thirteenth+EuroSys+Conference%2C+2018%2E)
24. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep Residual Learning for Image Recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
25. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
26. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
27. Kevin Hsieh, Ganesh Ananthanarayanan, Peter Bodik, Shivaram Venkataraman, Paramvir Bahl, Matthai Philipose, Phillip B. Gibbons, and Onur Mutlu. Focus: Querying Large Video Datasets with Low Latency and Low Cost. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementat… [find](https://scholar.google.com/scholar?q=Kevin+Hsieh%2C+Ganesh+Ananthanarayanan%2C+Peter+Bodik%2C+Shivaram+Venkataraman%2C+Paramvir+Bahl%2C+Matthai+Philipose%2C+Phillip+B%2E+Gibbons%2C+and+Onur+Mutlu%2E+Focus%3A+Querying+Large+Video+Datasets+with+Low+Latency+an)
28. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dehao+Chen%2C+Mia+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V+Le%2C+Yonghui+Wu%2C+et+al%2E+GPipe%3A+Efficient+Training+of+Giant+Neural+Networks+Using+Pipel)
29. Angela H. Jiang, Daniel L.-K. Wong, Christopher Canel, Lilia Tang, Ishan Misra, Michael Kaminsky, Michael A. Kozuch, Padmanabhan Pillai, David G. Andersen, and Gregory R. Ganger. Mainstream: Dynamic Stem-Sharing for Multi-Tenant Video Processing. In Proceedings of the 2018 USENIX Annual Technical Co… [find](https://scholar.google.com/scholar?q=Angela+H%2E+Jiang%2C+Daniel+L%2E%2DK%2E+Wong%2C+Christopher+Canel%2C+Lilia+Tang%2C+Ishan+Misra%2C+Michael+Kaminsky%2C+Michael+A%2E+Kozuch%2C+Padmanabhan+Pillai%2C+David+G%2E+Andersen%2C+and+Gregory+R%2E+Ganger%2E+Mainstream%3A+Dynamic+S)
30. Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, and Matei Zaharia. NoScope: Optimizing Neural Network Queries over Video at Scale. Proceedings of the VLDB Endowment, 10(11):1586–1597, 2017. [find](https://scholar.google.com/scholar?q=Daniel+Kang%2C+John+Emmons%2C+Firas+Abuzaid%2C+Peter+Bailis%2C+and+Matei+Zaharia%2E+NoScope%3A+Optimizing+Neural+Network+Queries+over+Video+at+Scale%2E+Proceedings+of+the+VLDB+Endowment%2C+10%2811%29%3A1586%E2%80%931597%2C+2017%2E)
31. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling Laws for Neural Language Models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
32. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, 2020.
33. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. [Natural Questions: a Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:4…
34. Woosuk Kwon, Gyeong-In Yu, Eunji Jeong, and Byung-Gon Chun. Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning. Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Gyeong%2DIn+Yu%2C+Eunji+Jeong%2C+and+Byung%2DGon+Chun%2E+Nimble%3A+Lightweight+and+Parallel+GPU+Task+Scheduling+for+Deep+Learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
35. Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation, pages 611–626,… [find](https://scholar.google.com/scholar?q=Yunseong+Lee%2C+Alberto+Scolari%2C+Byung%2DGon+Chun%2C+Marco+Domenico+Santambrogio%2C+Markus+Weimer%2C+and+Matteo+Interlandi%2E+PRETZEL%3A+Opening+the+Black+Box+of+Machine+Learning+Prediction+Serving+Systems%2E+In+Proc)
36. Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Bai Liu, Changjie Fan, Xiaoxi Mao, and Zeng Zhao. Easy and Efficient Transformer: Scalable Inference Solution For large NLP model. arXiv preprint arXiv:2104.12470, 2021. [find](https://scholar.google.com/scholar?q=Gongzheng+Li%2C+Yadong+Xi%2C+Jingzhen+Ding%2C+Duan+Wang%2C+Bai+Liu%2C+Changjie+Fan%2C+Xiaoxi+Mao%2C+and+Zeng+Zhao%2E+Easy+and+Efficient+Transformer%3A+Scalable+Inference+Solution+For+large+NLP+model%2E+arXiv+preprint+arX)
37. Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. 2021. [find](https://scholar.google.com/scholar?q=Opher+Lieber%2C+Or+Sharir%2C+Barak+Lenz%2C+and+Yoav+Shoham%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+2021%2E)
38. Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, pages 7005–7015, 2021. [find](https://scholar.google.com/scholar?q=Xudong+Lin%2C+Gedas+Bertasius%2C+Jue+Wang%2C+Shih%2DFu+Chang%2C+Devi+Parikh%2C+and+Lorenzo+Torresani%2E+Vx2text%3A+End%2Dto%2Dend+learning+of+video%2Dbased+text+generation+from+multimodal+inputs%2E+In+Proceedings+of+the+2021)
39. Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks, pages 881–897. 2020. [find](https://scholar.google.com/scholar?q=Lingxiao+Ma%2C+Zhiqiang+Xie%2C+Zhi+Yang%2C+Jilong+Xue%2C+Youshan+Miao%2C+Wei+Cui%2C+Wenxiang+Hu%2C+Fan+Yang%2C+Lintao+Zhang%2C+and+Lidong+Zhou%2E+Rammer%3A+Enabling+Holistic+Deep+Learning+Compiler+Optimizations+with+rTasks)
40. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381– 2391, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+Suit+of+Armor+Conduct+Electricity%3F+A+New+Dataset+for+Open+Book+Question+Answering%2E+In+Proceedings+of+the+2018+Conference+on+Empiri)
41. Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. [Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond](https://md.research.elamur.ai/free/en/papers/nallapati-2016-abstractive-text-summarization-sequence-sequence/). In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, 2016.
42. Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. TensorFlow-Serving: Flexible, High-Performance ML Serving. Workshop on Machine Learning Systems at NIPS 2017, 2017. [find](https://scholar.google.com/scholar?q=Christopher+Olston%2C+Noah+Fiedel%2C+Kiril+Gorovoy%2C+Jeremiah+Harmsen%2C+Li+Lao%2C+Fangwei+Li%2C+Vinu+Rajashekhar%2C+Sukriti+Ramesh%2C+and+Jordan+Soyke%2E+TensorFlow%2DServing%3A+Flexible%2C+High%2DPerformance+ML+Serving%2E+Wor)
43. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations…
44. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
45. Romain Paulus, Caiming Xiong, and Richard Socher. [A Deep Reinforced Model for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). In Proceedings of the 6th International Conference on Learning Representations, 2018.
46. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-Learn: Machin… [find](https://scholar.google.com/scholar?q=Fabian+Pedregosa%2C+Ga%C3%ABl+Varoquaux%2C+Alexandre+Gramfort%2C+Vincent+Michel%2C+Bertrand+Thirion%2C+Olivier+Grisel%2C+Mathieu+Blondel%2C+Peter+Prettenhofer%2C+Ron+Weiss%2C+Vincent+Dubourg%2C+Jake+Vanderplas%2C+Alexandre+Pass)
47. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E+2019%2E)
48. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020.
50. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://md.research.elamur.ai/free/en/papers/rajbhandari-2022-deepspeed-moe-advancing-mixture-experts/). arXiv preprint arXiv:2201.05596, 2022.
51. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. [Zero-Shot Text-to-Image Generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). In Proceedings of the 38th International Conference on Machine Learning, pages 8821–8831, 2021.
52. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. [Recipes for Building an Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lingui…
53. Timo Schick and Hinrich Schütze. [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255– 269, 2021.
54. Abigail See, Peter J. Liu, and Christopher D. Manning. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, 2017. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+Get+To+The+Point%3A+Summarization+with+Pointer%2DGenerator+Networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+L)
55. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. [Mesh-TensorFlow: Deep Learning for Supercomputers](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/). Advances in Neural Information Processing Systems, 2018.
56. Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 322–337, 2019. [find](https://scholar.google.com/scholar?q=Haichen+Shen%2C+Lequn+Chen%2C+Yuchen+Jin%2C+Liangyu+Zhao%2C+Bingyu+Kong%2C+Matthai+Philipose%2C+Arvind+Krishnamurthy%2C+and+Ravi+Sundaram%2E+Nexus%3A+A+GPU+Cluster+Engine+for+Accelerating+DNN%2DBased+Video+Analysis%2E+In+P)
57. Haichen Shen, Seungyeop Han, Matthai Philipose, and Arvind Krishnamurthy. Fast Video Classification via Adaptive Cascading of Deep Models. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 3646–3654, 2017. [find](https://scholar.google.com/scholar?q=Haichen+Shen%2C+Seungyeop+Han%2C+Matthai+Philipose%2C+and+Arvind+Krishnamurthy%2E+Fast+Video+Classification+via+Adaptive+Cascading+of+Deep+Models%2E+In+Proceedings+of+the+2017+IEEE+Conference+on+Computer+Vision)
58. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2DLM%3A+Training+Multi%2DBillion+Parameter+Language+Models+Using+Model+Parallelism%2E+arXiv+preprin)
59. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwa… — [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 2017.
61. Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. LightSeq: A High Performance Inference Library for Transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 11… [find](https://scholar.google.com/scholar?q=Xiaohui+Wang%2C+Ying+Xiong%2C+Yang+Wei%2C+Mingxuan+Wang%2C+and+Lei+Li%2E+LightSeq%3A+A+High+Performance+Inference+Library+for+Transformers%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+t)
62. Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. Clip-gen: Language-free training of a text-to-image generator with clip. arXiv preprint arXiv:2203.00386, 2022. — [arXiv:2203.00386](https://arxiv.org/abs/2203.00386) [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Wei+Liu%2C+Qian+He%2C+Xinglong+Wu%2C+and+Zili+Yi%2E+Clip%2Dgen%3A+Language%2Dfree+training+of+a+text%2Dto%2Dimage+generator+with+clip%2E+arXiv+preprint+arXiv%3A2203%2E00386%2C+2022%2E+%E2%80%94+%5BarXiv%3A2203%2E00386%5D%28https%3A%2F%2Farx)
63. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. [Finetuned Language Models are Zero-Shot Learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In Proceedings of the 10th International Conference on Learning Representations, 2022.
64. Ronald J. Williams and David Zipser. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2):270–280, 1989. [find](https://scholar.google.com/scholar?q=Ronald+J%2E+Williams+and+David+Zipser%2E+A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks%2E+Neural+Computation%2C+1%282%29%3A270%E2%80%93280%2C+1989%2E)
65. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In Proceedings of the 32nd International Conference on Machine Learning, pages 2048–2057, 2015.
66. Zhilin Yang, Ye Yuan, Yuexin Wu, William W. Cohen, and Ruslan R. Salakhutdinov. Review Networks for Caption Generation. Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Ye+Yuan%2C+Yuexin+Wu%2C+William+W%2E+Cohen%2C+and+Ruslan+R%2E+Salakhutdinov%2E+Review+Networks+for+Caption+Generation%2E+Advances+in+Neural+Information+Processing+Systems%2C+2016%2E)
67. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Tr… — [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)

*Updated 2 November 2026*