---
title: "From Words to Watts — Samsi 2023"
description: "Most published estimates of the energy cost of large language models focus on training."
authors: ["Siddharth Samsi", "Dan Zhao", "Joseph McDonald", "Baolin Li", "Adam Michaleas", "Michael Jones", "William Bergeron", "Jeremy Kepner", "Devesh Tiwari", "Vijay Gadepally"]
year: 2023
venue: "arXiv:2310.03003"
source: "https://arxiv.org/abs/2310.03003"
doi: "10.48550/arxiv.2310.03003"
arxiv: "2310.03003"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/samsi-2023-words-watts-benchmarking-energy-costs/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/samsi-2023-words-watts-benchmarking-energy-costs/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

# From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

**Authors:** Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally · **Year:** 2023 · **Venue:** arXiv:2310.03003 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.03003) · [PDF](https://arxiv.org/pdf/2310.03003) · [DOI 10.48550/arxiv.2310.03003](https://doi.org/10.48550/arxiv.2310.03003) · [arXiv 2310.03003](https://arxiv.org/abs/2310.03003) · [OpenAlex](https://openalex.org/W4390189088)

**Citations:** 213 citations (openalex-snapshot · 2026-10-14) · [75](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most published estimates of the energy cost of large language models focus on training. Inference gets far less attention, even though deployed models answer queries constantly and may account for a larger share of their total energy use. This paper measures what it takes, in time, computation and energy, to run inference with Meta AI's LLaMA. The authors test the 7B, 13B and 65B variants on the MIT Supercloud cluster, using two GPU generations (NVIDIA V100 and A100). They shard the model across up to 32 GPUs on multiple nodes. Two workloads are used to see whether the type of task changes the energy profile: instruction-following prompts from the Alpaca dataset and grade-school math problems from GSM8K. The authors present this as one of the first inference energy studies at this multi-node scale. They intend it as a baseline that encourages more open reporting of performance across hardware and models. *Note: the quantitative results section was missing from the supplied extract, so no measured figures are given here.*

## Key points
- **Hardware:** The experiments ran on MIT Supercloud.
  - The cluster has 448 GPU nodes. Each node has two Intel Xeon Gold 6248 CPUs, 384 GB of RAM and two 32 GB V100 GPUs.
  - Four extra nodes with A100 GPUs were also used.
  - All runs used only the 25 Gb/s Ethernet interconnect, not the 100 Gb/s Omnipath fabric.
- **Model:** LLaMA 65B is the main subject. It has a model dimension of 8,192, 80 layers and 64 attention heads, and was trained on 1.4 trillion tokens. The 7B and 13B variants serve as single-node baselines.
- **Datasets:**
  - Alpaca: 52,000 instruction tasks.
  - GSM8K: about 8,500 math word problems.
  - 4,096 inputs were sampled from each dataset, because running the full sets would add cost without changing the conclusions.
- **Tooling:** Multi-GPU inference used model sharding, with FairScale cited. GPU monitoring relied on NVIDIA tooling, with nvidia-smi and DCGM cited.

## Interesting ideas
- **Benchmark with contrasting task types.** Math prompts differ from free-form instructions in input length, output length and number of decoded tokens. These differences can shift throughput and energy, so cost benchmarks should cover more than one kind of workload.
- **Subsample the evaluation data on purpose.** A fixed sample of 4,096 inputs per dataset keeps the cost of the experiments themselves reasonable. This applies to any expensive profiling study, including our own.
- **Report small models alongside the large one.** Including 7B and 13B runs gives a reference point for how costs scale with model size. This helps teams choose the smallest model that meets their needs.
- **Document the exact setup.** Reporting the interconnect, node layout and GPU generation makes energy numbers reproducible and comparable across clusters. This practice is worth adopting whenever we publish compute or energy figures.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 73 not yet.

- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{samsi2023words,
  title = {{From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference}},
  author = {Samsi, Siddharth and Zhao, Dan and McDonald, Joseph and Li, Baolin and Michaleas, Adam and Jones, Michael and Bergeron, William and Kepner, Jeremy and Tiwari, Devesh and Gadepally, Vijay},
  year = {2023},
  doi = {10.48550/arxiv.2310.03003},
  eprint = {2310.03003},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03003}
}
```

- [← previous From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://md.research.elamur.ai/free/en/papers/li-2023-quantity-quality-boosting-llm-performance/)
- [next → FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stability-AI, “Stable Diffusion,” [https://github.com/Stability-AI/](https://github.com/Stability-AI/) StableDiffusion, 2023. [↗](https://github.com/Stability-AI/)
2. D. Foster, Generative deep learning. ” O’Reilly Media, Inc.”, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Foster%2C+Generative+deep+learning%2E+%E2%80%9D+O%E2%80%99Reilly+Media%2C+Inc%2E%E2%80%9D%2C+2022%2E)
3. Z. Epstein, A. Hertzmann, the Investigators of Human Creativity et al., “Art and the science of generative ai,” Science, vol. 380, no. 6650, pp. 1110–1111, 2023. [Online]. Available: [https://www.science.org/doi/abs/10.1126/science.adh4451](https://www.science.org/doi/abs/10.1126/science.adh4451) — [doi:10.1126/science.adh4451](https://doi.org/10.1126/science.adh4451) [↗](https://www.science.org/doi/abs/10.1126/science.adh4451)
4. B. Meskó and E. J. Topol, “The imperative for regulatory oversight of large language models (or generative ai) in healthcare,” npj Digital Medicine, vol. 6, no. 1, p. 120, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Mesko%CC%81+and+E%2E+J%2E+Topol%2C+%E2%80%9CThe+imperative+for+regulatory+oversight+of+large+language+models+%28or+generative+ai%29+in+healthcare%2C%E2%80%9D+npj+Digital+Medicine%2C+vol%2E+6%2C+no%2E+1%2C+p%2E+120%2C+2023%2E)
5. H. Zohny, J. McMillan, and M. King, “Ethics of generative ai,” Journal of Medical Ethics, vol. 49, no. 2, pp. 79–80, 2023. [Online]. Available: [https://jme.bmj.com/content/49/2/79](https://jme.bmj.com/content/49/2/79) [↗](https://jme.bmj.com/content/49/2/79)
6. E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3645– 3650. [find](https://scholar.google.com/scholar?q=E%2E+Strubell%2C+A%2E+Ganesh%2C+and+A%2E+McCallum%2C+%E2%80%9CEnergy+and+policy+considerations+for+deep+learning+in+NLP%2C%E2%80%9D+in+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E+Floren) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
7. M. Shoeybi, M. Patwary, R. Puri et al., “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri+et+al%2E%2C+%E2%80%9CMegatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C%E2%80%9D+2020%2E)
8. D. Narayanan, M. Shoeybi, J. Casper et al., “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021. [find](https://scholar.google.com/scholar?q=D%2E+Narayanan%2C+M%2E+Shoeybi%2C+J%2E+Casper+et+al%2E%2C+%E2%80%9CEfficient+large%2Dscale+language+model+training+on+gpu+clusters+using+megatron%2Dlm%2C%E2%80%9D+2021%2E)
9. “Nvidia/megatron-lm: Ongoing research training transformer models at scale,” [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM), 2023. [↗](https://github.com/NVIDIA/Megatron-LM)
10. J. Sevilla, L. Heim, A. Ho et al., “Compute trends across three eras of machine learning,” 2022. [find](https://scholar.google.com/scholar?q=J%2E+Sevilla%2C+L%2E+Heim%2C+A%2E+Ho+et+al%2E%2C+%E2%80%9CCompute+trends+across+three+eras+of+machine+learning%2C%E2%80%9D+2022%2E)
11. D. Patel, “The ai brick wall – a practical limit for scaling dense transformer models, and how gpt 4 will break past it,” [https://www.semianalysis.com/p/the-ai-brick-wall-a-practical-limit](https://www.semianalysis.com/p/the-ai-brick-wall-a-practical-limit), 2023. [↗](https://www.semianalysis.com/p/the-ai-brick-wall-a-practical-limit)
12. R. Desislavov, F. Martı́nez-Plumed, and J. Hernández-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,” Sustainable Computing: Informatics and Systems, vol. 38, p. 100857, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Desislavov%2C+F%2E+Mart%C4%B1%CC%81nez%2DPlumed%2C+and+J%2E+Herna%CC%81ndez%2DOrallo%2C+%E2%80%9CTrends+in+ai+inference+energy+consumption%3A+Beyond+the+performance%2Dvs%2Dparameter+laws+of+deep+learning%2C%E2%80%9D+Sustainable+Computing%3A+Informatics)
13. D. Zhao, N. C. Frey, J. McDonald et al., “A green(er) world for a.i.” in 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2022, pp. 742–750. [find](https://scholar.google.com/scholar?q=D%2E+Zhao%2C+N%2E+C%2E+Frey%2C+J%2E+McDonald+et+al%2E%2C+%E2%80%9CA+green%28er%29+world+for+a%2Ei%2E%E2%80%9D+in+2022+IEEE+International+Parallel+and+Distributed+Processing+Symposium+Workshops+%28IPDPSW%29%2C+2022%2C+pp%2E+742%E2%80%93750%2E)
14. H. Touvron, T. Lavril, G. Izacard et al., “Llama: Open and efficient foundation language models,” 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+T%2E+Lavril%2C+G%2E+Izacard+et+al%2E%2C+%E2%80%9CLlama%3A+Open+and+efficient+foundation+language+models%2C%E2%80%9D+2023%2E)
15. “Different development paths of llms,” [https://www.interconnects.ai/p/](https://www.interconnects.ai/p/) llm-development-paths. [↗](https://www.interconnects.ai/p/)
16. A. Vaswani, N. Shazeer, N. Parmar et al., “Attention is all you need,” 2017. [find](https://scholar.google.com/scholar?q=A%2E+Vaswani%2C+N%2E+Shazeer%2C+N%2E+Parmar+et+al%2E%2C+%E2%80%9CAttention+is+all+you+need%2C%E2%80%9D+2017%2E)
17. R. Gozalo-Brizuela and E. C. Garrido-Merchan, “Chatgpt is not all you need. a state of the art review of large generative ai models,” 2023. [find](https://scholar.google.com/scholar?q=R%2E+Gozalo%2DBrizuela+and+E%2E+C%2E+Garrido%2DMerchan%2C+%E2%80%9CChatgpt+is+not+all+you+need%2E+a+state+of+the+art+review+of+large+generative+ai+models%2C%E2%80%9D+2023%2E)
18. A. Reuther, J. Kepner, C. Byun et al., “Interactive supercomputing on 40,000 cores for machine learning and data analysis,” in 2018 IEEE High Performance extreme Computing Conference (HPEC). IEEE, 2018, pp. 1–6. [find](https://scholar.google.com/scholar?q=A%2E+Reuther%2C+J%2E+Kepner%2C+C%2E+Byun+et+al%2E%2C+%E2%80%9CInteractive+supercomputing+on+40%2C000+cores+for+machine+learning+and+data+analysis%2C%E2%80%9D+in+2018+IEEE+High+Performance+extreme+Computing+Conference+%28HPEC%29%2E+IEEE%2C+201)
19. Facebook Research, online, 2023. [Online]. Available: [https://github.com/facebookresearch/llama](https://github.com/facebookresearch/llama) [↗](https://github.com/facebookresearch/llama)
20. FairScale authors, “Fairscale: A general purpose modular pytorch library for high performance and large scale training,” [https://github.com/](https://github.com/) facebookresearch/fairscale, 2021. [↗](https://github.com/)
21. R. Taori, I. Gulrajani, T. Zhang et al., “Stanford alpaca: An instruction-following llama model,” [https://github.com/tatsu-lab/stanford](https://github.com/tatsu-lab/stanford) alpaca, 2023. [↗](https://github.com/tatsu-lab/stanford)
22. K. Cobbe, V. Kosaraju, M. Bavarian et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Cobbe%2C+V%2E+Kosaraju%2C+M%2E+Bavarian+et+al%2E%2C+%E2%80%9CTraining+verifiers+to+solve+math+word+problems%2C%E2%80%9D+arXiv+preprint+arXiv%3A2110%2E14168%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
23. NVIDIA. Nvidia-smi. [Online]. Available: [http://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf](http://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf) [↗](http://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf)
24. ——. Nvidia data center GPU manager (dcgm). [Online]. Available: [https://developer.nvidia.com/dcgm](https://developer.nvidia.com/dcgm) [↗](https://developer.nvidia.com/dcgm)
25. J. McDonald, B. Li, N. Frey et al., “Great power, great responsibility: Recommendations for reducing energy for training language models,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 1962–1970. [find](https://scholar.google.com/scholar?q=J%2E+McDonald%2C+B%2E+Li%2C+N%2E+Frey+et+al%2E%2C+%E2%80%9CGreat+power%2C+great+responsibility%3A+Recommendations+for+reducing+energy+for+training+language+models%2C%E2%80%9D+in+Findings+of+the+Association+for+Computational+Linguistics%3A)
26. J. Devlin, M.-W. Chang, K. Lee et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Paper… [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E%2DW%2E+Chang%2C+K%2E+Lee+et+al%2E%2C+%E2%80%9CBERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2C%E2%80%9D+in+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+As)
27. NVIDIA, “Multi-Process Service,” [https://docs.nvidia.com/deploy/mps/](https://docs.nvidia.com/deploy/mps/), 2023. [↗](https://docs.nvidia.com/deploy/mps/)
28. NVIDIA, “NVIDIA Multi Instance GPU User Guide,” [https://docs.nvidia.com/datacenter/tesla/mig-user-guide/](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/), 2023. [↗](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/)
29. B. Li, T. Patel, S. Samsi et al., “Miso: exploiting multi-instance gpu capability on multi-tenant gpu clusters,” in Proceedings of the 13th Symposium on Cloud Computing, 2022, pp. 173–189. [find](https://scholar.google.com/scholar?q=B%2E+Li%2C+T%2E+Patel%2C+S%2E+Samsi+et+al%2E%2C+%E2%80%9CMiso%3A+exploiting+multi%2Dinstance+gpu+capability+on+multi%2Dtenant+gpu+clusters%2C%E2%80%9D+in+Proceedings+of+the+13th+Symposium+on+Cloud+Computing%2C+2022%2C+pp%2E+173%E2%80%93189%2E)