---
title: "TPU v4 — paper by Zhou; and Patterson (2023)"
description: "The paper presents three novelties of a system in production since 2020. 62 references, cited by 107 papers here."
authors: ["Norman P. Jouppi; George Kurian; Sheng Li; Peter Ma; Rahul Nagarajan; Lifeng Nai; Nishant Patil; Suvinay Subramanian; Andy Swing; Brian Towles; Cliff Young; Xiang Zhou; Zongwei Zhou;", "David Patterson"]
year: 2023
venue: "arXiv:2304.01433"
source: "https://arxiv.org/abs/2304.01433"
doi: "10.48550/arxiv.2304.01433"
arxiv: "2304.01433"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jouppi-2023-tpu-v4-optically-reconfigurable-supercomputer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jouppi-2023-tpu-v4-optically-reconfigurable-supercomputer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

# TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

**Authors:** Norman P. Jouppi; George Kurian; Sheng Li; Peter Ma; Rahul Nagarajan; Lifeng Nai; Nishant Patil; Suvinay Subramanian; Andy Swing; Brian Towles; Cliff Young; Xiang Zhou; Zongwei Zhou; and [David Patterson](https://md.research.elamur.ai/free/en/authors/david-patterson/) · **Year:** 2023 · **Venue:** [arXiv:2304.01433](https://md.research.elamur.ai/free/en/venues/proceedings-international-symposium-on-computer-architecture-proceedings/) ([Proceedings - International Symposium on Computer Architecture/Proceedings 2023](https://md.research.elamur.ai/free/en/venues/proceedings-international-symposium-on-computer-architecture-proceedings/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.01433) · [PDF](https://arxiv.org/pdf/2304.01433) · [DOI 10.48550/arxiv.2304.01433](https://doi.org/10.48550/arxiv.2304.01433) · [arXiv 2304.01433](https://arxiv.org/abs/2304.01433) · [OpenAlex](https://openalex.org/W4380874786)

**Citations:** 508 citations (OpenAlex, 2026-10-14) · [107](https://md.research.elamur.ai/free/en/papers/jouppi-2023-tpu-v4-optically-reconfigurable-supercomputer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Google's TPU v4 paper tackles a hard problem: building a machine-learning supercomputer with thousands of chips that stays reliable and efficient while workloads keep changing. Over time, demand moved from MLPs and RNNs toward Transformers and LLMs, alongside embedding-heavy recommender models (DLRMs). The paper presents three novelties of a system in production since 2020.

1. **Optical interconnect.** Electrically cabled 4×4×4 cubes of 64 chips are linked through 48 Palomar optical circuit switches (OCSes) built on MEMS mirrors. Together they form a 4096-chip 3D torus. Its topology can be reconfigured, including twisted tori, and it can bypass failed CPU hosts.
2. **SparseCore.** This dataflow unit for embeddings speeds up DLRMs 5x–7x while using roughly 5% of die area and power.
3. **Their interaction.** Embeddings create all-to-all traffic, which benefits from flexible topologies with higher bisection bandwidth.

Against TPU v3, TPU v4 is 2.1x faster with 2.7x better performance/Watt. The optics cost under 5% of the system and draw under 3% of power. PaLM trained at 57.8% of peak FLOPS. The lesson is that a flexible interconnect can be a cheap lever for scale, uptime and efficiency.

## Key points
- **System size.** A full machine has 4096 chips in 64 racks, with 4 TPUs per host (about 1K hosts).
  - Each 64-chip block exposes 96 optical links, which attach to 48 OCSes.
  - The Palomar OCS is 136×136: 128 working ports plus 8 spares for testing and repair.
- **Chip design.** Each chip holds two TensorCores sharing a 128 MiB CMEM. Each TensorCore contains:
  - four 128×128 MXUs,
  - a 128-lane VPU,
  - 16 MiB of VMEM.
- **Workload shift.** By October 2022, Transformers took 57% of TPU v4 training (LLM 31%, BERT 26%), while RNNs dropped to 2%.
- **Comparisons with similar-sized systems.**
  - Versus Graphcore IPU Bow: about 4.3x–4.5x faster.
  - Versus Nvidia A100: 1.2x–1.7x faster and 1.3x–1.9x lower power.
  - In Google Cloud: about 2–6x less energy and about 20x less CO2e than typical on-premise DSAs.
- **Availability and deployment.** Without OCSes, hosts would need 99.9% availability to give reasonable slice goodput. With OCSes, each rack could enter service on its own instead of waiting for the full system.

## Interesting ideas
- **Small blocks plus a reprogrammable switch layer.** Building from small, independently testable blocks joined by a reprogrammable "patchboard" lets capacity come online gradually and lets failed parts be skipped. The same pattern suits compute clusters, data pipelines, or any system assembled from unreliable components.
- **Goodput with spares as an evaluation metric.** Measuring effective throughput while explicitly reserving spare capacity, rather than counting raw peak, gives an honest picture of what users actually get. This is worth reusing when planning shared GPU budgets.
- **Logging the real workload mix over time.** The authors tracked which model types actually consumed their hardware over several years and used that record to justify design choices. An NLP group could keep a similar record to decide where optimization effort pays off.
- **Matching topology to communication pattern.** Users pick the slice shape per job, including twisted tori, to fit how that job communicates (all-reduce versus all-to-all). The general lesson is to let the job's traffic pattern drive the infrastructure configuration.
- **Layered carbon accounting.** The CO2e estimate combines chip efficiency, data-center efficiency and the grid's carbon-free share. This layered approach gives a reusable template for reporting the footprint of model training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 81 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 64 works

🧭 Topics (1): [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 19 works

💡 Concepts (1): [💡 Machine learning](https://md.research.elamur.ai/free/en/concepts/machine-learning/) · 51 works

## Cite

Cite as: Norman P. Jouppi; George Kurian; Sheng Li; Peter Ma; Rahul Nagarajan; Lifeng Nai; Nishant Patil; Suvinay Subramanian; Andy Swing; Brian Towles; Cliff Young; Xiang Zhou; Zongwei Zhou; and David Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings. arXiv:2304.01433.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2023tpu,
  title = {{TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings}},
  author = {Norman P. Jouppi; George Kurian; Sheng Li; Peter Ma; Rahul Nagarajan; Lifeng Nai; Nishant Patil; Suvinay Subramanian; Andy Swing; Brian Towles; Cliff Young; Xiang Zhou; Zongwei Zhou; and David Patterson},
  year = {2023},
  doi = {10.48550/arxiv.2304.01433},
  eprint = {2304.01433},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.01433}
}
```

- [← previous Instruction-Following Evaluation for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)
- [next → Role-Play with Large Language Models](https://md.research.elamur.ai/free/en/papers/shanahan-2023-role-play-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

62 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, R., Gadanho, S., Huang, D., Jacob, N., Li, Z., Lin, D., Phillips, T., Pop, C., Regan, K., Shamir, G.I. and Shivanna, R., 2022. On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models. 16th ACM Conference on Recommender Systems. Barroso, L.A., and Hölzle, U., 2009, F… [find](https://scholar.google.com/scholar?q=Anil%2C+R%2E%2C+Gadanho%2C+S%2E%2C+Huang%2C+D%2E%2C+Jacob%2C+N%2E%2C+Li%2C+Z%2E%2C+Lin%2C+D%2E%2C+Phillips%2C+T%2E%2C+Pop%2C+C%2E%2C+Regan%2C+K%2E%2C+Shamir%2C+G%2EI%2E+and+Shivanna%2C+R%2E%2C+2022%2E+On+the+Factory+Floor%3A+ML+Engineering+for+Industrial%2DScale+Ads+Recom)
2. Barroso, L.A., Hölzle, U. and Ranganathan, P., 2018. The datacenter as a computer: Designing warehouse-scale machines, Third Edition. Synthesis Lectures on Computer Architecture, 13(3), pp.i-189. [find](https://scholar.google.com/scholar?q=Barroso%2C+L%2EA%2E%2C+H%C3%B6lzle%2C+U%2E+and+Ranganathan%2C+P%2E%2C+2018%2E+The+datacenter+as+a+computer%3A+Designing+warehouse%2Dscale+machines%2C+Third+Edition%2E+Synthesis+Lectures+on+Computer+Architecture%2C+13%283%29%2C+pp%2Ei%2D189%2E)
3. Bloomberg October 26, 2015. Google turning its lucrative web search over to AI machines. [find](https://scholar.google.com/scholar?q=Bloomberg+October+26%2C+2015%2E+Google+turning+its+lucrative+web+search+over+to+AI+machines%2E)
4. Bouknight, W.J., Denenberg, S.A., McIntyre, D.E., Randall, J.M., Sameh, A.H. and Slotnick, D.L., 1972. The ILLIAC IV system. Proceedings of the IEEE, 60(4), pp. 369-388. [find](https://scholar.google.com/scholar?q=Bouknight%2C+W%2EJ%2E%2C+Denenberg%2C+S%2EA%2E%2C+McIntyre%2C+D%2EE%2E%2C+Randall%2C+J%2EM%2E%2C+Sameh%2C+A%2EH%2E+and+Slotnick%2C+D%2EL%2E%2C+1972%2E+The+ILLIAC+IV+system%2E+Proceedings+of+the+IEEE%2C+60%284%29%2C+pp%2E+369%2D388%2E)
5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems 33 (NeurIPS 2020).
6. Camara, J.M., Moreto, M., Vallejo, E., Beivide, R., Miguel-Alonso, J., Martínez, C. and Navaridas, J., 2010. Twisted torus topologies for enhanced interconnection networks. IEEE Transactions on Parallel and Distributed Systems, 21(12), pp. 1765-1778. [find](https://scholar.google.com/scholar?q=Camara%2C+J%2EM%2E%2C+Moreto%2C+M%2E%2C+Vallejo%2C+E%2E%2C+Beivide%2C+R%2E%2C+Miguel%2DAlonso%2C+J%2E%2C+Mart%C3%ADnez%2C+C%2E+and+Navaridas%2C+J%2E%2C+2010%2E+Twisted+torus+topologies+for+enhanced+interconnection+networks%2E+IEEE+Transactions+on+Parall)
7. Camarero, C., Martinez, C. and Beivide, R., 2014. Lattice graphs for high-scale interconnection topologies. IEEE Transactions on Parallel and Distributed Systems, 26(9), pp. 2506-2519. [find](https://scholar.google.com/scholar?q=Camarero%2C+C%2E%2C+Martinez%2C+C%2E+and+Beivide%2C+R%2E%2C+2014%2E+Lattice+graphs+for+high%2Dscale+interconnection+topologies%2E+IEEE+Transactions+on+Parallel+and+Distributed+Systems%2C+26%289%29%2C+pp%2E+2506%2D2519%2E)
8. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S. and Schuh, P., 2022. [PaLM: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
9. Colamco, 2022. Mellanox QM8790 - Quantum HDR Switch, [https://www.colamco.com/product/mellanox-infiniband-switch-mq](https://www.colamco.com/product/mellanox-infiniband-switch-mq) m8790-hs2f-1503410?utm_source=froogle&utm_medium=referral. [↗](https://www.colamco.com/product/mellanox-infiniband-switch-mq)
10. Covington, P., Adams, J. and Sargin, E., 2016. Deep neural networks for Youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198. [find](https://scholar.google.com/scholar?q=Covington%2C+P%2E%2C+Adams%2C+J%2E+and+Sargin%2C+E%2E%2C+2016%2E+Deep+neural+networks+for+Youtube+recommendations%2E+In+Proceedings+of+the+10th+ACM+conference+on+recommender+systems%2C+pp%2E+191%2D198%2E)
11. Dally, W.J. and Towles, B.P., 2004. Principles and practices of interconnection networks. Elsevier. [find](https://scholar.google.com/scholar?q=Dally%2C+W%2EJ%2E+and+Towles%2C+B%2EP%2E%2C+2004%2E+Principles+and+practices+of+interconnection+networks%2E+Elsevier%2E)
12. Deepmind, Nov 18, 2019, Advanced machine learning helps Play Store users discover personalised apps. [find](https://scholar.google.com/scholar?q=Deepmind%2C+Nov+18%2C+2019%2C+Advanced+machine+learning+helps+Play+Store+users+discover+personalised+apps%2E)
13. Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., 2018. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
14. Dungworth, M., Harrell, J., Levine, M., Nelson, S., Oberlin, S. and Reinhardt, S.P., 2011. CRAY T3E. [find](https://scholar.google.com/scholar?q=Dungworth%2C+M%2E%2C+Harrell%2C+J%2E%2C+Levine%2C+M%2E%2C+Nelson%2C+S%2E%2C+Oberlin%2C+S%2E+and+Reinhardt%2C+S%2EP%2E%2C+2011%2E+CRAY+T3E%2E)
15. Fan, X., Weber, W.D. and Barroso, L.A., 2007. Power provisioning for a warehouse-sized computer. In 2007 ACM/IEEE 34th International Symposium on Computer Architecture (ISCA), pp.13-23. [find](https://scholar.google.com/scholar?q=Fan%2C+X%2E%2C+Weber%2C+W%2ED%2E+and+Barroso%2C+L%2EA%2E%2C+2007%2E+Power+provisioning+for+a+warehouse%2Dsized+computer%2E+In+2007+ACM%2FIEEE+34th+International+Symposium+on+Computer+Architecture+%28ISCA%29%2C+pp%2E13%2D23%2E)
16. Farrington, N. ., G. Porter, S. Radhakrishnan, H. H. Bazzaz, V. Subramanya, Y. Fainman, G. Papen, and A. Vahdat. Helios: A hybrid electrical/optical switch architecture for modular data centers. In Proceedings of ACM SIGCOMM, Aug. 2010. [find](https://scholar.google.com/scholar?q=Farrington%2C+N%2E+%2E%2C+G%2E+Porter%2C+S%2E+Radhakrishnan%2C+H%2E+H%2E+Bazzaz%2C+V%2E+Subramanya%2C+Y%2E+Fainman%2C+G%2E+Papen%2C+and+A%2E+Vahdat%2E+Helios%3A+A+hybrid+electrical%2Foptical+switch+architecture+for+modular+data+centers%2E+In+Pr)
17. Ghaemmaghami, B., Ozdal, M., Komuravelli, R., Korchev, D., Mudigere, D., Nair, K. and Naumov, M., 2022. Learning to Collide: Recommendation System Model Compression with Learned Hash Functions. arXiv preprint arXiv:2203.15837. — [arXiv:2203.15837](https://arxiv.org/abs/2203.15837) [find](https://scholar.google.com/scholar?q=Ghaemmaghami%2C+B%2E%2C+Ozdal%2C+M%2E%2C+Komuravelli%2C+R%2E%2C+Korchev%2C+D%2E%2C+Mudigere%2C+D%2E%2C+Nair%2C+K%2E+and+Naumov%2C+M%2E%2C+2022%2E+Learning+to+Collide%3A+Recommendation+System+Model+Compression+with+Learned+Hash+Functions%2E+arXiv+)
18. Google, Tracking our carbon free energy progress, [https://sustainability.google/progress/energy/](https://sustainability.google/progress/energy/). [↗](https://sustainability.google/progress/energy/)
19. Google, Google Data Center Efficiency, [https://www.google.com/about/datacenters/efficiency/](https://www.google.com/about/datacenters/efficiency/). [↗](https://www.google.com/about/datacenters/efficiency/)
20. Graphcore, 2022. IPU-POD64 Reference Design Datasheet, docs.graphcore.ai/projects/ ipu-pod64-datasheet/en/latest/ [find](https://scholar.google.com/scholar?q=Graphcore%2C+2022%2E+IPU%2DPOD64+Reference+Design+Datasheet%2C+docs%2Egraphcore%2Eai%2Fprojects%2F+ipu%2Dpod64%2Ddatasheet%2Fen%2Flatest%2F)
21. Horowitz, M., 2014. Computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14. [find](https://scholar.google.com/scholar?q=Horowitz%2C+M%2E%2C+2014%2E+Computing%27s+energy+problem+%28and+what+we+can+do+about+it%29%2E+In+2014+IEEE+International+Solid%2DState+Circuits+Conference+Digest+of+Technical+Papers+%28ISSCC%29%2C+pp%2E+10%2D14%2E)
22. Insight, 2022. Mellanox Quantum QM8790 - switch - 40 ports, [https://www.insight.com/en_US/shop/](https://www.insight.com/en_US/shop/) product/MQM8790-HS2F/Mellanox/MQM8790-HS2F/MellanoxQua ntumQM8790-swit/. [↗](https://www.insight.com/en_US/shop/)
23. International Energy Agency, Global Energy & CO2 Status Report 2019, Report Extract Emissions. [find](https://scholar.google.com/scholar?q=International+Energy+Agency%2C+Global+Energy+%26+CO2+Status+Report+2019%2C+Report+Extract+Emissions%2E)
24. Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T., Gottipati, R., Gulland, W., Hagmann, R., Ho, C.R., Hogberg, D.… [find](https://scholar.google.com/scholar?q=Jouppi%2C+N%2EP%2E%2C+Young%2C+C%2E%2C+Patil%2C+N%2E%2C+Patterson%2C+D%2E%2C+Agrawal%2C+G%2E%2C+Bajwa%2C+R%2E%2C+Bates%2C+S%2E%2C+Bhatia%2C+S%2E%2C+Boden%2C+N%2E%2C+Borchers%2C+A%2E%2C+Boyle%2C+R%2E%2C+Cantin%2C+P%2E%2C+Chao%2C+C%2E%2C+Clark%2C+C%2E%2C+Coriell%2C+J%2E%2C+Daley%2C+M%2E%2C+Dau%2C+M%2E%2C+)
25. Jouppi, N.P., Yoon, D.H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C. and Patterson, D., 2020. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7), pp. 67-78. [find](https://scholar.google.com/scholar?q=Jouppi%2C+N%2EP%2E%2C+Yoon%2C+D%2EH%2E%2C+Kurian%2C+G%2E%2C+Li%2C+S%2E%2C+Patil%2C+N%2E%2C+Laudon%2C+J%2E%2C+Young%2C+C%2E+and+Patterson%2C+D%2E%2C+2020%2E+A+domain%2Dspecific+supercomputer+for+training+deep+neural+networks%2E+Communications+of+the+ACM%2C+63)
26. Jouppi, N.P., Yoon, D.H., Ashcraft, M., Gottscho, M., Jablin, T.B., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., Norrie, T., Patil, N., Prasad, S., Young, C., Zhou, Z., and Patterson, D. 2021. Ten lessons from three generations shaped Google’s TPUv4i. In 2021 ACM/IEEE 48th Annual International Sy… [find](https://scholar.google.com/scholar?q=Jouppi%2C+N%2EP%2E%2C+Yoon%2C+D%2EH%2E%2C+Ashcraft%2C+M%2E%2C+Gottscho%2C+M%2E%2C+Jablin%2C+T%2EB%2E%2C+Kurian%2C+G%2E%2C+Laudon%2C+J%2E%2C+Li%2C+S%2E%2C+Ma%2C+P%2E%2C+Ma%2C+X%2E%2C+Norrie%2C+T%2E%2C+Patil%2C+N%2E%2C+Prasad%2C+S%2E%2C+Young%2C+C%2E%2C+Zhou%2C+Z%2E%2C+and+Patterson%2C+D%2E+2021%2E+Ten+)
27. Kamil, S., Pinar, A., Gunter, D., Lijewski, M., Oliker, L. and Shalf, J., 2007, May. Reconfigurable hybrid interconnection for static and dynamic scientific applications. In Proceedings of the 4th Int'l conference on computing frontiers, pp. 183-194. [find](https://scholar.google.com/scholar?q=Kamil%2C+S%2E%2C+Pinar%2C+A%2E%2C+Gunter%2C+D%2E%2C+Lijewski%2C+M%2E%2C+Oliker%2C+L%2E+and+Shalf%2C+J%2E%2C+2007%2C+May%2E+Reconfigurable+hybrid+interconnection+for+static+and+dynamic+scientific+applications%2E+In+Proceedings+of+the+4th+Int)
28. Kamil, S., Oliker, L., Pinar, A. and Shalf, J., 2009. Communication requirements and interconnect optimization for high-end scientific applications. IEEE Transactions on Parallel and Distributed Systems, 21(2), pp. 188-202. [find](https://scholar.google.com/scholar?q=Kamil%2C+S%2E%2C+Oliker%2C+L%2E%2C+Pinar%2C+A%2E+and+Shalf%2C+J%2E%2C+2009%2E+Communication+requirements+and+interconnect+optimization+for+high%2Dend+scientific+applications%2E+IEEE+Transactions+on+Parallel+and+Distributed+Syste)
29. Karandikar, S., Mao, H., Kim, D., Biancolin, D., Amid, A., Lee, D., Pemberton, N., Amaro, E., Schmidt, C., Chopra, A. and Huang, Q., 2018, June. FireSim: FPGA-accelerated cycle-exact scale-out system simulation in the public cloud. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Arc… [find](https://scholar.google.com/scholar?q=Karandikar%2C+S%2E%2C+Mao%2C+H%2E%2C+Kim%2C+D%2E%2C+Biancolin%2C+D%2E%2C+Amid%2C+A%2E%2C+Lee%2C+D%2E%2C+Pemberton%2C+N%2E%2C+Amaro%2C+E%2E%2C+Schmidt%2C+C%2E%2C+Chopra%2C+A%2E+and+Huang%2C+Q%2E%2C+2018%2C+June%2E+FireSim%3A+FPGA%2Daccelerated+cycle%2Dexact+scale%2Dout+system+)
30. Khani, M., Ghobadi, M., Alizadeh, M., Zhu, Z., Glick, M., Bergman, K., Vahdat, A., Klenk, B. and Ebrahimi, E., 2021, August. SiP-ML: high-bandwidth optical network interconnects for machine learning training. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (pp. 657-675). [find](https://scholar.google.com/scholar?q=Khani%2C+M%2E%2C+Ghobadi%2C+M%2E%2C+Alizadeh%2C+M%2E%2C+Zhu%2C+Z%2E%2C+Glick%2C+M%2E%2C+Bergman%2C+K%2E%2C+Vahdat%2C+A%2E%2C+Klenk%2C+B%2E+and+Ebrahimi%2C+E%2E%2C+2021%2C+August%2E+SiP%2DML%3A+high%2Dbandwidth+optical+network+interconnects+for+machine+learning+t)
31. Li, S., Tan, M., Pang, R., Li, A., Cheng, L., Le, Q.V. and Jouppi, N.P., 2021. Searching for Fast Model Families on Datacenter Accelerators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8085-8095. [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Tan%2C+M%2E%2C+Pang%2C+R%2E%2C+Li%2C+A%2E%2C+Cheng%2C+L%2E%2C+Le%2C+Q%2EV%2E+and+Jouppi%2C+N%2EP%2E%2C+2021%2E+Searching+for+Fast+Model+Families+on+Datacenter+Accelerators%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Visio)
32. Li, S., Andersen, G., Chen, T., Cheng, L., Grady, J., Huang, D., Le, Q., Li, A., Li, X., Li, Y., Liang, C., Lu, Y., Ni, Y., Pang, F., Ranganathan , P., Tan, M., Wicke, M., Wu, G., Zhu, S., and Jouppi, N., 2023. Hyperscale Hardware Optimized Neural Architecture Search, In Proceedings of the 28th Inte… [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Andersen%2C+G%2E%2C+Chen%2C+T%2E%2C+Cheng%2C+L%2E%2C+Grady%2C+J%2E%2C+Huang%2C+D%2E%2C+Le%2C+Q%2E%2C+Li%2C+A%2E%2C+Li%2C+X%2E%2C+Li%2C+Y%2E%2C+Liang%2C+C%2E%2C+Lu%2C+Y%2E%2C+Ni%2C+Y%2E%2C+Pang%2C+F%2E%2C+Ranganathan+%2C+P%2E%2C+Tan%2C+M%2E%2C+Wicke%2C+M%2E%2C+Wu%2C+G%2E%2C+Zhu%2C+S%2E%2C+and+Jouppi%2C)
33. Masanet, E., Shehabi, A., Lei, N., Smith, S. and Koomey, J., 2020. Recalibrating global data center energy-use estimates. Science, 367(6481), pp.984-986. [find](https://scholar.google.com/scholar?q=Masanet%2C+E%2E%2C+Shehabi%2C+A%2E%2C+Lei%2C+N%2E%2C+Smith%2C+S%2E+and+Koomey%2C+J%2E%2C+2020%2E+Recalibrating+global+data+center+energy%2Duse+estimates%2E+Science%2C+367%286481%29%2C+pp%2E984%2D986%2E)
34. Minkenberg, C., Rodriguez, G., Prisacari, B., Schares, L., Heidelberger, P., Chen, D. and Stunkel, C., 2016, March. Performance benefits of optical circuit switches for large-scale dragonfly networks. In 2016 Optical Fiber Communications Conference and Exhibition (OFC) (pp. 1-3). IEEE. [find](https://scholar.google.com/scholar?q=Minkenberg%2C+C%2E%2C+Rodriguez%2C+G%2E%2C+Prisacari%2C+B%2E%2C+Schares%2C+L%2E%2C+Heidelberger%2C+P%2E%2C+Chen%2C+D%2E+and+Stunkel%2C+C%2E%2C+2016%2C+March%2E+Performance+benefits+of+optical+circuit+switches+for+large%2Dscale+dragonfly+networks%2E)
35. MLCommons, V 2.0 Results, June 29, 2022, [https://mlcommons.org/en/training-normal-20/](https://mlcommons.org/en/training-normal-20/). [↗](https://mlcommons.org/en/training-normal-20/)
36. Mudigere, D., Hao, Y., Huang, J., Jia, Z., Tulloch, A., Sridharan, S., Liu, X., Ozdal, M., Nie, J., Park, J. and Luo, L., 2022, June. Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the 49th Annual International Symposium on Comput… [find](https://scholar.google.com/scholar?q=Mudigere%2C+D%2E%2C+Hao%2C+Y%2E%2C+Huang%2C+J%2E%2C+Jia%2C+Z%2E%2C+Tulloch%2C+A%2E%2C+Sridharan%2C+S%2E%2C+Liu%2C+X%2E%2C+Ozdal%2C+M%2E%2C+Nie%2C+J%2E%2C+Park%2C+J%2E+and+Luo%2C+L%2E%2C+2022%2C+June%2E+Software%2Dhardware+co%2Ddesign+for+fast+and+scalable+training+of+deep)
37. Nayak, P., 2021. MUM: A new AI milestone for understanding information. Google, May, 18. [find](https://scholar.google.com/scholar?q=Nayak%2C+P%2E%2C+2021%2E+MUM%3A+A+new+AI+milestone+for+understanding+information%2E+Google%2C+May%2C+18%2E)
38. Norrie, T., Patil, N., Yoon, D.H., Kurian, G., Li, S., Laudon, J., Young, C., Jouppi, N. and Patterson, D., 2021. The design process for Google's training chips: TPUv2 and TPUv3. IEEE Micro, 41(2), pp. 56-63. [find](https://scholar.google.com/scholar?q=Norrie%2C+T%2E%2C+Patil%2C+N%2E%2C+Yoon%2C+D%2EH%2E%2C+Kurian%2C+G%2E%2C+Li%2C+S%2E%2C+Laudon%2C+J%2E%2C+Young%2C+C%2E%2C+Jouppi%2C+N%2E+and+Patterson%2C+D%2E%2C+2021%2E+The+design+process+for+Google%27s+training+chips%3A+TPUv2+and+TPUv3%2E+IEEE+Micro%2C+41%282%29%2C+pp)
39. Nvidia, 2020. Nvidia A100 Tensor Core GPU Architecture. [find](https://scholar.google.com/scholar?q=Nvidia%2C+2020%2E+Nvidia+A100+Tensor+Core+GPU+Architecture%2E)
40. Nvidia, 2021. Nvidia DGX SuperPOD: Scalable Infrastructure for AI Leadership Reference Architecture. [find](https://scholar.google.com/scholar?q=Nvidia%2C+2021%2E+Nvidia+DGX+SuperPOD%3A+Scalable+Infrastructure+for+AI+Leadership+Reference+Architecture%2E)
41. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.M., Rothchild, D., So, D., Texier, M. and Dean, J., 2022. The carbon footprint of machine learning training will plateau then shrink, IEEE Computer, 55(7), pp. 18-28. [find](https://scholar.google.com/scholar?q=Patterson%2C+D%2E%2C+Gonzalez%2C+J%2E%2C+Le%2C+Q%2E%2C+Liang%2C+C%2E%2C+Munguia%2C+L%2EM%2E%2C+Rothchild%2C+D%2E%2C+So%2C+D%2E%2C+Texier%2C+M%2E+and+Dean%2C+J%2E%2C+2022%2E+The+carbon+footprint+of+machine+learning+training+will+plateau+then+shrink%2C+IEEE+Co)
42. Poutievski, L., Mashayekhi, O., Ong, J., Singh, A., Tariq, M., Wang, R., Zhang, J., Beauregard, V., Conner, P., Gribble, S. and Kapoor, R., 2022. Jupiter evolving: transforming Google's datacenter network via optical circuit switches and software-defined networking. In Proceedings of the ACM SIGCOMM… [find](https://scholar.google.com/scholar?q=Poutievski%2C+L%2E%2C+Mashayekhi%2C+O%2E%2C+Ong%2C+J%2E%2C+Singh%2C+A%2E%2C+Tariq%2C+M%2E%2C+Wang%2C+R%2E%2C+Zhang%2C+J%2E%2C+Beauregard%2C+V%2E%2C+Conner%2C+P%2E%2C+Gribble%2C+S%2E+and+Kapoor%2C+R%2E%2C+2022%2E+Jupiter+evolving%3A+transforming+Google%27s+datacenter+net)
43. Raghavendra, R., Ranganathan, P., Talwar, V., Wang, Z. and Zhu, X., 2008, March. No "power" struggles: coordinated multi-level power management for the data center. In Proceedings of the 13th International Conference on architectural support for programming languages and operating systems (ASPLOS),… [find](https://scholar.google.com/scholar?q=Raghavendra%2C+R%2E%2C+Ranganathan%2C+P%2E%2C+Talwar%2C+V%2E%2C+Wang%2C+Z%2E+and+Zhu%2C+X%2E%2C+2008%2C+March%2E+No+%22power%22+struggles%3A+coordinated+multi%2Dlevel+power+management+for+the+data+center%2E+In+Proceedings+of+the+13th+Internat)
44. Schwartz, R., Dodge, J., Smith, N.A. and Etzioni, O., 2020. [Green AI](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/). Communications of the ACM, 63(12), pp. 54-63.
45. Scott, S.L., 1996. Synchronization and communication in the T3E multiprocessor. In Proceedings of the 7th International conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 26-36. [find](https://scholar.google.com/scholar?q=Scott%2C+S%2EL%2E%2C+1996%2E+Synchronization+and+communication+in+the+T3E+multiprocessor%2E+In+Proceedings+of+the+7th+International+conference+on+Architectural+Support+for+Programming+Languages+and+Operating+Syst)
46. Sequin, C.H., 1981. Doubly twisted torus networks for VLSI processor arrays. In Proceedings of the 8th International annual Symposium on Computer Architecture (ISCA), pp. 471-480. [find](https://scholar.google.com/scholar?q=Sequin%2C+C%2EH%2E%2C+1981%2E+Doubly+twisted+torus+networks+for+VLSI+processor+arrays%2E+In+Proceedings+of+the+8th+International+annual+Symposium+on+Computer+Architecture+%28ISCA%29%2C+pp%2E+471%2D480%2E)
47. Sethi, G., Bhattacharya, P., Choudhary, D., Wu, C.-J., and Kozyrakis, C., 2022. FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Models, arxiv preprint arXiv:2301.02959. — [arXiv:2301.02959](https://arxiv.org/abs/2301.02959) [find](https://scholar.google.com/scholar?q=Sethi%2C+G%2E%2C+Bhattacharya%2C+P%2E%2C+Choudhary%2C+D%2E%2C+Wu%2C+C%2E%2DJ%2E%2C+and+Kozyrakis%2C+C%2E%2C+2022%2E+FlexShard%3A+Flexible+Sharding+for+Industry%2DScale+Sequence+Recommendation+Models%2C+arxiv+preprint+arXiv%3A2301%2E02959%2E+%E2%80%94+%5BarXi)
48. Shalf, J., Kamil, S., Oliker, L. and Skinner, D., 2005, November. Analyzing ultra-scale application communication requirements for a reconfigurable hybrid interconnect. In SC'05: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing (pp. 17-17). [find](https://scholar.google.com/scholar?q=Shalf%2C+J%2E%2C+Kamil%2C+S%2E%2C+Oliker%2C+L%2E+and+Skinner%2C+D%2E%2C+2005%2C+November%2E+Analyzing+ultra%2Dscale+application+communication+requirements+for+a+reconfigurable+hybrid+interconnect%2E+In+SC%2705%3A+Proceedings+of+the+20)
49. Shaw, D.E., Deneroff, M.M., Dror, R.O., Kuskin, J.S., Larson, R.H., Salmon, J.K., Young, C., Batson, B., Bowers, K.J., Chao, J.C. and Eastwood, M.P., 2008. Anton, a special-purpose machine for molecular dynamics simulation. Communications of the ACM, 51(7), pp. 91-99. [find](https://scholar.google.com/scholar?q=Shaw%2C+D%2EE%2E%2C+Deneroff%2C+M%2EM%2E%2C+Dror%2C+R%2EO%2E%2C+Kuskin%2C+J%2ES%2E%2C+Larson%2C+R%2EH%2E%2C+Salmon%2C+J%2EK%2E%2C+Young%2C+C%2E%2C+Batson%2C+B%2E%2C+Bowers%2C+K%2EJ%2E%2C+Chao%2C+J%2EC%2E+and+Eastwood%2C+M%2EP%2E%2C+2008%2E+Anton%2C+a+special%2Dpurpose+machine+for+molecul)
50. Singla, A., Singh, K. Ramachandran, L. Xu, and Y. Zhang. Proteus: A topology malleable data center network. In Proceedings of ACM Hotnets, Oct. 2010. [find](https://scholar.google.com/scholar?q=Singla%2C+A%2E%2C+Singh%2C+K%2E+Ramachandran%2C+L%2E+Xu%2C+and+Y%2E+Zhang%2E+Proteus%3A+A+topology+malleable+data+center+network%2E+In+Proceedings+of+ACM+Hotnets%2C+Oct%2E+2010%2E)
51. Taylor, P., 2022. Data center average annual power usage effectiveness (PUE) worldwide 2007-2022, Nov 22. [find](https://scholar.google.com/scholar?q=Taylor%2C+P%2E%2C+2022%2E+Data+center+average+annual+power+usage+effectiveness+%28PUE%29+worldwide+2007%2D2022%2C+Nov+22%2E)
52. Teh, M. Y., S. Zhao, P. Cao, and K. Bergman. Couder: Robust topology engineering for optical circuit switched data center networks. arXiv preprint arXiv:2010.00090, Sept. 2020. — [arXiv:2010.00090](https://arxiv.org/abs/2010.00090) [find](https://scholar.google.com/scholar?q=Teh%2C+M%2E+Y%2E%2C+S%2E+Zhao%2C+P%2E+Cao%2C+and+K%2E+Bergman%2E+Couder%3A+Robust+topology+engineering+for+optical+circuit+switched+data+center+networks%2E+arXiv+preprint+arXiv%3A2010%2E00090%2C+Sept%2E+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E00090%5D%28ht)
53. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.T., Jin, A., Bos, T., Baker, L., Du, Y. and Li, Y., 2022. [LaMDA: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
54. Urata, R., Liu, H., Yasumura, K., Mao, E., Berger, J., Zhou, X., Lam, C., Bannon, R., Hutchinson, D., Nelson, D. and Poutievski, L., 2022. Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale. arXiv preprint arXiv:2208.10041. — [arXiv:2208.10041](https://arxiv.org/abs/2208.10041) [find](https://scholar.google.com/scholar?q=Urata%2C+R%2E%2C+Liu%2C+H%2E%2C+Yasumura%2C+K%2E%2C+Mao%2C+E%2E%2C+Berger%2C+J%2E%2C+Zhou%2C+X%2E%2C+Lam%2C+C%2E%2C+Bannon%2C+R%2E%2C+Hutchinson%2C+D%2E%2C+Nelson%2C+D%2E+and+Poutievski%2C+L%2E%2C+2022%2E+Mission+Apollo%3A+Landing+Optical+Circuit+Switching+at+Datacent)
55. US Energy Information Agency, 2022. Oklahoma State Profile and Energy Estimates, [https://www.eia.gov/state/](https://www.eia.gov/state/) ?sid=OK\#:~:text=In%202021%2C%20wind%20supplied%2041,ele ctricity%20net%20generation%20from%20wind. [↗](https://www.eia.gov/state/)
56. Vahdat, A., H. Liu, X. Zhao, and C. Johnson, 2011. The emerging optical data center. In Proceedings of the Optical Fiber Communications Conference. [find](https://scholar.google.com/scholar?q=Vahdat%2C+A%2E%2C+H%2E+Liu%2C+X%2E+Zhao%2C+and+C%2E+Johnson%2C+2011%2E+The+emerging+optical+data+center%2E+In+Proceedings+of+the+Optical+Fiber+Communications+Conference%2E)
57. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30 (NeurIPS 2017).
58. Wang, S., Wei, J., Sabne, A., David, A., Llbeyi, B., Hechtman, B., Chen, D., Murthy, K. S., Maggioni, M., Zhang, Q., Kumar, S., Guo, T., Xu, Y., and Zhou, Z., 2023. Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models, In Proceedings of the 28th Internatio… — [arXiv:2202.00433](https://arxiv.org/abs/2202.00433) [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Wei%2C+J%2E%2C+Sabne%2C+A%2E%2C+David%2C+A%2E%2C+Llbeyi%2C+B%2E%2C+Hechtman%2C+B%2E%2C+Chen%2C+D%2E%2C+Murthy%2C+K%2E+S%2E%2C+Maggioni%2C+M%2E%2C+Zhang%2C+Q%2E%2C+Kumar%2C+S%2E%2C+Guo%2C+T%2E%2C+Xu%2C+Y%2E%2C+and+Zhou%2C+Z%2E%2C+2023%2E+Overlap+Communication+with+Dependen)
59. Williams, S., Waterman, A. and Patterson, D., 2009. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4), pp.65-76. [find](https://scholar.google.com/scholar?q=Williams%2C+S%2E%2C+Waterman%2C+A%2E+and+Patterson%2C+D%2E%2C+2009%2E+Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures%2E+Communications+of+the+ACM%2C+52%284%29%2C+pp%2E65%2D76%2E)
60. Wu, Q., Deng, Q., Ganesh, L., Hsu, C.H., Jin, Y., Kumar, S., Li, B., Meza, J. and Song, Y.J., 2016. Dynamo: Facebook’s Data Center-Wide Power Management System. 2016 ACM/ IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, (2016), pp. 469–480. [find](https://scholar.google.com/scholar?q=Wu%2C+Q%2E%2C+Deng%2C+Q%2E%2C+Ganesh%2C+L%2E%2C+Hsu%2C+C%2EH%2E%2C+Jin%2C+Y%2E%2C+Kumar%2C+S%2E%2C+Li%2C+B%2E%2C+Meza%2C+J%2E+and+Song%2C+Y%2EJ%2E%2C+2016%2E+Dynamo%3A+Facebook%E2%80%99s+Data+Center%2DWide+Power+Management+System%2E+2016+ACM%2F+IEEE+43rd+Annual+Internationa)
61. Xu, Y., Lee, H., Chen, D., Hechtman, B., Huang, Y., Joshi, R., Krikun, M., Lepikhin, D., Ly, A., Maggioni, M. and Pang, R., 2021. GSPMD: general and scalable parallelization for ML computation graphs. arXiv preprint arXiv:2105.04663. [find](https://scholar.google.com/scholar?q=Xu%2C+Y%2E%2C+Lee%2C+H%2E%2C+Chen%2C+D%2E%2C+Hechtman%2C+B%2E%2C+Huang%2C+Y%2E%2C+Joshi%2C+R%2E%2C+Krikun%2C+M%2E%2C+Lepikhin%2C+D%2E%2C+Ly%2C+A%2E%2C+Maggioni%2C+M%2E+and+Pang%2C+R%2E%2C+2021%2E+GSPMD%3A+general+and+scalable+parallelization+for+ML+computation+graphs%2E)
62. Zhao, M., Agarwal, N., Basant, A., Gedik, B., Pan, S., Ozdal, M., Komuravelli, R., Pan, J., Bao, T., Lu, H. and Narayanan, S., 2022, June. Understanding data storage and ingestion for large-scale deep recommendation model training: Industrial product. In Proceedings of the 49th Annual International… [find](https://scholar.google.com/scholar?q=Zhao%2C+M%2E%2C+Agarwal%2C+N%2E%2C+Basant%2C+A%2E%2C+Gedik%2C+B%2E%2C+Pan%2C+S%2E%2C+Ozdal%2C+M%2E%2C+Komuravelli%2C+R%2E%2C+Pan%2C+J%2E%2C+Bao%2C+T%2E%2C+Lu%2C+H%2E+and+Narayanan%2C+S%2E%2C+2022%2C+June%2E+Understanding+data+storage+and+ingestion+for+large%2Dscale+deep)

*Updated 29 October 2026*