--- title: "NVIDIA Tensor Core Programmability, Performance & Precision" description: "The paper asks whether traditional HPC codes, not just deep learning, can benefit from the Tensor… 4 references, cited by 117 papers here." authors: ["Stefano Markidis", "Steven Wei Der Chien", "Erwin Laure", "Ivy Bo Peng", "Jeffrey S. Vetter"] year: 2018 venue: "IPDPSW 2018 (arXiv)" source: "https://arxiv.org/abs/1803.04014" doi: "10.1109/ipdpsw.2018.00091" arxiv: "1803.04014" licence: "arXiv — non-exclusive distribution license (arXiv OAI)" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/markidis-2018-nvidia-tensor-core-programmability-performance/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/markidis-2018-nvidia-tensor-core-programmability-performance/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / NVIDIA Tensor Core Programmability, Performance & Precision # NVIDIA Tensor Core Programmability, Performance & Precision **Authors:** Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng and Jeffrey S. Vetter · **Year:** 2018 · **Venue:** IPDPSW 2018 (arXiv) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://doi.org/10.1109/ipdpsw.2018.00091) · [PDF](https://arxiv.org/pdf/1803.04014) · [DOI 10.1109/ipdpsw.2018.00091](https://doi.org/10.1109/ipdpsw.2018.00091) · [arXiv 1803.04014](https://arxiv.org/abs/1803.04014) · [OpenAlex](https://openalex.org/W2791673912) **Citations:** 348 citations (OpenAlex, 2026-10-14) · [117](https://md.research.elamur.ai/free/en/papers/markidis-2018-nvidia-tensor-core-programmability-performance/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks whether traditional HPC codes, not just deep learning, can benefit from the Tensor Cores in NVIDIA's Volta GPUs. These units perform a 4×4 matrix-multiply-and-accumulate each clock cycle. They take half-precision inputs and accumulate in single precision. The authors study three questions on a Tesla V100: - **Programmability:** they compare the three available interfaces, from lowest to highest level. These are the CUDA 9 WMMA API, the templated CUTLASS library built on WMMA, and cuBLAS GEMM. - **Performance:** they benchmark large GEMMs and batched small GEMMs, and compare the results with ordinary CUDA cores. - **Precision:** they measure how much accuracy is lost when inputs are rounded to half precision, across different matrix sizes. They also propose a way to reduce that loss by doing extra computation. The best measured throughput is about 83 Tflops/s in mixed precision. That is roughly seven times single-precision speed and three times half-precision speed. These results matter because upcoming supercomputers such as Summit and Sierra rely on V100 GPUs. HPC developers therefore need concrete guidance on how to program these units and what accuracy to expect. ## Key points - **Hardware:** a Tesla V100 has 80 SMs and 640 Tensor Cores. Each Tensor Core performs 64 FMA operations per cycle. - **Theoretical peaks:** 125 Tflops/s for Tensor Cores, compared with 31.4 (FP16), 15.7 (FP32) and 7.8 (FP64) Tflops/s on the regular cores. - **Scale:** Summit, with six V100s per node across about 4,600 nodes, will contain nearly 18M Tensor Cores. - **WMMA:** the API works on 16×16 tiles (non-square shapes from CUDA 9.1) per warp of 32 threads. It is a preview feature with no backward-compatibility guarantee. - **Measured results:** large GEMMs reach about 83 Tflops/s. A batched GEMM written with WMMA reaches about 4 Tflops/s. ## Interesting ideas - **Compare abstraction levels on one operation.** The authors benchmark the same GEMM through a low-level API, a template library and a vendor BLAS. This separates the cost of expressiveness from raw speed, and the same design works for evaluating any new accelerator or library stack. - **Overdecompose the problem.** WMMA exposes only tiles larger than the 4×4 hardware unit, so many more threads run than there are compute units. The extra threads help hide memory and instruction latency, a principle that applies to other parallel kernels too. - **Report accuracy as a function of problem size.** Measuring error against matrix size, rather than at one size, shows how low-precision error grows. This is a useful evaluation habit whenever reduced-precision arithmetic or quantization is considered. - **Trade compute for accuracy.** When the hardware's native precision is too low, spending extra fast low-precision operations can recover much of the lost accuracy. This trade-off is worth weighing for any quantized pipeline. - **Benchmark batched small problems separately.** Batched small GEMMs behave very differently from one large GEMM, so they deserve their own tests. This matters for workloads made of many small independent operations. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [KTH Royal Institute of Technology](https://md.research.elamur.ai/free/en/universities/kth-royal-institute-of-technology/) — University, Stockholm, Sweden · 20 works · 1,723 citations - [Oak Ridge National Laboratory](https://md.research.elamur.ai/free/en/labs/oak-ridge-national-laboratory/) — Research institute, Oak Ridge, United States · 6 works · 881 citations 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works 🧭 Topics (1): [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works ## Cite Cite as: Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng and Jeffrey S. Vetter. 2018. NVIDIA Tensor Core Programmability, Performance & Precision. IPDPSW 2018 (arXiv). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{markidis2018nvidia, title = {{NVIDIA Tensor Core Programmability, Performance \& Precision}}, author = {Stefano Markidis, Steven Wei {Der Chien}, Erwin Laure, Ivy Bo Peng and Jeffrey S. Vetter}, year = {2018}, doi = {10.1109/ipdpsw.2018.00091}, eprint = {1803.04014}, archivePrefix = {arXiv}, url = {https://doi.org/10.1109/ipdpsw.2018.00091} } ``` - [← previous Joint Embedding of Words and Labels for Text Classification](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/) - [next → Graph-to-Sequence Learning using Gated Graph Neural Networks](https://md.research.elamur.ai/free/en/papers/beck-2018-graph-sequence-learning-gated-graph/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. [27] “NVIDIA CUDA toolkit release notes,” accessed: 2018-03-01. [Online]. Available: [http://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html](http://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) [↗](http://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) 2. [28] N. J. Higham, “The accuracy of floating point summation,” SIAM Journal on Scientific Computing, vol. 14, no. 4, pp. 783–799, 1993. [find](https://scholar.google.com/scholar?q=%5B28%5D+N%2E+J%2E+Higham%2C+%E2%80%9CThe+accuracy+of+floating+point+summation%2C%E2%80%9D+SIAM+Journal+on+Scientific+Computing%2C+vol%2E+14%2C+no%2E+4%2C+pp%2E+783%E2%80%93799%2C+1993%2E) 3. [29] P. Luszczek, J. Kurzak, I. Yamazaki, and J. Dongarra, “Towards numerical benchmark for half-precision floating point arithmetic,” in High Performance Extreme Computing Conference (HPEC), 2017 IEEE. [find](https://scholar.google.com/scholar?q=%5B29%5D+P%2E+Luszczek%2C+J%2E+Kurzak%2C+I%2E+Yamazaki%2C+and+J%2E+Dongarra%2C+%E2%80%9CTowards+numerical+benchmark+for+half%2Dprecision+floating+point+arithmetic%2C%E2%80%9D+in+High+Performance+Extreme+Computing+Conference+%28HPEC%29%2C+2017+IEE) 4. [30] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “[cuDNN: Efficient primitives for deep learning](https://md.research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/),” arXiv preprint arXiv:1410.0759, 2014. *Updated 2 November 2026*