--- title: "NVIDIA A100 Tensor Core GPU Architecture — NVIDIA 2020" description: "This NVIDIA whitepaper (2020) describes the A100 Tensor Core GPU, built on the GA100 chip of the Ampere… Cited by 129 papers here." authors: ["NVIDIA"] year: 2020 venue: "NVIDIA whitepaper" source: "https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf" licence: "NVIDIA, free download — read only" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/nvidia-2020-nvidia-a100-tensor-core-gpu/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nvidia-2020-nvidia-a100-tensor-core-gpu/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / NVIDIA A100 Tensor Core GPU Architecture # NVIDIA A100 Tensor Core GPU Architecture **Authors:** [NVIDIA](https://md.research.elamur.ai/free/en/authors/nvidia/) · **Year:** 2020 · **Venue:** NVIDIA whitepaper · **Licence:** NVIDIA, free download — read only **Links:** [Publisher's page](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf) · [PDF](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf) **Citations:** [129](https://md.research.elamur.ai/free/en/papers/nvidia-2020-nvidia-a100-tensor-core-gpu/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This NVIDIA whitepaper (2020) describes the A100 Tensor Core GPU, built on the GA100 chip of the Ampere architecture and positioned as the eighth-generation data center GPU after Tesla V100. Modern clouds run very different workloads, including training, inference, HPC and analytics. NVIDIA wanted one accelerator that scales up for large jobs and can also be split across many small ones. The main hardware changes are: - third-generation Tensor Cores that support all common DL and HPC data types, including the new TF32 format; - hardware acceleration for fine-grained 2:4 structured sparsity; - 40 GB of HBM2 and a 40 MB L2 cache; - third-generation NVLink and PCIe Gen 4 with SR-IOV; - Multi-Instance GPU (MIG), which divides one card into isolated instances. On the CUDA side, new features include asynchronous copy, asynchronous barriers, faster task graphs and L2 residency control. On accuracy, an appendix reports that 2:4-sparse networks match their dense FP16 baselines across vision and NLP tasks. The A100 matters because it was the reference platform for large-scale model training at that time, for example in the DGX A100 system. ## Key points - The full GA100 chip has 128 SMs, of which the shipping A100 enables 108. The paper cites a 1410 MHz boost clock for its decode measurements. - MIG can expose up to 7 isolated GPU Instances on a single A100, with fault isolation between them. Each GPU Instance can be further split into Compute Instances. - 2:4 sparsity results (dense vs sparse FP16) from Table 11: | Model | Metric | Dense | 2:4 sparse | |---|---|---|---| | ResNet-50 | ImageNet Top-1 | 76.6 | 76.8 | | MaskRCNN-ResNet-50 | COCO bbox AP | 37.9 | 37.9 | | GNMT | WMT'14 En-De BLEU | 24.6 | 24.9 | | FairSeq Transformer | WMT'14 En-De BLEU | 28.2 | 28.5 | | BERT Large | SQuAD v1.1 F1 | 91.1 | 91.5 | | Transformer XL | enwik8 BPC | 1.06 | 1.06 | - Mixed-precision Tensor Cores are applied to HPC through the TCAIRS solver. It reaches FP64-level accuracy faster than a baseline FP64 direct solver. - The chip also includes fixed-function NVJPG and NVDEC decoders and an Optical Flow Accelerator to feed DL pipelines. The DGX A100 system combines eight A100 GPUs. ## Interesting ideas - **Prune, then fine-tune into a regular pattern.** The 2:4 scheme keeps 2 of every 4 weights, so the sparsity is regular enough for hardware to exploit, unlike arbitrary pruning. The paper prunes trained dense networks and fine-tunes them to recover accuracy. This recipe could be tried for compressing our NLP models before deployment. - **Cheap precision first, refine afterwards.** TCAIRS does the heavy work in low precision and then refines iteratively to full FP64 accuracy. The same principle could apply wherever a fast approximate step can be corrected by a cheap exact one. - **Hard partitioning instead of time-sharing.** MIG gives each tenant dedicated, isolated resources, which yields predictable latency and fault containment. This is a useful design model for sharing one expensive GPU among several small inference or annotation services. - **Hide data movement behind compute.** Asynchronous copy, asynchronous barriers and L2 residency control let programs prefetch data and keep it close to the cores while computation continues. The general lesson for data pipelines is to overlap loading with processing. - **Report compression on varied tasks with native metrics.** Table 11 checks the sparsity trade-off across classification, detection, translation and language modelling, each with its own metric (Top-1, bbox AP, BLEU, BPC, F1). This makes the "no accuracy loss" claim far more convincing than a single benchmark would. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (1): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works 🧭 Topics (1): [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works ## Cite Cite as: NVIDIA. 2020. NVIDIA A100 Tensor Core GPU Architecture. NVIDIA whitepaper. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{nvidia2020nvidia, title = {{NVIDIA A100 Tensor Core GPU Architecture}}, author = {NVIDIA}, booktitle = {NVIDIA whitepaper}, year = {2020}, url = {https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf} } ``` - [← previous Linguistic Features for Readability Assessment](https://md.research.elamur.ai/free/en/papers/deutsch-2020-linguistic-features-readability-assessment/) - [next → Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) *Updated 2 November 2026*