NVIDIA Ampere GA100 ‘Worlds Biggest 7nm GPU’ Official – Full Architecture Deep Dive, 8192 Cores, 48 GB HBM2, 20X Faster Than Volta

May 14, 2020

NVIDIA has officially lifted the curtains off its greatest and most powerful GPU to date, the 7nm Ampere GPU. The first product to feature the new Ampere architecture is a GPU called GA100 & this chip is currently the largest GPU to be produced on the bleeding edge TSMC's 7nm process node. Today, we will be taking a deep-dive in the Ampere GA100 GPU architecture, specifications & the first products that it would be featured inside.

NVIDIA's Ampere GA100 GPU Official - World's Biggest 7nm GPU With Insane Specs

The Ampere GA100 GPU is by far the largest 7nm GPU ever designed. The GPU is designed entirely for the HPC market with applications such as scientific research, Artificial Intelligence, Deep Neural Networking, and AI Inferencing. There's a lot of specifications and a lot of products to talk about so let's start.

NVIDIA GeForce RTX 30 Ampere Gaming Graphics Card Launching in 24 GB, 20 GB, 10 GB Variants, Rumors Points To RTX 3080 Ti & RTX 3080 Custom Cards at Launch

First of all, the NVIDIA Ampere GA100 GPU will be available in various form factors. Ranging from a singular Mezzanine Modular card to full-length PCIe 4.0 graphics card form factors. The GPU also comes in various configurations but the one NVIDIA is highlighting today is the Tesla A100 which is used on the DGX A100 and HGX A100 system.

The NVIDIA 7nm Ampere GA100 GPU Architecture & Specifications

When it comes to core specifications, the Ampere GA100 GPU from NVIDIA is a complete monster. Measuring in at a massive 826mm2 which is even bigger than the Volta GV100 GPU which was 815 mm2. The GPU also features more than twice the number of transistors at 54 Billion versus 21.1 on its predecessor which is very impressive. Given the die size and the transistor count, the Ampere GA100 GPU is single-handily the densest GPU ever built.

The full implementation of the NVIDIA Ampere GA100 GPU includes the following units:

  • 8 GPCs, 8 TPCs/GPC, 2 SMs/TPC, 16 SMs/GPC, 128 SMs per full GPU
  • 64 FP32 CUDA Cores/SM, 8192 FP32 CUDA Cores per full GPU
  • 4 third-generation Tensor Cores/SM, 512 third-generation Tensor Cores per full GPU
  • 6 HBM2 stacks, 12 512-bit memory controllers

The A100 Tensor Core GPU implementation of the NVIDIA Ampere GA100 GPU includes the following units:

  • 7 GPCs, 7 or 8 TPCs/GPC, 2 SMs/TPC, up to 16 SMs/GPC, 108 SMs
  • 64 FP32 CUDA Cores/SM, 6912 FP32 CUDA Cores per GPU
  • 4 third-generation Tensor Cores/SM, 432 third-generation Tensor Cores per GPU
  • 5 HBM2 stacks, 10 512-bit memory controllers

Figure 4 shows a full GA100 GPU with 128 SMs. The A100 is base

NVIDIA To Host GeForce Special Event on 1st September, Will Announce Next-Gen Ampere Gaming ‘GeForce RTX’ Graphics Cards

While the Tesla A100 features cut-down specifications due to early 7nm yields which are still great considering the size of this 'SUPER GPU', the NVIDIA Ampere GA100 GPU in its full-fat version is what we're going to be looking at first.

Featuring 128 SMs with 8192 CUDA cores, the NVIDIA Ampere GA100 also houses the largest single GPU core count we've ever seen. It comes with 8192 FP32 cores, 4096 FP64 cores, and 512 tensor cores. There are 8 Graphics Processing Clusters on the GPU, each with 16 SM units and 8 TPCs. The GA100 GPU has a TDP of 400W for its Tesla A100 variant.

The NVIDIA A100 GPU is a technical design breakthrough fueled by five key innovations:

  • NVIDIA Ampere architecture — At the heart of A100 is the NVIDIA Ampere GPU architecture, which contains more than 54 billion transistors, making it the world’s largest 7-nanometer processor.
  • Third-generation Tensor Cores with TF32 — NVIDIA’s widely adopted Tensor Cores are now more flexible, faster, and easier to use. Their expanded capabilities include new TF32 for AI, which allows for up to 20x the AI performance of FP32 precision, without any code changes. In addition, Tensor Cores now support FP64, delivering up to 2.5x more compute than the previous generation for HPC applications.
  • Multi-instance GPU — MIG, a new technical feature, enables a single A100 GPU to be partitioned into as many as seven separate GPUs so it can deliver varying degrees of compute for jobs of different sizes, providing optimal utilization and maximizing return on investment.
  • Third-generation NVIDIA NVLink — Doubles the high-speed connectivity between GPUs to provide efficient performance scaling in a server.
  • Structural sparsity — This new efficiency technique harnesses the inherently sparse nature of AI math to double the performance.

Other specifications for the NVIDIA Ampere GA100 GPU include a huge 6144-bit bus interface which features up to 48 GB HBM2e memory in six HBM2 stacks that are scattered around the GPU die. Each stack has 2 GB VRAM capacity per die so to reach 48 GB, you would need 4-hi stacks. Each 4-hi stack would consist of 8GB capacity and 6 stacks equal 48 GB capacity. The memory is stated to be running at over 2.0 Gbps pin speeds which would result in around 1.6 Tbps bandwidth.

The NVIDIA Ampere GPU will come with several HBM memory configurations but it maxes out at 48 GB unless NVIDIA wants to offer a 6-hi or 8-hi variant in the future which would raise the memory capacity to 72 or even 96 GB. NVIDIA's Tesla V100S already double the HBM capacity of the Tesla V100, offering 32 GB vs 16 GB so it's entirely possible NVIDIA could do the same with a future variant of the Tesla A100.

NVIDIA Ampere GA100 GPU Block Diagram:

NVIDIA Ampere GA100 GPU SM Block Diagram:

NVIDIA Ampere GA100 Compute

GPUKepler GK110Maxwell GM200Pascal GP100Volta GV100Ampere GA100
Compute Capability3.
Threads / Warp3232323232
Max Warps / Multiprocessor6464646464
Max Threads / Multiprocessor20482048204820482048
Max Thread Blocks / Multiprocessor1632323232
Max 32-bit Registers / SM6553665536655366553665536
Max Registers / Block6553632768655366553665536
Max Registers / Thread255255255255255
Max Thread Block Size10241024102410241024
CUDA Cores / SM192128646464
Shared Memory Size / SM Configurations (bytes)16K/32K/48K96K64K96K164K

The NVIDIA Tesla A100 Accelerator - Specs & Performance

With the specifications of the full-fat NVIDIA Ampere GA100 GPU covered, let's talk about the Tesla A100 graphics accelerator itself. The Tesla A100 makes use of a cut-down variant of the Ampere GA100 GPU that offers 108 SMs featuring 6912 FP32 cores, 3456 FP64 cores, and 432 Tensor cores. The card comes with a 5120-bit bus interface with a maximum VRAM capacity of 40 GB HBM2. It is interesting here because 40 GB HBM2 would suggest either a 5-hi stack design which seems unlikely or a 6-hi stack with a defective DRAM chip on each stack. In the case of the former, a spacer would be introduced on the GA100 HBM stack to fill up its space.

The NVIDIA Ampere Tesla A100 features a 400W TDP which is 100W more than the Tesla V100 Mezzanine unit. The PCIe variant comes with a 300W TDP but has lowered down clock speeds. The Mezzanine board has a GPU-to-GPU connection through the new NVLINK switches which enables up to 600 Gb/s GPU-To-GPU interconnect and 4.8 Tb/s bi-directional channel. The PCIe variant has a Mellanox switch on board along with two next-gen NVLINK connections and two EDR ports.

V100 A100 A100  Sparsity1  A100 Speedup A100 Speedup with Sparsity
A100 FP16 vs. V100 FP16  31.4 TFLOPS 78 TFLOPS N/A 2.5x N/A
A100 FP16 TC vs. V100 FP16 TC 125 TFLOPS 312 TFLOPS 624 TFLOPS 2.5x 5x
A100 BF16 TC vs.V100 FP16 TC 125 TFLOPS 312 TFLOPS 624 TFLOPS 2.5x 5x
A100 FP32 vs. V100 FP32 15.7 TFLOPS 19.5 TFLOPS N/A 1.25x N/A
A100 TF32 TC vs. V100 FP32  15.7 TFLOPS 156 TFLOPS 312 TFLOPS 10x 20x
A100 FP64 vs. V100 FP64 7.8 TFLOPS 9.7 TFLOPS N/A 1.25x N/A
A100 FP64 TC vs. V100 FP64 7.8 TFLOPS 19.5 TFLOPS N/A 2.5x N/A
A100 INT8 TC vs. V100 INT8 62 TOPS 624 TOPS 1248 TOPS 10x 20x
A100 INT4 TC N/A 1248 TOPS 2496 TOPS N/A N/A
A100 Binary TC N/A 4992 TOPS N/A N/A N/A

In terms of performance, the NVIDIA Ampere GA100 GPU delivers 1 Peta-OPs which is a 20x increase over the Volta GV100 GPU. The double-precision performance is rated at 2.5x higher over NVIDIA's Volta GV100 GPU which should end up somewhere around 19.5 TFLOPs FP64 since Volta had around 8 TFLOPs FP64 compute power. This would mean that the single-precision performance is rated at over 19.5 standard rates and up to 156 TFLOPs (FP32) which would be mind-blowing for the HPC segment.

NVIDIA Ampere GA100 GPU Based Tesla A100 Specs:

NVIDIA Tesla Graphics CardTesla K40
Tesla M40
Tesla P100
Tesla P100 (SXM2)Tesla V100 (SXM2)Tesla V100S (PCIe)NVIDIA A100 (SXM4)NVIDIA A100 (PCIe4)
GPUGK110 (Kepler)GM200 (Maxwell)GP100 (Pascal)GP100 (Pascal)GV100 (Volta)GV100 (Volta)GA100 (Ampere)GA100 (Ampere)
Process Node28nm28nm16nm16nm12nm12nm7nm7nm
Transistors7.1 Billion8 Billion15.3 Billion15.3 Billion21.1 Billion21.1 Billion54.2 Billion54.2 Billion
GPU Die Size551 mm2601 mm2610 mm2610 mm2815mm2815mm2826mm2826mm2
FP32 CUDA Cores Per SM192128646464646464
FP64 CUDA Cores / SM644323232323232
FP32 CUDA Cores28803072358435845120512069126912
FP64 CUDA Cores96096179217922560256034563456
Tensor CoresN/AN/AN/AN/A640640432432
Texture Units240192224224320320432432
Boost Clock875 MHz1114 MHz1329MHz1480 MHz1530 MHz1601 MHz1410 MHz1410 MHz
TOPs (DNN/AI)N/AN/AN/AN/A125 TOPs130 TOPs1248 TOPs
2496 TOPs with Sparsity
1248 TOPs
2496 TOPs with Sparsity
FP16 ComputeN/AN/A18.7 TFLOPs21.2 TFLOPs30.4 TFLOPs32.8 TFLOPs312 TFLOPs
624 TFLOPs with Sparsity
312 TFLOPs
624 TFLOPs with Sparsity
FP32 Compute5.04 TFLOPs6.8 TFLOPs10.0 TFLOPs10.6 TFLOPs15.7 TFLOPs16.4 TFLOPs156 TFLOPs
(19.5 TFLOPs standard)
156 TFLOPs
(19.5 TFLOPs standard)
FP64 Compute1.68 TFLOPs0.2 TFLOPs4.7 TFLOPs5.30 TFLOPs7.80 TFLOPs8.2 TFLOPs19.5 TFLOPs
(9.7 TFLOPs standard)
19.5 TFLOPs
(9.7 TFLOPs standard)
Memory Interface384-bit GDDR5384-bit GDDR54096-bit HBM24096-bit HBM24096-bit HBM24096-bit HBM26144-bit HBM2e6144-bit HBM2e
Memory Size12 GB GDDR5 @ 288 GB/s24 GB GDDR5 @ 288 GB/s16 GB HBM2 @ 732 GB/s
12 GB HBM2 @ 549 GB/s
16 GB HBM2 @ 732 GB/s16 GB HBM2 @ 900 GB/s16 GB HBM2 @ 1134 GB/s40 GB HBM2 @ 1.6 TB/s40 GB HBM2 @ 1.6 TB/s
L2 Cache Size1536 KB3072 KB4096 KB4096 KB6144 KB6144 KB40960 KB40960 KB

The NVIDIA Ampere GA100 GPU Hardware

NVIDIA's Tesla V100 which is based on the Ampere GA100 GPU will be powering the company's both DGX & HGX stations. The DGX stations focus purely on AI research and HPC workloads and HGX stations focus their prowess at cloud computing and datacenter environments. The systems being introduced by NVIDIA include the 3rd Generation DGX-A100 and the HGX-A100.

NVIDIA's partners have already announced their new 1U, 2U, 4U, and up to 10U GPU servers. Each server is outfitted with up to 8 NVIDIA Ampere GA100 based Tesla A100 boards, making use of PCIe Gen 4.0 x16 links. An HGX A100 4 GPU board is also available for improved performance while keeping the costs to a more affordable range.

Finally, NVIDIA will be announcing its next-generation DGX-A100 system which Jensen Huang teased a few days ago. The DGX-A100 will deliver 5 Petaflops of peak performance with eight NVIDIA Ampere based Tesla A100 GPUs.

The system itself is 20x faster than the previous DGX based on NVIDIA's Volta GPU architecture. The reference cluster design features 140 DGX-A100 GPUs with a 200 Gbps Mellanox Infiniband interconnect. The NVIDIA Ampere powered DGX-A100 system is going to start at $199,000 and is shipping as of today.