NVIDIA Ampere GA100 GPU Powered Tesla A100: Worlds Largest 7nm GPU, 54 Billion Transistors, 1 Petaflops Compute & Up To 96 GB HBM2 Memory


NVIDIA has unveiled the GA100 GPU, their first and also the world's largest 7nm chip based on its next-gen Ampere GPU architecture. Featuring 20 times the performance of its predecessor, the Volta GPU, Ampere ushers in a new era of high-performance computing, being the first GPU in the world to deliver a peak compute power of greater than 1 Peta-Ops per second for AI/DNN.

NVIDIA Unveils The Worlds Largest 7nm GPU, The Ampere GA100 GPU - Powering The Tesla A100 With 54 Billion Transistors and Up To 96 GB Undisputed & Fastest HBM2 Memory

Powered by the next-generation Ampere GPU architecture, the Tesla A100 is an impressive board for the HPC market. The first thing that we have to talk about any HPC GPU is its specs & Ampere is a monster of a chip. NVIDIA went all out with  7nm process node, making GA100 the largest 7nm chip in production but that's not all, it's also the most advanced and feature-pack chip in the industry as of right now.

NVIDIA Launches A2 Tensor Core GPU, An Entry-Level Design Powered By Ampere GA107 GPU & 16 GB GDDR6 Memory

The Ampere GA100 GPU is once again based on a bleeding-edge 7nm process node and has a gargantuan count of 54 Billion transistors packed within it. The chip is expected to pack 128 SM units, equalling a total of 8192 CUDA cores. That alone is a 50% increase in the total number of cores. For memory, we are looking at six HBM stacks that point out a 6144-bit bus interface. The memory dies are definitely from Samsung who has been NVIDIA's strategic memory partner for HPC-centric GPUs.

NVIDIA's Ampere GA100 GPU is a massive chip featuring 54 billion transistors. (Image Credits: EETimes via Videocardz)

Samsung has recently announced its HBM2E DRAM which features 16 Gb dies. Depending on the height of the stacks, NVIDIA could offer anywhere from 48 GB (4-hi) to all the way up to 96 GB (8-Hi) which is just insane amounts of VRAM compared to the existing Tesla V100 which maxes out at 32 GB. The HBM2E stacks also deliver increased speeds of up to 3.2 Gbps, allowing for up to 410 GB/s bandwidth or 2.5 TB/s bandwidth or even faster if NVIDIA decides to go for the 4.2 Gbps dies that will result in 3.2 TB/s bandwidth for the entire chip which is an amazing technical feat.

In terms of performance, the Ampere GA100 GPU delivers 1 Peta-OPs which is a 20x increase over the Volta GV100 GPU. The double-precision performance is rated at 2.5x higher over NVIDIA's Volta GV100 GPU which should end up somewhere around 20 TFLOPs FP64 since Volta features around 8 TFLOPs FP64 compute power. This would mean that the single-precision performance is rated at over 40 TFLOPs (FP32) which would be mind-blowing for the HPC segment.

NVIDIA Ampere GA100 GPU Based Tesla A100 Specs:

NVIDIA Tesla Graphics CardTesla K40
Tesla M40
Tesla P100
Tesla P100 (SXM2)Tesla V100 (SXM2)Tesla V100S (PCIe)NVIDIA A100 (SXM4)NVIDIA A100 (PCIe4)
GPUGK110 (Kepler)GM200 (Maxwell)GP100 (Pascal)GP100 (Pascal)GV100 (Volta)GV100 (Volta)GA100 (Ampere)GA100 (Ampere)
Process Node28nm28nm16nm16nm12nm12nm7nm7nm
Transistors7.1 Billion8 Billion15.3 Billion15.3 Billion21.1 Billion21.1 Billion54.2 Billion54.2 Billion
GPU Die Size551 mm2601 mm2610 mm2610 mm2815mm2815mm2826mm2826mm2
FP32 CUDA Cores Per SM192128646464646464
FP64 CUDA Cores / SM644323232323232
FP32 CUDA Cores28803072358435845120512069126912
FP64 CUDA Cores96096179217922560256034563456
Tensor CoresN/AN/AN/AN/A640640432432
Texture Units240192224224320320432432
Boost Clock875 MHz1114 MHz1329MHz1480 MHz1530 MHz1601 MHz1410 MHz1410 MHz
TOPs (DNN/AI)N/AN/AN/AN/A125 TOPs130 TOPs1248 TOPs
2496 TOPs with Sparsity
1248 TOPs
2496 TOPs with Sparsity
FP16 ComputeN/AN/A18.7 TFLOPs21.2 TFLOPs30.4 TFLOPs32.8 TFLOPs312 TFLOPs
624 TFLOPs with Sparsity
312 TFLOPs
624 TFLOPs with Sparsity
FP32 Compute5.04 TFLOPs6.8 TFLOPs10.0 TFLOPs10.6 TFLOPs15.7 TFLOPs16.4 TFLOPs156 TFLOPs
(19.5 TFLOPs standard)
156 TFLOPs
(19.5 TFLOPs standard)
FP64 Compute1.68 TFLOPs0.2 TFLOPs4.7 TFLOPs5.30 TFLOPs7.80 TFLOPs8.2 TFLOPs19.5 TFLOPs
(9.7 TFLOPs standard)
19.5 TFLOPs
(9.7 TFLOPs standard)
Memory Interface384-bit GDDR5384-bit GDDR54096-bit HBM24096-bit HBM24096-bit HBM24096-bit HBM26144-bit HBM2e6144-bit HBM2e
Memory Size12 GB GDDR5 @ 288 GB/s24 GB GDDR5 @ 288 GB/s16 GB HBM2 @ 732 GB/s
12 GB HBM2 @ 549 GB/s
16 GB HBM2 @ 732 GB/s16 GB HBM2 @ 900 GB/s16 GB HBM2 @ 1134 GB/sUp To 40 GB HBM2 @ 1.6 TB/s
Up To 80 GB HBM2 @ 1.6 TB/s
Up To 40 GB HBM2 @ 1.6 TB/s
Up To 80 GB HBM2 @ 2.0 TB/s
L2 Cache Size1536 KB3072 KB4096 KB4096 KB6144 KB6144 KB40960 KB40960 KB

NVIDIA's Ampere GA100 also features a new Tensor operation compute indicator known as Tensor Float 32 or TF32 which is based on the 3rd Generation Tensor Cores, offering higher AI/DNN core output. The Tensor cores also natively support double-precision compute which allows the GA100 GPU to hit a 2.5x performance increase over its predecessor. As of right now, nothing from the competition that has been announced comes close to this beast.

PCIe 5.0 12-Pin Power Cable Pictured, Reminiscent of NVIDIA’s Power Cable Architecture

The DGX-A100 - The First HPC System With 140 Peta-OPs Compute Shipping Now For $199,000

Finally, NVIDIA will be announcing its next-generation DGX-A100 system which Jensen Huang teased a few days ago. The DGX-A100 will deliver 5 Petaflops of peak performance with its six Ampere based Tesla A100 GPUs.

The system itself is 20x faster than the previous DGX based on NVIDIA's Volta GPU architecture. The reference cluster design features 140 DGX-A100 GPUs with a 200 Gbps Mellanox Infiniband interconnect. The whole system is going to start at $199,000 and is shipping as of today.