AMD EPYC Milan With 7nm+ Zen 3 Cores and NVIDIA Volta-Next GPU With Greater Than 7 TFLOPs FP64 Compute To Power Perlmutter Supercomputer
It looks like both AMD and NVIDIA will be coming together to power the Perlmutter Exascale super-computer. Designed by CRAY, the supercomputer would pave the way to exascale computing but what’s most interesting about this supercomputer is the fact that it will utilize AMD’s and NVIDIA’s next-generation products under the hood.
AMD 7nm+ EPYC Milan CPUs and NVIDIA’s Volta-Next GPUs To Power The Exascale Perlmutter Supercomputer
The new supercomputer was discussed in a presentation by the US Department of Energy where it was stated that both next-generation AMD and NVIDIA hardware would be powering it. During the Supercomputing 2018 event, Cray displayed their Shasta Computer Blade server which will be used as the main platform inside the supercomputer. We can expect thousands of nodes inside the new supercomputer.
Starting off with the details, we are first looking at the entirely water-cooled chassis which will be able to house 8 of AMD’s next-generation EPYC Milan processors. The system is split into a dual-section design with one featuring cooper water blocks over the Milan CPUs and the other four CPUs on an inverted PCB, that is also water cooled. There are a total of 64 DIMM slots which are also water cooled.
Some little tidbits of the EPYC Milan CPUs were also mentioned with the highlight being them based on the new Zen 3 architecture that utilizes the 7nm+ process node from TSMC. There will also be support for AVX2 SIMD (256-bit) instructions while the core count is yet to be finalized so the company is currently using the maximum 64 core count of the upcoming EPYC Rome 7nm processors as a baseline for their platform. The CPUs will feature 8 channel memory for a total of 256 GB capacity per node. Each node will be using a 25 GB/s Slingshot connection, acting as the main interconnect for the system.
AMD CPU Roadmap (2018-2020):
|Ryzen Family||Ryzen 1000 Series||Ryzen 2000 Series||Ryzen 3000 Series||Ryzen 4000 Series|
|Architecture||Zen (1)||Zen (1) / Zen+||Zen (2)||Zen (3)|
|Process Node||14nm||14nm / 12nm||7nm||7nm+|
|High End Server (SP3)||EPYC 'Naples'||EPYC 'Naples'||EPYC 'Rome'||EPYC 'Milan'|
|Max Server Cores / Threads||32/64||32/64||64/128||TBD|
|High End Desktop (TR4)||Ryzen Threadripper 1000 Series||Ryzen Threadripper 2000 Series||Ryzen Threadripper 3000 Series (Castle Peak)||Ryzen Threadripper 4000 Series|
|Max HEDT Cores / Threads||16/32||32/64||32/64?||TBD|
|Mainstream Desktop (AM4)||Ryzen 1000 Series (Summit Ridge)||Ryzen 2000 Series (Pinnacle Ridge)||Ryzen 3000 Series (Matisse)||Ryzen 4000 Series (Vermeer)|
|Max Mainstream Cores / Threads||8/16||8/16||8/16?
|Budget APU (AM4)||N/A||Ryzen 2000 Series (Raven Ridge)||Ryzen 3000 Series (Picasso) Zen+?||Ryzen 4000 Series (Renior)|
Coming to the GPU side of things, the NVIDIA based Volta-Next GPUs will be offering the bulk of compute power on the supercomputer. Each node will contain 4 Volta-Next GPUs. The Volta-Next naming convention means that the GPU will be the successor to Volta and will offer greater than 7.0 TFLOPs of compute (GV100 currently does 7.5 TFLOPs), more than 32 GB of HBM2 VRAM and next-gen NVLINK for fast interconnect between the GPUs. Details are not known regarding the new NVIDIA HPC GPUs but we can expect them next year at GTC ’19. The GPUs are said to perform up to 4 times faster than the current Cori system.
Perlmutter, a pre-exascale system coming in 2020 to the DOE’s National Energy Research Scientific Computing Center (NERSC), will feature NVIDIA Tesla GPUs. The system is expected to deliver three times the computational power currently available on the Cori supercomputer at NERSC.
Tesla GPUs provide universal acceleration for scientific computing, AI and machine learning applications, reducing the time it takes to make groundbreaking discoveries. Research organizations large and small are recognizing this potential by turning to GPUs for next-generation supercomputers. via NVIDIA
Perlmutter will be using an all-flash filesystem with 4 TB/s of sustained bandwidth, 7,000,000 IOPS and 30 petabytes of usable capacity. It will be used for large-scale simulations and data analysis of experimental facilities. The delivery date for the supercomputer is expected sometime in 2020 so we can see some design and specs changes as we approach its launch schedule.