Detailed Analysis of Frontier and Supercomputer Fugaku

Introduction:

A supercomputer is a computer with superior performance to a general-purpose computer. A supercomputer’s performance is often measured in floating-point operations per second (FLOPS) rather than in million instructions per second (MIPS) (MIPS). Since 2017, there have been supercomputers that can do more than “1017 FLOPS (a hundred quadrillion FLOPS, 100 petaFLOPS or 100 PFLOPS).”[1] Supercomputers of today are capable of processing massive quantities of data in parallel by dividing up the computing work across thousands of CPUs, allowing them to do the computations at lightning speed.

Supercomputers can be found conducting mathematical computations, as well as gathering, compiling, classifying, and analyzing massive amounts of data, in research centers, government organizations, and corporations. They play an essential role in the area of computational science, and are used for a broad range of computationally expensive activities in diverse domains such as “quantum mechanics, weather forecasting, climate research, oil and gas exploration, molecular modeling (computing the structures and characteristics of chemical compounds, biological macromolecules, polymers, and crystals), and physical simulations (such as simulations of the early moments of the universe, airplane and spacecraft aerodynamics, the detonation of nuclear weapons, and nuclear fusion).”[1]

In this paper, I am going to analyze and compare the two topmost supercomputers listed in the TOP500 list [2] that rank the world’s most powerful computers in terms of raw processing power.

1. FRONTIER — HPE CRAY EX235A, AMD OPTIMIZED 3RD GENERATION EPYC 64C 2GHZ, AMD INSTINCT MI250X, SLINGSHOT-11

Overview:

On May 30, 2022, Hewlett Packard Enterprise reported that Hewlett Packard Enterprise Frontier, also known as OLCF-5, a brand-new supercomputer it created for the U.S. Department of Energy’s Oak Ridge National Laboratory (ORNL), had attained 1.1 exaflops. This made it the first supercomputer to break the exascale speed barrier and the fastest supercomputer in the world, according to the Top500 ranking of the world’s most powerful supercomputers. With a performance of 6.88 exaflops, Frontier also took first place in the mixed-precision computing category, which assesses efficiency in artificial intelligence-related file formats. With 52.23 gigaflops of performance per watt, the new supercomputer also secured the top place on the Green500 list as the most energy-efficient supercomputer in the world. [3]

2. SUPERCOMPUTER FUGAKU — SUPERCOMPUTER FUGAKU, A64FX 48C 2.2GHZ, TOFU INTERCONNECT D

Overview:

Fugaku (Japanese: 富岳), a petascale supercomputer at the Riken Center for Computational Science in Kobe, Japan, is currently listed as the second-best supercomputer on the TOP500 list [2]. It began development in 2014 as the K computer’s replacement and debuted in 2020. The name is derived from Mount Fuji’s alternate name and it was created in collaboration with RIKEN and Fujitsu. In the June 2020 TOP500 ranking, it became the world’s fastest supercomputer, as well as the first ARM architecture-based computer to do so. It also reached 1.42 exaFLOPS utilizing the mixed fp16/fp64 precision HPL-AI benchmark at the time. In 2021, it began regular operations. Frontier surpassed Fugaku as the world’s fastest supercomputer in May 2022. [5]

COMPARISON BETWEEN THE TWO SUPERCOMPUTERS:

A. Architecture:

FRONTIER: The following cutting-edge technologies were incorporated into Frontier’s design and construction by HPE, provided through its HPE Cray EX supercomputers:

1. 74 HPE Cray EX cabinets, each weighing over 8,000 pounds. A total of 9,408 CPUs and 37,632 GPUs are present in the system, with each node having one optimized 3rd Gen AMD EPYC™ processor and four AMD Instinct™ MI250x accelerators.

2. HPE Slingshot networking cables totaling 90 miles supply the world’s first high-performance Ethernet fabric built for next-generation HPC and AI systems. Larger, data-intensive workloads are among them, as are demands for increased speed and congestion control to ensure that applications operate smoothly and perform better.

3. Frontier’s Orion storage system can offer 700 petabytes of storage capacity, peak write rates of more than 35 terabytes per second, and more than 15 billion random-read input/output operations per second thanks to the Cray Clusterstor E1000 storage system.

SUPERCOMPUTER FUGAKU: The A64FX (Figure 3) was developed as a processor for the supercomputer Fugaku which features 158,976 processors.[8] Two nodes (two CPUs) are installed on a board known as the CPU Memory Unit (CMU), and one CPU constitutes one node. A “Bunch of Blades (BoB)” consists of eight CMUs, i.e., 16 nodes per BoB. Each Shelf contains 48 nodes since a Shelf is made up of three BoBs. In a computer rack, there are eight shelves (384 nodes) (some racks have 192 nodes). Fugaku is composed of 432 racks, 396 racks with 384 nodes, and 36 racks with 192 nodes. This brings the total number of nodes to 158,976. [9]

Figure 4 depicts the A64FX CPU’s block diagram. A ring bus network on a chip (NoC) with the Tofu Interconnect D (TofuD) interface and PCI Express (PCIe) interface is used to link the four core memory groups (CMGs), each of which is made up of 13 cores (12 of which are employed as computational cores and one serves as an assistance core), level 2 cache, and memory controller. The Arm architecture was used in the development of the A64FX to have the supercomputer Fugaku accepted by a larger variety of software developers than the K computer, which had a full-scale launch in 2012, and putting in place an environment that allows the use of the latest software.

Built on the most recent HPE Cray EX235a architecture, Frontier features a total of 8,730,112 cores and is powered by AMD EPYC 64C 2GHz processors while with 7,630,848 cores, Fugaku has become the second-most powerful supercomputer. [7]

B. Computational capability:

FRONTIER: A Rmax of 1.102 exaFLOPS, or 1.102 quintillion operations per second, was attained by Frontier. The supercomputer is the most efficient supercomputer on the Green500 list, with a calculated efficiency of 62.68 gigaflops/watt. [1] According to the High-Performance Linpack Benchmark test, Frontier’s total performance of 1.1 exaflops corresponds to more than one quintillion floating point operations per second or flops. Each flop symbolizes a different type of computation, such as addition, subtraction, multiplication, or division. With 148.6 petaflops, Frontier’s first performance on the Linpack benchmark is more than seven times more than Summit’s. The High-Capability Linpack-Accelerator Introspection, or HPL-AI, test assessed Frontier’s mixed-precision computing performance at around 6.88 exaflops, or more than 6.8 quintillion flops per second. The HPL-AI test assesses computation rates in the computing formats frequently employed by the machine-learning techniques that fuel improvements in artificial intelligence. [10]

FUGAKU: In the FP64 high-performance LINPACK test, which is utilized by the TOP500, Fugaku’s first performance was reported as having a Rmax of 416 petaFLOPS. Fugaku’s performance reached a Rmax of 442 petaFLOPS after the November 2020 increase in processor count. Fugaku also ranked first in several rankings that evaluate computers on various workloads in 2020, including the Graph500, HPL-AI, and HPCG benchmarks. Never before has a supercomputer held the top spot in each of the four rankings simultaneously. Fugaku improved its performance on the new mixed-precision HPC-AI benchmark to 2.0 exaflops as of November 2020, surpassing its previous 1.4 exaflops milestone. This improvement was made possible via a hardware upgrade. These are the first benchmark readings exceeding one exaflop at any level of precision on any kind of hardware. While the number of Arm A64FX cores was only raised by 4.5% to 7,630,848, observed performance on that test improved much more (although the system does not employ other computational resources, such as GPUs), and slightly more on TOP500, or by 6.4%, to 442 petaflops. With 16.0 HPCG-petaflops, it outperforms Summit by 5.4 times on the HPCG benchmark. [5]

As the first “real” exascale supercomputer, Frontier is a conventional supercomputer — i.e., a computing device not based on quantum computing principles — with a performance capacity of more than one exaflop. While Fugaku in Japan was touted as the first exascale supercomputer, Frontier appears to be able to use its peak 1.1 exaflop processing capacity for operations that aren’t merely for benchmarks (that is in real-world tasks).

C. Forms of parallelism:

FRONTIER: With Frontier’s split Lustre/ClusterStor system, fundamental data storage processing is separated from server and target nodes for metadata storage, flash-based data storage, and capacity disk-based storage, as well as router nodes, so that data referencing or transferring compute operations. This allows the entire distributed structure to function in parallel and at high speed. Frontier requires such a complicated multi-component system to keep its compute nodes supplied with the data they require and remove (write) the data they create without experiencing bottlenecks that cause IO delays to freeze cores. Also, Frontier uses the Cray Slingshot network with dragonfly topology. A Dragonfly group is a collection of endpoints linked to switches that are all-to-all connected, that is every group is linked to at least one other group (as shown in Fig 5). [11] Also, “the GPUs hide between 1,000 and 10,000-way concurrency inside their pipelines so the users don’t have to think about as much parallelism.” [12]

FUGAKU: For the supercomputer Fugaku, the A64FX was created as a processor. “As external input/output interfaces, the A64FX has a TofuD for realizing a massively parallel system by mutually connecting CPUs and a PCIe bus for connecting I/O devices.” [8] The TofuD interconnects up to 10 CPUs with a bandwidth of 6.8 GB/s using 20 lanes of high-speed serial communications with a 28-Gbps transmission speed. 16 high-speed serial communication lanes with an 8-Gbps transmission rate and a 16 GB/s bandwidth make up the PCIe. TofuD’s block diagram is shown in Figure 6. The TofuD features six network interfaces known as TNIs that connect to ten CPUs via a network router. [8]

D. Power Requirements:

FRONTIER: “Frontier consumes 21 MW (compared to its predecessor Summit’s 13 MW); it has been estimated that Frontier’s successor, Aurora, will consume around 60 MW.” [4] With an efficiency of 52.23 gigaflops per watt, the HPE/AMD system offers 1.102 Linpack exaflops of computational power in a 21.1-megawatt power envelope.

FUGAKU: At peak performance, Fugaku consumes a total of 28MW. Because Fugaku is extremely energy efficient, its power usage is just twice that of the K computer while providing computational power 40 times quicker. The cooling system was increased following the fact that the heat generated by each computer rack of the Fugaku computer is significantly more than that of the K computer. The Fugaku computer has a water-to-air cooling ratio of 9:1, whereas the K computer has a ratio of 2:1. [13]

E. Typical Applications:

FRONTIER: Frontier’s unmatched computing capacity and capabilities will be utilized to accelerate significant scientific advancement in a variety of domains, from improving human understanding of fundamental sciences like particle physics and cosmology to broadening social benefits through the creation of more energy-efficient technologies. The groundbreaking Frontier Supercomputer will do computations up to 10 times quicker than the most advanced supercomputers available today. Frontier users can simulate a nuclear reactor’s whole lifetime, discover disease genetics, and draw on current scientific and technological advances to better combine artificial intelligence with data analytics, modeling, and simulation. C, C++, Fortran, CrayMPI, AMD ROCm, AMD HIP, Cray software toolkit, OpenSHMEM, and numerous libraries like BLAS, LAPACK, ScaLAPACK, Trilinos, Qt, X11, Motif, and others are supported by Frontier

FUGAKU: Matsuoka, in his capacity as head of Riken’s Center for Computational Science, and his group have identified nine societally significant application areas for Fugaku to focus on. These fields include medical, pharmacology, disaster prediction and prevention, environmental sustainability, and energy. [15] The use of Fugaku, for instance, helped researchers investigate COVID-19 therapy options and show how droplets that may transmit the coronavirus from the mouth. [14] Java, C++11, C++14, Fortran 2008, Python, and Ruby are supported by it. Additionally, BLAS, LAPACK, ScaLAPACK, and other libraries are supported.

TABULAR COMPARISON:

Tabular comparison between Frontier and Fugaku

Images:

Fig 1: The Exascale-class HPE Cray EX Supercomputer at Oak Ridge National Laboratory [4]

Fig 2: PRIMEHPC FX1000 (Fugaku node) at SC19 [5]

Fig 3: Die photo of A64FX CPU. [8]

Fig 4: Block diagram of A64FX CPU [8]

Fig 5: Dragonfly topology [11]

Fig 6: Block diagram of TofuD [8]

References:

[1] https://en.wikipedia.org/wiki/Supercomputer

[2] https://top500.org/

[3] https://www.hpe.com/us/en/newsroom/press-release/2022/05/hewlett-packard-enterprise-ushers-in-new-era-with-worlds-first-and-fastest-exascale-supercomputer-frontier-for-the-us-department-of-energys-oak-ridge-national-laboratory.html

[4] https://en.wikipedia.org/wiki/Frontier_(supercomputer)

[5] https://en.wikipedia.org/wiki/Fugaku_(supercomputer)

[6] https://www.hpe.com/us/en/newsroom/press-release/2022/05/hewlett-packard-enterprise-ushers-in-new-era-with-worlds-first-and-fastest-exascale-supercomputer-frontier-for-the-us-department-of-energys-oak-ridge-national-laboratory.html

[7] https://www.thehindu.com/sci-tech/technology/frontier-dethrones-fugaku-as-the-worlds-fastest-supercomputer/article65480206.ece

[8] https://www.fujitsu.com/global/documents/about/resources/publications/technicalreview/2020-03/article03.pdf

[9] https://www.r-ccs.riken.jp/en/fugaku/about/

[10] https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier

[11] https://olcf.ornl.gov/wp-content/uploads/Frontiers-Architecture-Frontier-Training-Series-final.pdf

[12] https://smc.ornl.gov/wp-content/uploads/2019/09/Geist-presentation-2019.pdf

[13] https://www.r-ccs.riken.jp/en/fugaku/facility/

[14] https://english.kyodonews.net/news/2022/05/aa38efc4fb93-japans-fugaku-loses-title-as-worlds-fastest-supercomputer.html

[15] https://www.japantimes.co.jp/news/2021/01/07/business/tech/japans-fugaku-supercomputer/