Deep dive into the building of the LUMI data center
Posted by Julia Werner •
All-AMD LUMI Becomes Europe's Most Powerful Supercomputer
EuroHPC 's Joint Undertaking recently deployed Europe's first pre-exascale supercomputer, LUMI, in Finland's city of Kajaani. The all-AMD build ranks as the 3rd fastest computer according to the Top500 rankings owing to its 550 Petaflops performance. It also claims the title of the world's third-greenest supercomputer, thanks to the impressive energy efficiency of the Epyc CPUs and CDNA-based MI250X GPUs. According to LUMI, achieving this performance with the world's fastest laptops would require a 23 km-high pile of them - roughly 28 times as high as the world's tallest building, Dubai's Burj Khalifa.
LUMI was engineered to derive 100 percent of its energy from renewable sources. It can be fed up to 200 MW of energy through hydropower, while the supercomputer's waste heat will be repurposed towards the Kajaani district, whose atmospheric temperatures can reach -18 Cº in winter. It's a circular energy consumption topology and opens the doors to truly environmentally-conscious, carbon-negative supercomputing.
LUMI, the result of a 10-country consortium, aims to provide researchers in Europe and across the world with a scalable platform that can supercharge climate-forward computations, medical and artificial intelligence research, and even quantum computing simulations. The underlying software is powered by AMD's ROCm software stack, which stands in contrast to NVIDIA's CUDA thanks to its open-source approach. It also features a translation mechanism that can adapt CUDA-based code to AMD's software stack with minimal software engineering efforts.
LUMI's architecture features a star-like topology with with eight interlinked nodes. (Image credit: LUMI)
LUMI's architecture features a star-like topology, with eight partitioned nodes converging through Cray's Slingshot high-speed interconnect (200 Gb/s). One of those, LUMI-G, is purely GPU-accelerated, with no less than 10,240 AMD Instinct MI250X GPUs distributed throughout 2,560 nodes (each node featuring four GPUs and a single, 64-core AMD Epyc 7763 "Trento" CPU), with four 200 Gb/s links between GPU nodes. Each AMD Instinct MI250X features 128 GB of HBM3 VRAM (3,2 TB/s bandwidth), and is rated at 42.2 TFLOP/s of performance in HPL benchmarks. All in all, AMD's GPUs offer 432,128 TFLOP/s performance and over 1 TB of addressable VRAM.
The LUMI-C partion features CPU-based acceleration, deploying 1,536 dual-socket, 64-core AMD Trento CPUs (totaling 196,608 AMD Zen 3 cores). At the same time, another partition is dedicated to in-process workloads, offering a total of 32 TB of RAM that can fit immense datasets. As for data storage, LUMI adopted a hybrid topology: hot data is kept in an 8 PB (PetaByte) all-flash Lustre system. A total of 80 PB of classical, HDD-based storage flows through Lustre. A cloud-enabled node is responsible for keeping LUMI's power at the fingertips of any researcher across the world that features an internet connection.
Plans for LUMI's expansion include LUMI-Q, a node fully dedicated to the emerging frontier of quantum computing. LUMI sees the future as quantum-classical, building upon a hybrid HPC (High-Performance Computing)+ QC (Quantum Computing) as the place where real-world applications will find their quantum advantage. While this node still hasn't been locally installed, LUMI has so far been linked successfully with two quantum computers: the Swedish Chalmers/Wallenberg Centre for Quantum Technology QAL 900, and Finland's first quantum computer, the Helmi, operated by VTT.
However, LUMI's deployment has been affected by the ongoing semiconductor crush, and the system has been added to in phases throughout this year. The pilot launch of the system is expected in August 2022, with its supercomputing prowess opening up for general availability in September of this year. The supercomputer's footprint will take up almost 400m2 - about the size of two tennis courts. It also belongs to the heavyweight category, with around 150,000 kilograms (150 metric tons) of computing power.
LUMI stands not only as Europe's crown jewel for supercomputing but is also a badge-bearer for AMD's increasingly relevant HPC chops and climate-conscious architectures, and its design stands as a testament to humanity's collaborative ingenuity.
The LUMI Supercomputer
LUMI (Large Unified Modern Infrastructure) is the first of three pre-exascale supercomputers built to ensure that Europe is among the world leaders in computing capacity. Norway, through Sigma2, owns part of the LUMI supercomputer which is funded by the EU and consortium countries.
For Norwegian research and business, it is important to have access to world-class computing power. This technology can contribute to solving major societal challenges, in disciplines such as climate, energy, environment and health.
Read more about the system and who can use it below.
Deep dive into the building of the LUMI data center
The data center for LUMI at Renforsin Ranta business park in Kajaani, Finland, has undergone a complete transformation from an empty former paper mill hall to one of the world’s most ecological data centers, and will soon be ready to welcome LUMI. Despite the complexity of the process and the global pandemic, the data center has been completed on schedule.
Building the premises in a middle of a global pandemic has in many ways been a non-trivial process. The data center project started in June 2019 when the LUMI consortium was selected to host one of the EuroHPC’s flagship supercomputers. The physical building process at the data center began in February 2020 after the tendering rounds for a dozen of sub-projects were completed. Now in spring 2021, the data center is almost complete in its commissioning phase.
– We have built a world-class data center for a world-class supercomputer. On top of that, this data center is one of the world’s most ecological places for supercomputers: LUMI, run with 100% hydropower and with efficient waste heat management, among other benefits, gives it a zero or even a negative carbon footprint, says Jukka-Pekka Partanen, Director for Kajaani data center program at CSC.
Back in 2019, LUMI’s risk management list predicting potential issues that may hinder the process didn’t include one word: pandemic.
– Despite the tight schedule, a large project with a workforce of a dozen contractors plus CSC employees, and the pandemic, we are now in the final stages of the data center construction process. LUMI will be here in no time, Partanen continues.
A glimpse of the LUMI white space. Jukka-Pekka Partanen, Director for Kajaani data center program at CSC, in front of the white space. Image: Juha Torvinen, CSC
Green, greener, LUMI data center
LUMI is built on world-class environmental sustainability and cost-efficiency. It helps the European ICT sector become greener and more cost-efficient, which is necessary to reach the EU’s ambitious climate targets and pave the way for the green transition.
LUMI will be furnished with liquid circulation cooling. The supercomputer’s waste heat will account for about 20 percent of the district heating in the city of Kajaani and will substantially reduce the entire city’s carbon footprint. Loiste Lämpö has been constructing the district heating connection for the project, and the company will also take care of the utilization of waste heat the computer produces.
The need for cooling is also reduced by the fact that the outdoor temperature and, as a result, the building’s thermal stress is much lower in Kajaani than what it would be in Southern Europe, for instance. The brownfield building solution also decreases LUMI’s carbon footprint.
LUMI’s energy consumption will be covered with power produced 100% with hydroelectricity supplied by Vattenfall.
– Reducing CO 2 emissions is a globally critical target. In addition to its other Green Deal objectives, the EU aims to make European data centers climate neutral by 2030. At CSC’s data center, there is no need to wait that long as this objective has already been reached. In terms of sustainability, Kajaani is an ideal place for supercomputers. Additionally, the power grid at the data center is very reliable, and electricity costs are extremely low. The data center is also a crucial part of the high-speed Nordic backbone network, Partanen explains.
Not your average data center
The former paper mill hall has undergone a complete makeover in just over a year: the technical space, including, e.g., cabling, cooling systems, power distribution, and excess heat utilization system, is located under the white space where the HPE (Hewlett Packard Enterprise) Cray EX racks will be located. Moreover, there will be a center for visitors with a balcony for viewing the entire white space from above.
In addition to technical and structural transformations, attention has also been paid to the visual appearance of the data center. LUMI, meaning snow in Finnish, will get a unique shield for the white space resembling snowflakes or a snow blanket. The structure is made of perforated aluminum composite panels which will also be illuminated.
– Inspiration for the design came from the Finnish meaning of the word lumi and also from the northern latitude of the data center. The goal of the project was to create a functional shield for the supercomputer and to wow the visitors of the data center, explains Mikael Haasmaa, CEO of Synopsis Architects Ltd., the principal designer in the LUMI design project.
The aluminium structures were designed with the help of Geometria Architecture Ltd architects using 3D modelling and computational design to help the manufacturing of the complex design of the shield.
Image of how the LUMI data center white space shield will look like. Copyright: Synopsis Architects Ltd. and Geometria Architecture Ltd.
In total, the building process of LUMI has employed around 100 people working in multiple companies.
Now the data center is ready to welcome the LUMI supercomputer to the premises to start its duty to as a world-leading scientific instrument for European scientists.
– We want to thank all involved parties in this data center project: contractors, the LUMI consortium and all employees involved in the project, not forgetting the great collaboration with EuroHPC JU. The data center is now complete; yet this is just the beginning for LUMI! Furthermore, we are looking forward to other research infrastructures to see the eco- and cost-efficiency of the Kajaani data center, and choose it as their future location – we would have space and power left for at least 20 installations of similar capacity, says Dr. Pekka Manninen, the director of LUMI Leadership Computing Facility.
See the making of videos from the LUMI data center:
Part 1:
Part 2:
Facts: LUMI will not be alone at Renforsin Ranta business park: CSC’s national supercomputers reside also in the business park, and additionally, there are approximately 40 companies representing various industries. The business park employs more people than the paper mill did at the end of its operations. LUMI will be one of the world’s fastest computer systems, having theoretical computing power of more than 550 petaflops which means 550 quintillion calculations per second. LUMI will also be one of the world’s leading platforms for artificial intelligence. LUMI itself will occupy 150m2 of space, thanks to HPE’s high-density hardware installations, which is about the size of a tennis court and it weighs around 150,000 kg. LUMI is an investment of over 200 million euros, covering the whole lifecycle of the system. The CSC Kajaani data center is highly scalable for large hardware installations and extensions or other potential infrastructure, making it ideal for building a sustainable data center ecosystem. The reliable and fast data communications networks of the data center are also designed for HPC. In addition, the area has a unique focus on data analytics thanks to the Kajaani University of Applied Sciences and numerous ICT companies. Illustration of the HPE Cray EX cabinets. Copyright: Hewlett Packard Enterprise
Author: Anni Jakobsson, CSC – IT Center for Science, Finland
Tagged: