High-performance scientific computing has infinite potential, but power consumption can put a strain on finite resources. How could CINECA maximize computing power while keeping costs under control?
CINECA worked with partner E4 Computer Engineering to implement state-of-the-art, energy-efficient IBM Power Systems servers as the foundation for its new supercomputing cluster, D.A.V.I.D.E.
2.5 timesfaster application runtime enables scientists to run bigger, more complex jobs
20%cut in power consumption reduces energy costs and helps lower carbon emissions
Cost savingscan be reinvested in research activities
Business challenge story
Finding the right balance
Scientists specializing in subjects as diverse as nuclear physics and nanotechnology all have one thing in common: the need for high-performance computing (HPC) resources for data analysis and modeling. As scientific research continues to advance and data sets get larger, demand for computing power is growing rapidly.
CINECA, a consortium of 70 Italian universities, eight research institutions and the Italian Ministry of Higher Education, provides scientists with the HPC resources they need to support their research. As the operator of the largest supercomputing center in Italy, CINECA knows all too well the challenges this brings.
Carlo Cavazzoni, manager for HPC Development in the HPC Business Unit at CINECA, begins: “To meet demand for more computing power, we refresh our HPC infrastructure every few years. However, as performance increases, so too does power consumption—which drives up energy costs.”
CINECA currently has several highly specialized HPC clusters in place, requiring a power budget in excess of 3 MW. This has serious implications from both an economic and an environmental perspective, and greater efficiencies would need to be found for CINECA to keep providing HPC resources to scientists.
CINECA, along with partner organizations E4 Computer Engineering and the University of Bologna, and within a PRACE (Partnership for Advanced Computing in Europe) project, set out to develop a cost-effective HPC solution that would deliver the processing power that researchers need while keeping energy consumption as low as possible.
Carlo Cavazzoni explains: “In recent years, improvements in microprocessor and board design have helped to make HPC more energy-efficient. But running a supercomputer while keeping power consumption manageable is still a huge challenge—and one that CINECA and PRACE were determined to take on.”
Investing in the best
To give researchers the computing power they crave while keeping costs under control, PRACE decided to build a new, energy-efficient cluster based on IBM Power Systems technology and deploy the cluster at CINECA. Together with E4 Computer Engineering, PRACE and CINECA selected the state-of-the-art IBM Power Systems S822LC for High Performance Computing as the foundation for the cluster. The teams worked to augment the Power Systems servers with custom-designed components for monitoring and capping the power consumption and applying liquid cooling to the processors and the GPUs. Next, they converted the servers to an Open Connectivity Foundation form, enabling applications to run effectively on the customized hardware.
Cosimo Gianfreda, CTO of E4 Computer Engineering, elaborates: “IBM Power Systems S822LC proved to be a very reliable, developer-friendly server. We chose it because the architecture had the horsepower and versatility we needed for such an ambitious project. Another key deciding factor was the strong support services that IBM and the OpenPOWER foundation offered, which we felt would be valuable when working on such a complex project.”
CINECA’s new IBM Power Systems-based cluster, aptly named D.A.V.I.D.E. (Development of an Added Value Infrastructure Designed in Europe), is an energy-efficient, petaflop-class HPC cluster. Soon after installation, D.A.V.I.D.E. was ranked in the TOP500 and Green500 lists of the most powerful and most energy-efficient supercomputers in the world.
“We are very proud of what we have achieved with D.A.V.I.D.E.,” says Cosimo Gianfreda. “None of this would have been possible without IBM’s support. The implementation went very smoothly – the first node of the cluster was up and running in a matter of hours. Scientists are already using the D.A.V.I.D.E. cluster to support research projects and there are plans for further deployments in the future.”
Delivering higher performance with less energy
With IBM Power Systems S822LC technology, CINECA has honed the D.A.V.I.D.E. system into an energy-efficient computing powerhouse, running applications 2.5 times faster than its existing HPC cluster, and consuming 20 percent less power on average while doing so.
“The efficiency of the D.A.V.I.D.E. system is revolutionary,” says Cosimo Gianfreda. “We believe we have achieved a major milestone for HPC. The energy savings speak for themselves. We’re also proud to say we have had a lot of interest from the scientific community, who have put these new efficiencies to good use in some really exciting computational projects.”
To date, CINECA has devoted D.A.V.I.D.E. to four groundbreaking scientific communities, which are taking advantage of the system’s high performance to further their research. One community is currently running open-source Quantum ESPRESSO software on the D.A.V.I.D.E. cluster to model materials at the nanoscale. The NEMO (Nucleus for European Modelling of the Ocean) community is using it to power oceanic modeling systems, while SPECFEM3D and BQCD (Berlin Quantum ChromoDynamics) are using D.A.V.I.D.E. to perform complex calculations relating to seismic activity and quantum field theory respectively.
In addition to offering scientists high levels of performance and processing power, E4 and University of Bologna in collaboration with CINECA have also developed a technology to monitor and cap power consumption on a data center-wide level. “We determined that the efficiency of the D.A.V.I.D.E. system could be improved even further,” says Andrea Bartolini, University of Bologna. “So, we developed a tool that enables granular management of the system’s energy consumption, and allowed managers to set a hard limit to keep within power restrictions.”
The success of the D.A.V.I.D.E. cluster has given PRACE and CINECA new confidence in the future of energy-efficient supercomputing. The organizations believe that the success of D.A.V.I.D.E. can translate into an exascale-level computing system, capable of making one billion billion calculations per second (1018, floating point operations per second).
Carlo Cavazzoni concludes: “With D.A.V.I.D.E., we believe a major milestone has been reached. We have shown the path to exascale computing can be achieved through tight co-design from leading-edge institutions and working with world-class technology like IBM Power Systems. D.A.V.I.D.E. is currently in production mode at CINECA and, once we have gathered enough statistics and data points on usage, performance and energy efficiency, we will use that information to design a system that will ultimately scale to exascale. With that kind of power behind them, there’s no telling what our scientists can achieve.”
CINECA is a non-profit consortium, made up of 70 Italian universities, eight research institutions and the Italian Ministry of Higher Education. As Italy’s largest computing center, CINECA acts as a link between academia and industry, providing scientists with easy access to high-performance scientific computing resources to help further vital research. Based in Bologna, CINECA employs around 700 people.
Take the next step
To learn more about IBM Power Systems, please contact your IBM representative or IBM Business Partner, or visit the following website: ibm.com/power
E4 Computer Engineering is an international IT engineering firm, with a goal of getting the best from new innovations in technology. For more information, visit: e4company.com
PRACE is a European partnership designed to enable high-impact scientific discovery, and engineering research and development by offering world-class computing and data management resources and services. For more information, visit: prace-ri.eu
The University of Bologna, founded in 1088, is considered the oldest university in the western world. It currently has over 80,000 students, and offers 210 degree programs across its five campuses in Bologna, Cesena, Forli, Ravenna and Rimini. For more information, visit: unibo.it