What is linux hpc cluster

What is High-performance computing (HPC)? [part 1]

High-Performance Computing is the procedure of combining computational resources together as a single resource. The combined resources are often referred to as a supercomputer or a compute cluster. The reason this is done is to make it possible to deliver computational intensity and the ability to process complex computational workloads and applications at high speeds and in parallel. Those workloads require computing power and performance that is often beyond the capabilities of a typical desktop computer or a workstation.

What are HPC clusters?

HPC clusters are often made of a group of servers; these servers are generally referred to as compute nodes. Some clusters can be as small as a few nodes and some have hundreds and even thousands of compute nodes all connected together over a network so they can work together to solve these advanced computational workloads. These networks sometimes use high-speed interconnects or other low latency network solutions to reduce computational latency. The workloads often have significant storage requirements, either in terms of data size or throughput. So, it’s common to deploy both a high-performance storage solution, often referred to as scratch used for in-flight computation storage, along with a general-purpose solution for user data, applications, and archival. To use all these resources as effectively as possible and in parallel, sometimes a message passing interface generally referred to as MPI is used.

The workloads generally run in batches and are managed by a batch scheduler. This scheduler is a vital component of HPC clusters, it keeps track of the available resources of a cluster, it queues workloads when there are not sufficient computational resources and when computational resources become available the scheduler efficiently assigns workloads to available resources.

HPC clusters and solutions are available anywhere and can be deployed on-premise, in the cloud, or as a hybrid solution combining both on-premise and off and even on the edge.

What are the main use cases of HPC?

HPC is used to solve some of the most advanced and toughest computational problems we have today. These problems exist in all types of environments, such as science, engineering, or business. Thanks to HPC, computational fluid dynamics (CFD) workloads can simulate the flow of fluids to solve problems in numerous fields such as aerodynamics, industrial system design or weather simulation, and many more. High-performance data analytics (HPDA) is the combination of HPC and Data Analytics where parallel processing and powerful analytics are used to analyse huge data sets at incredible speeds. This is used for anything from real-time data analysis, high-frequency stock trading, and even some highly complex analytics problems found in scientific research. Large computational clusters are also used to render whole movies or create visual effects for certain scenes. Genome processing and sequencing is another field that needs HPC due to the huge data sets that are analysed and interpreted to figure out hereditary conditions or other medical anomalies. HPC can even be used to figure out large-scale logistic and supply problems such as those existing in Retail. Whatever the need, HPC has the ability to solve it.

Summary

This blog has introduced you to high-performance computing, the components that make HPC clusters and how they are used to solve the toughest computational problems we have today. We have also covered some of the many ways high-performance computing can be used in the industry with a brief overview of computational fluid dynamics and high-performance data analytics.

If you are interested in more information take a look at how Scania is Mastering multi cloud for HPC systems with Juju, or dive into our blog on Data Centre Automation for HPC, Or explore some of our other HPC content.

In the next blog, we will be highlighting some of the many ways a HPC cluster can be deployed.

Источник

Getting started with High Performance Computing (HPC) in Red Hat Enterprise Linux 8

High Performance Computing (HPC) generally refers to processing data with complex calculations at high speeds. While historically, so-called “supercomputers” were single fast processors, current HPC uses often massive clusters of CPUs, aggregating computing power to deliver significantly higher performance than single desktops or servers for solving large numeric problems in engineering, science, and business. In an HPC cluster, each component computer is often referred to as a node.

Technical background

HPC clusters run batches of computations. The core of any HPC cluster is the scheduler, used to keep track of available resources, allowing job requests to be efficiently assigned to compute resources (CPU and GPU).

The most common way for an HPC job to use more than one cluster node is via the Message Passing Interface (MPI). MPI is a specification for the developers and users of message passing libraries. MPI allows you to launch a job running across the entire cluster with a single command, and it also provides application-level communication across the cluster.

MPI constitutes a standardized and portable message-passing system which consists of a library and a protocol to support parallel computing. MPI enables passing information between various nodes of a HPC cluster or between particular clusters, and has different implementations that provide the libraries to run HPC applications in a distributed manner across different physical nodes.

Red Hat provides the following MPI implementations:

• Open MPI — provided in the openmpi package; an open source and freely available implementation of both the MPI-1 and MPI-2 standards; combines technologies and resources from several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI)
• MPICH — provided in the mpich package; a high-performance and widely portable implementation of the MPI-1, MPI-2, and MPI-3 standards
• MVAPICH — provided in the mvapich package; an implementation of MPICH with Infiniband support

The openmpi, mpich and mvapich packages are included in Red Hat Enterprise Linux.

Red Hat recommends Open MPI as the MPI implementation for HPC clusters.

HPC systems often replace the MPI libraries that ship in RHEL with vendor specific versions that take full advantage of high-performance interconnects such as Infiniband.

High Performance Computing on Red Hat Enterprise Linux

The Red Hat HPC offering is a special use case that cost effectively addresses HPC clusters. It is based on standard RHEL Server components and uses standard installation and entitlement. To be eligible for HPC, the workload must be non-interactive and externally scheduled, usually bound by computational resources.

Two separate subscriptions are needed to build a RHEL HPC cluster:

• Head Nodes — control the entire cluster; used for management, job control, and launching jobs across the cluster
• Compute Nodes — perform the actual HPC calculations

An HPC cluster must include at least one Head Node and one Compute Node.

How to install head node and compute node

Previous versions of RHEL for HPC were built using subsets of RHEL Server. In RHEL 8, both head and compute nodes are based on the standard Red Hat Enterprise Linux Server software stack. HPC systems are identified at installation time by subscribing each system with a HPC Head Node or HPC Compute Node entitlement.

A head node is basically a Red Hat Enterprise Linux Server subscribed to one of the head node entitlements.
A compute node is basically a Red Hat Enterprise Linux Server subscribed to one of the compute node entitlements .

1. Download the ISO image files of the Red Hat Enterprise Linux Server 8 from the Red Hat Customer Portal.Download Red Hat Enterprise Linux Server. Use one of two basic types of installation media:
   • Binary DVD is a full installation image which can be used to boot the installation program and perform an entire installation without additional package repositories.
   • Boot ISO is a minimal boot image which can be used to boot the installation program. The boot ISO is smaller than the full installation image, but the installation from the boot ISO is a network-based installation that requires the creation of a local repository from which the software will be installed. For more information on working with yum repositories, see Configuring yum and yum repositories.

See Performing a standard RHEL installation for further information on the installation media.

Communication between head node and compute nodes

To run jobs on an HPC cluster, you need to install a scheduler or a manager for the HPC cluster. The RHEL for HPC offering does not include an HPC scheduler. You can choose a third-party scheduler, such as SLURM (open source), PBS Pro, Condor, MOAB, or Spectrum LSF. Install the scheduler of your choice, and configure the connectivity between the head node and the compute nodes according to the instructions of the scheduler.

Источник