THE BRIGHT BLOG

Auto-scaling HPC and Kubernetes

Bright CEO Bill Wagner’s recent blog post “The Convergence of HPC and A.I.” pointed out that creating HPC and Kubernetes silos within a shared HPC infrastructure is more like “coexistence than convergence”. The solution I will be highlighting  is Bright Auto-scaler, which automatically resizes HPC and Kubernetes clusters (workload engines) according to workload demand and configured policies. This post describes how to configure Bright Auto-scaler to achieve true convergence of HPC and A.I. through a scenario.

Each year we ask our customers to participate in a survey that helps us understand how they use and benefit from Bright Cluster Manager. We ask them a series of questions about the technologies they use, and how they use Bright in their day-to-day operations. And some of you had some very interesting things to say.

One of Bright Cluster Manager’s most popular features is its built-in monitoring system. It is lightweight and efficient, and it works right out of the box. But what people generally don’t know is that they can use Bright to monitor non-Bright nodes; nodes that were neither provisioned by, nor managed by, Bright. The Bright Lightweight CMDaemon can be used to monitor and healthcheck auxiliary servers that support the cluster but aren’t part of it, for example, database servers, authentication servers, and file servers.

Data center managers are continually under pressure to deliver more results from their existing computing resources. One strategy that has proven successful is the aggregation of what have traditionally been computing silos into a modern, and efficient shared compute cluster.

If you’re working in deep learning using NVIDIA GPUs, you may have heard about the NVIDIA GPU Cloud or NGC for short. NGC provides pre-integrated GPU-accelerated containers that you can use power your own AI projects. 

I am delighted to share news that Bright has launched an online training portal, exclusively for our customers and partners.

There are times when you need to customize the operating system that servers in your cluster are running in order to enable some special capability. One example is enabling a server to take advantage of accelerator or GPU hardware by installing a kernel module. Bright does this automatically for some of the most popular devices, like NVIDIA GPUs, but you may have other needs. For example, kernel modules often are required to control particular storage, network, or other devices.

Carrying out such low-level modifications on a server can be risky. If done incorrectly, kernel modifications could cause the server to fail. The risk is compounded in a clustered environment since different servers may need different kernel modules to load every time they are restarted.

Fortunately, Bright Cluster Manager can minimize the risk out of installing kernel modules, and automate the process so that the cluster’s operation remains consistent.

Here’s how it’s done: