▷ the Bright blog

These words are being uttered around the globe today as the announcement went out that Rich Brueckner from insideHPC has passed away. Many called him a friend, including myself. An industry icon has gone, and we will mourn the loss for some time to come. Today, I reflect on the man and the icon.

In recent weeks, I’ve been blogging about some of the cool new features in Bright Cluster Manager 9.0, to share an insight into the depth and breadth of the latest version of our technology.

Typically, when an organization extends an on-premises cluster into Amazon Web Services (AWS) or Microsoft Azure, there isn’t the same storage mounted on-premises as there is in the cloud. If you have a very fast connection and if you would mount your on-premises storage on your cloud nodes, there wouldn’t be a problem, as all data would be available both on-premises and in the cloud. However, if you don't have a fast link, the input data for a job would need to be moved to the cloud, and the output data would eventually need to come back to the on-prem cluster. 

I just finished reading an article on The Next Platform – “The Outlook for Infrastructure is Cloudy – in a Good Way”, and feel compelled to comment, especially given my previous post on High Performance Clusters in the Cloud?

The development of advanced driver assistance systems (ADAS) - also known as autonomous driving (AD) - relies heavily on massive amounts of real-world training data that consists of data gathered over the course of millions of miles of test driving and thousands of concurrent simulations. To be successful, car manufacturers must be able to simultaneously ingest thousands of concurrent streams of data, apply artificial intelligence (AI) techniques to Petabyte size training data sets, and archive these datasets for decades to come.

Even though Bright Cluster Manager provides a fantastic monitoring environment for high performance computing, AI, data science, big data, OpenStack, machine and deep learning environments, it’s important to recognize that a Bright cluster is not an island in an organization's IT infrastructure. Many organizations will have infrastructure-wide monitoring and reporting environments in place, in addition to Bright’s capabilities. We also have to accept that there may be companies that wish to use monitoring tools other than Bright View to see monitoring data.

For the majority of our customers, the usual way of installing a Bright cluster is by downloading the Bright ISO, going through a head node installation, and then PXE booting your compute nodes to allow them to join the cluster.

Now, why would I want to do that? Cloud resources can be expensive; I have to depend on a third party for some of my most important applications; performance may suffer; my data ends up outside my organization…all valid concerns. Which likely explains why most HPC continues to be hosted in-house. But another question is, is it even an option to effectively leverage the cloud for clustered computing? How do I extend my on-premise cluster to cloud servers and still manage them as a single cluster?

Whether you are a Bright customer, a Bright reseller, or in the process of evaluating Bright, you will be aware that Bright Cluster Manager is great at monitoring your cluster. From the moment that you set up your cluster for the first time, Bright’s monitoring capabilities set to work, providing comprehensive metrics and alerts.

To extract the most value from your HPC cluster, you need to ensure that system resources are being properly utilized.  HPC system users are notorious for over-requesting resources for their jobs, resulting in idle or underutilized resources that could otherwise be doing work for other jobs.  While one reason for this can be users hoarding resources to ensure they have what they need, another common reason why users over request resources is that they simply don’t know what resources their jobs will need to complete the job in a specified time.  For administrators to ensure that their precious and expensive cluster resources aren’t being squandered, they need to get actionable details regarding how the resources are being used.  More specifically, they need to know things like which jobs are using which resources, which jobs aren’t using resources that they’ve provisioned and which users are repeatedly hoarding resources unnecessarily, as well as other things.