Visualizing one or more GPU metrics using a graph is very easy with Bright Cluster Manager — you just select a GPU metric from the GPU section in the metrics tree on the left and drag it into an empty graph area on the right. You can also dynamically zoom into an area of the graph by dragging a window over that area with your mouse.

GPU Management

Bright Cluster Manager allows for alerts and actions to be triggered automatically when GPU metric thresholds are exceeded. A configuration wizard is available to guide you through the steps of defining a rule.

Bright Cluster Manager allows for alerts and actions to be triggered automatically when GPU metric thresholds are exceeded. Such rules are completely configurable to suit your requirements, and any built-in cluster management command, Linux command, or shell script can be used as an action.

For example, if you would like to automatically receive an email and shut down a GPU node when its GPU temperature exceeds a set value, this can easily be configured in Bright Cluster Manager.

Cluster Health Management for GPU Clusters

Bright Cluster Manager allows for alerts and actions to be triggered automatically when GPU metric thresholds are exceeded. A configuration wizard is available to guide you through the steps of defining a rule.

Cluster Health Management can also include health checks for GPU cards and GPU Computing Systems in GPU clusters. Any of the supported GPU metrics can be used in regular and prejob health checks.

For example, you could configure a prejob health check called "AllFansRunning" and define an appropriate action when the health check has status FAIL. The screenshot of the Rackview on the right shows that this indicator for GPU Unit 41 has status FAIL.

Supported NVIDIA GPU Cards and Computing Systems

The table below shows which metrics are available on various NVIDIA, NextIO and Dell GPU cards and units*:

GPU Metric	M2050 M2070 M2090*	S2050	NextIO vCORE Express	NextIO vCORE Express Extreme	Dell C410x
GPU temperatures	-
GPU compute mode
GPU display mode	n/a	n/a	n/a	n/a	n/a
GPU persistence mode
GPU exclusivity mode
GPU fan speeds	n/a	n/a	n/a	n/a	n/a
Unit fan speeds	n/a
Unit/Device serial
Unit firmware version	-
Unit temperature	n/a
Unit power usage	n/a
Voltages and currents	n/a
Unit LED states	n/a
GPU ECC statistics
Board serial number
GPU utilization
Driver version	-	-	-	-	-
Memory utilization
PCI information

Read more about the NVIDIA Tesla GPU models on the NVIDIA website.
Read more about the NextIO GPU models on the NextIO website.
Read more about the Dell GPU models on the Dell website.