One of Bright Cluster Manager’s most popular features is its built-in monitoring system. It is lightweight and efficient, and it works right out of the box. But what people generally don’t know is that they can use Bright to monitor non-Bright nodes; nodes that were neither provisioned by, nor managed by, Bright. The Bright Lightweight CMDaemon can be used to monitor and healthcheck auxiliary servers that support the cluster but aren’t part of it, for example, database servers, authentication servers, and file servers.
As its name suggests, the Lightweight CMDaemon is a lightweight implementation of the Bright CMDaemon. It is written in Python and supports Python 2 (in Bright 8.2) and Python 3 (in Bright 8.3 and beyond). It runs on any operating system that supports Python including Linux, Windows, MacOS, and many others.
This article describes how to collect metrics from non-Bright-managed nodes using the Lightweight CMDaemon.
1. Install the cm-lite-daemon package on your headnode.
[root@kube-head ~]# yum install cm-lite-daemon
[root@kube-head ~]# rpm -ql cm-lite-daemon
The package contains a zip file that contains the cm-lite-daemon directory. Copy the zip file to your lite node and unzip it.
[root@kube-head ~]# scp \ /cm/shared/apps/cm-lite-daemon-dist/cm-lite-daemon.zip \ root@lite01:/opt/cm-lite-daemon.zip
[root@kube-head ~]# ssh root@lite01
[root@lite01 ~]# cd /opt
[root@lite01 opt]# unzip cm-lite-daemon.zip
[root@lite01 opt]# ls -l cm-lite-daemon
-rwxr-xr-x. 1 root root 4986 Apr 11 06:58 cm-lite-daemon
-rwxr-xr-x. 1 root root 708 Apr 11 06:58 cm-lite-daemon_ctl
-rwxr-xr-x. 1 root root 2469 Apr 11 06:58 connection_test
drwxr-xr-x. 2 root root 130 Apr 11 07:06 etc
drwxr-xr-x. 2 root root 76 Apr 11 06:58 examples
drwxr-xr-x. 2 root root 32 Apr 11 07:05 log
-rwxr-xr-x. 1 root root 5762 Apr 11 06:58 register_node
-rwxr-xr-x. 1 root root 2808 Apr 11 06:58 request_certificate
drwxr-xr-x. 2 root root 78 Apr 11 06:58 service
-rwxr-xr-x. 1 root root 3907 Apr 11 06:58 unregister_node
2. Install Python pip if it’s not already installed.
[root@lite01 opt]# yum install python-pip
3. Install the required Python modules.
[root@lite01 opt]# pip install websocket-client pyyaml psutil \ py-cpuinfo uptime subprocess32 netifaces
Note: You may need to install additional packages depending on the build of your Lightweight CMDaemon node. I had to run ‘yum install python-devel’ and ‘pip install pyopenssl’.
4. Register the Lightweight node with the Bright head node. Registration adds the node to the Bright configuration (as node type ‘litenode’) and starts the Lightweight CMDaemon. The register_node command assumes that master:8081 is reachable from the registrant node. If that’s not the case, add the --host flag, as shown below.
[root@lite01 opt]# cd cm-lite-daemon
[root@lite01 cm-lite-daemon]# ./register_node --host \ kube-head.rmstober.net
Note: Depending on network setup these options may also be necessary:
--disable-hostname-check => Do not check SSL certificate hostname
--disable-follow-redirect => Do not follow forward to shared HA IP
5. Configure the Lightweight CMDaemon to collect desired metrics. The Lightweight CMDaemon does not collect any metrics or run any health checks until you configure it to do so. Here we add a data producer of type ‘metric’ with an arbitrary name ‘simple-metric’. The type metric denotes a “single line metric”. The single line metric will create a metric with the same name as the data producer, which is simple-metric in this case.
[kube-head->monitoring->setup]% add metric simple-metric
Note: For a healthcheck you need to use ‘add healthcheck <name>’
The script is the path to the data producer. We will use one of the examples that are included in the cm-lite-daemon zip file.
[kube-head->monitoring->setup*[simple-metric*]]% set script \ /opt/cm-lite-daemon/examples/simple.metric.py
[kube-head->monitoring->setup*[simple-metric]]% set consolidator \ default
Add a filter of type ‘lite’ to restrict the data producer to nodes having type ‘lite’.
[kube-head->monitoring->setup*[simple-metric*]->nodeexecutionfilters]% add type lite
[kube-head->monitoring->setup*[simple-metric*]->nodeexecutionfilters*[lite*]]% set litenode yes
Now we can see the metrics that are being produced. Note that the name of the metric is simple-metric, the same name as the data producer.
[kube-head->device[lite01]]% dumpmonitoringdata -10m now simple-metric