Understanding the Bright Cluster Manager 9.1 Integration with Jupyter


By Robert Stober | April 15, 2021 | Bright Cluster Manager, Jupyter



Bright Cluster Manager 9.1 includes an integration with Jupyter Notebook that accomplishes several important things that make Jupyter a more effective and powerful tool for users:

  • Bright makes it possible and easy for Jupyter Notebooks to run on a cluster, increasing the scope of work that can be performed through Jupyter by increasing the resources available for work within the notebook. This is especially important for areas like machine learning, where the scope of work escalates quickly as data is leveraged to train models.
  • Bright’s integration with Jupyter provides a point-and-click interface for users that are not familiar with the complexity of submitting jobs to a cluster.
  • Bright’s integration with Jupyter Notebook includes support for HPC workload managers, making it possible for HPC practitioners to leverage Jupyter’s power and ease-of-use. In this way, you will be able to see all of your jobs and perform operations on them, as well as submit jobs from a notebook and see the output being inlined.
  • By running Jupyter Notebooks on Bright managed clusters, users also have control over the environment that their notebook is running in, such as where and how their kernels are run, the number of tasks, the consumable resources, the job name prefix, the directory the kernel runs in, and which queue the job is submitted to. This allows you to quickly tweak parameters inside of kernel definitions without having to open up a text editor to edit JSON content.
  • We have also made it possible to run X11 applications on the cluster and view the output in your web browser. This enables scientists to run input/output data visualization tools on the cluster, so you no longer need to transfer data to your workstation.

Jupyter Notebook is a popular open-source development tool used by data scientists, programmers, analysts, and engineers to create and share “notebooks” that integrate live code, equations, computational output, visualizations, and other multimedia resources, along with explanatory text.  

Jupyter Notebooks are effectively bound to a single server, so the computing capacity that can be brought to bear on work performed in a Jupyter Notebook is limited to that server. For those who are not experts, clusters can be intimidating to use from the command line. At Bright, we believe Jupyter is a great new way for clusters to be used by end-users. All you need to do is point your web browser to a cluster and use Jupyter Notebook to drive your computation.

Let’s not forget how easy it is to install Jupyter on a Bright cluster. Jupyter itself runs on the login nodes of a cluster, and it will schedule Jupyter kernels through HPC schedulers or Kubernetes.

To get started or learn more about our Jupyter integration, please watch this demo or get in touch.