Leverage Integrated Development Environments using Bright’s Jupyter Dev Server


By Bright Staff | September 21, 2021 | Bright Cluster Manager, Jupyter Dev Server



The Jupyter Notebook is a popular open-source development tool used by data scientists, programmers, analysts, and engineers to create and share “notebooks” that integrate live code, equations, computational output, visualizations, and other multimedia resources, along with explanatory text. Redesigned in Bright Cluster Manager 9.1, Bright’s Jupyter integration takes this tool a step further and accomplishes several important things that make Jupyter a more effective and powerful tool for users. Namely, data scientists and ML practitioners can leverage this integration to conveniently run applications by writing notebooks in their browsers. Bright makes it possible and easy for Jupyter notebooks to run on a cluster, increasing the scope of work that can be performed through Jupyter and providing a point-and-click interface for users that are not familiar with the complexity of submitting jobs to a cluster. This workflow makes it easy to run different experiments, enables rapid prototyping, and allows fine-tuning ML models.

With that in mind, writing Jupyter notebooks in a browser may prove to be limiting or complex while developing large applications. In this instance, scientists may want to take advantage of the rich set of features provided by a Python Integrated Development Environment (IDE), such as code inspection, autocompletion, and on-the-fly error checking. Two of the most popular tools for Python development are PyCharm Professional and Visual Studio Code, both of which support Jupyter Notebook.

The guides used to configure a remote Jupyter server and connect PyCharm or Visual Studio Code to a Bright Cluster where Jupyter is deployed are described in a dedicated section of the IDEs documentation. However, at this current time, developers can’t set up a remote server directly pointing to the Jupyter Cluster because both PyCharm Professional and Visual Studio Code do not support JupyterHub, which is a fundamental component of Bright’s Jupyter integration.

To navigate this limitation, Bright’s clever team of engineers developed an elegant solution called Jupyter Dev Server. This lightweight component can be executed on the machine where the IDE is installed and will act as a proxy―while running, it will transparently and securely redirect requests from the IDE to Bright’s JupyterHub cluster.


PyCharm Professional with Bright's Jupyter Kernels


Visual Studio Code with Bright's Jupyter Kernels

Bright has set up a host of materials to help guide you through the process of setting up the Jupyter Dev Server and PyCharm Professional (the procedure to configure Visual Studio Code is similar):

  • Our ML team has prepared this excellent walkthrough demo for PyCharm Professional. This demo will showcase how to set the dev server and PyCharm professional, generate tokens for JupyterHub, and run some notebooks with Bright’s kernels.
  • Check out the GitHub repository for this tool, here you will find further technical details about IDE’s limitations and helpful notes to install Bright’s proxy.
  • Visit our Knowledge Base article for more information about our Jupyter Dev Server, as well as helpful links and screenshots explaining more about running Jupyter kernels with an IDE.

Some scientists may be involved more in data analysis and visualization than development. In these cases, other environments may be more suitable for their needs. For example, the new JetBrains DataSpell makes it even easier for scientists to visualize data with all the popular Python scientific libraries. Jupyter Dev Server will make it also possible to connect the Bright cluster to JetBrains DataSpell and seamlessly use it to edit and run both local and remote notebooks.

DataSpellBright Cluster connected to DataSpell

Bright’s Jupyter integration makes the complexity of dealing with cluster resources and workload managers transparent to the end-users. Bright’s Jupyter Dev Server takes another step in this direction, eliminating the necessity for users to leave their development environment. In fact, while coding in their IDE, the entire Bright cluster resources will be only displayed in the form of a dropdown menu with the available Jupyter kernels. Developers will not need to know a cluster exists and how it is configured: they will only straightforwardly use its resources while taking advantage of the IDE features and plugins.

To learn more about Bright’s Jupyter integration or Bright Cluster Manager, reach out to us directly at info@brightcomputing.com.