Cluster on Demand – A Great Idea Whose Time Has Come


By Piotr Wachowicz | April 21, 2016 |



I’ve written in the past about how Bright is moving towards the use of Cluster on Demand (COD), earlier referenced to as Cluster as a Service (CaaS), which allows you to stand up and tear down a cluster in whatever configuration you need, whenever you need it.

We built our own in-house COD solution to be able to rapidly provision virtual clusters inside of Bright’s own OpenStack private cloud during our development work. Our initial goal was to avoid the issues we kept running into when repeatedly building and tearing physical clusters. It was so successful in our lab that some of our customers wanted to be able to do the same thing in their own environments.

That’s why we decided to make this functionality part of the Bright OpenStack offering. COD takes care of deploying the clusters swiftly, on-demand, in a fulInfrastructure_Graphic_Updated.pngly unattended manner.

So when would someone want to use COD?

The simple answer is – whenever they want to have complete control over their environment. That’s one of the differences between using Sahara to deploy Hadoop as a service versus using COD + Bright + Hadoop to deploy a Hadoop cluster managed by Bright. In the former case, you end up with a highly opinionated Hadoop cluster, which you cannot really easily modify. In the latter case, you have complete control over all of the layers. This includes the operating system, Hadoop services, and other services. You get unified monitoring, using single pane of glass management. This means you can tweak the Hadoop (or whatever it might be running inside of the COD) much better, and to a much higher degree.

How will COD be enhanced with Bright OpenStack Version 7.3?

COD makes extensive use of Heat templates, which are text files that can be treated like code. Heat lets users launch multiple composite cloud applications. Bright OpenStack Version 7.3 comes with Mitaka, which has vastly improved the OpenStack Heat service by optimizing it for horizontal scalability. Several bugs have been ironed out in OpenStack by the community, so using COD with Bright OpenStack 7.3 will result in a much smoother end user experience.

What’s coming down the parkway for COD?

We are thinking that containers might begin to come in handy in some cases. For example, we have tested deploying clusters inside of ‘systemd-nspawn,” which is similar to Docker and LXC, but a bit rough on the edges. Containers will not work for all use cases, for example, whenever we have a requirement for a specific kernel. But it might work pretty well for other cases, like scalability testing.

But using containers for COD is still only somewhere on the near horizon. Recently, we’ve been working on speeding up the cluster creation process by making more aggressive use of Ceph’s copy-on-write functionality.

Make sure to stay tuned for upcoming versions of Bright OpenStack as we’re preparing to offer full functionality of COD with Bright OpenStack in the upcoming versions.

EU_flag_7751.pngThis project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 711315 Bright Beyond HPC

Bright Cloud Bursting