This year marks the 10th anniversary of the creation of Hadoop. Over that decade, the open-source software platform has grown into the de facto standard of the big data marketplace. Part of this is due to the overall growth in spending on analytics, which research firm IDC recently said would reach $125 billion this year. But it also is due to the advantages that open-source Hadoop has over proprietary systems.
In December 2014, the online technology community Wikibon issued a report by lead big data analyst Jeff Kelly titled Hadoop-NoSQL Software and Services Market Forecast, 2013-2017. He highlights three problems with traditional database management systems:
Cost – IT departments need to manage rapidly growing data stores while, in many cases, their budgets are growing little, if at all. Recent surveys have shown that up to 48% of big data practitioners have yet to realize the full value of their analytics investments. “Simple mathematics dictates that the current data management paradigm is simply unsustainable from an economic standpoint,” says Kelly. “Enterprises are forced to devote more and more of their stagnant IT budgets to scale existing, traditional data management systems, leaving fewer funds to support innovation and value-add projects.” This is backed up by research firm Gartner Inc., which recently forecast overall IT spending increases of 2.4 percent in 2015, down from its earlier projections.
Performance – Older database management systems simply are not designed to manage big data and so cannot deliver the expected performance. As Kelly explains, “Conversations with members of the Wikibon community make clear that as both data volumes and the complexity of analytic workloads increase, relational database management systems and related data management tools are unable to provide the level of performance required to meet demanding business conditions.”
Agility – The time it takes to prepare and model data with traditional RDBMS makes them unsuitable for business environments that require rapid answers.
The same three factors—cost, performance, and agility—are leading ever-larger numbers of enterprises to make the switch to Hadoop and NoSQL to meet their analytic needs. According to Wikibon’s Big Data Analytics Adoption Survey, 2014 – 2015, released last August, nearly a third of enterprises using big data already had deployed Hadoop in production environments. “[Hadoop] has emerged as the de facto foundational technology in the modern data management stack,” says Kelly.
While Hadoop itself is an open-source Apache Foundation project, and the software can be downloaded for free, the way to get the most out of the Hadoop framework is to take advantage of the third-party software and services that have been developed to limit or totally eliminate the need for additional service requirements for Hadoop clusters. Last year, customers spent $621 million for Hadoop distribution software, support subscriptions, and professional services—a figure expected to nearly triple to $1.7 billion by 2017. The software, support, and professional services market is expected to grow from $411 million to more than $1.5 billion during the same time period, according to the Wikibon report.
“As Hadoop has matured, so have the commercial entities that are developing products and services around them, along with the enterprise practitioners that are deploying them,” writes Kelly. For example, Bright Cluster Manager makes it possible to install a complete Hadoop cluster within an hour, and then to monitor, manage, use, and scale that cluster typically without any requirement for professional services.
Drew Robb has been freelancing since 1996 with a focus on the energy industry, power, oil & gas, water, IT, enterprise applications, business intelligence, CRM, and more.
Remember to subscribe to our blog to stay on top of the latest tips, innovations, and best practices for cluster management.