“Convergence”. It’s a term tech lovers love to hate. It’s a term that has made the rounds (remember converged video, converged networks, converged communications?) and now it’s our turn. “Convergence” is changing the way we look at HPC and Big Data.
In many ways, the challenges of building production systems for Big Data are very similar to those of HPC or technical computing. Both need to process data without any stoppages, and deliver on the key metric of time-to-value.
IDC analyst, Steve Conway has been talking about the use cases driving this convergence for some time now. Citing the need for high performance fraud detection, healthcare informatics, and smart grids, he believes there is a growing market for high performance data analysis. In fact, IDC predicts that the revenue from servers used for high performance data analysis will reach $1.4 billion by 2017.
We agree with him. Our own customers see these high performance data market drivers:
- Need for more powerful scientific instruments/networks
- Need for more protection against fraud or terrorism
- Need to process more iterations in time available
- Need for more powerful mathematical models and algorithms
Good job schedulers are more important than virtualization in large-scale production big data deployments because moving all of that data around isn’t easy – especially over long distances. Virtualization is rare because of the impact it has on the processor and the length of time it takes to process large jobs. Data consolidation is as important as good job scheduling and the infrastructure tends to be concentrated around specific "pools" of data.
Big Data systems used for development are often built in cloud environments so that different elements of the solution can be tested easily without purchasing hardware. However, when it comes time to put these systems into production, practical issues such as the cost and time needed to move large amounts of data being into the cloud often dictates the use of on-premise solutions.
Data sets are growing very quickly in size and complexity, driving the need for high performance data analysis systems with end-to-end management solutions to keep them running at their best.
To get more information on managing high performance data solutions, contact us.