It's big. it's powerful. It's complicated. Apache Hadoop creates a huge opportunity, but that cuts both ways. Aside from the great potential to store, access and leverage immense amounts of data, the security of any of these operations has been, and continues to be, questioned.
Security was not a priority in the development of the architecture, almost by definition, as the limits of the platform are purposefully undefined. As individual security measures were patched into place on an as-needed basis, overall security confirmation has changed from very difficult to almost impossible.
There is software on the horizon, however. Intermediate business-friendly apps are being developed to work with Hadoop. Such systems would circumvent the need for an outside whiz, simplifying an adaption to a specific business goal. "We're starting to see applications that can be added to the Hadoop system, which is a big change," says Michael Brown in Computerworld.
Specific providers and users are highlighted in that article. One app can deliver encryption, access control, policy enforcement and uninterrupted data up-time. Another is able to provide centralized capabilities around data security for easier governance.
The necessary proofs of concept can take new directions with new apps; one tester described "a massively parallel ingest on thousands of gnarly formatted data files" which was ultimately successful. A pediatric hospital in Atlanta has been testing and using Hadoop projects for about a year. As an example of an organization that is turning to new technology in a timely fashion, they are streamlining existing and new hardware to best take advantage of software capabilities as their database expands.
Cluster management is emerging as an even more vital element for the success of future big data mining. For more specific information on your unique organizational needs, contact us.