In the days before big data, traditional business applications relied on squeaky-clean scrubbed data input, which, to be usable had to be:

• warehoused in a precise and organized way in rows and columns where data discipline resisted scaling and change
• amenable to standard querying, subject to the “GIGO” (garbage-in-garbage-out) caveat
• collected with a view towards answering questions or serving a purpose that was both narrow and known well in advance
• tied into making better decisions that were both human-based and principally off line
• restricted to either data analysis/reporting or data writing, but unable to do both at once

So any activity “outside the loop” was not accommodated well without application restructuring, which could produce unpredictable results. Databases and their data reporting applications form a closed-loop data dependency. That dependency results from an array of refined input that protects the integrity of all the data through rules, which resist casual tinkering.

There is, on the other hand, an emerging class of business and industry applications that use big data, where:

• the same database stores both the data and does real-time analysis on the fly
• the database can support offline analytics on another cluster of the same database
• the data processing between the online data and local enterprise data warehouse is faster and far less complicated
• the database can deal with unrefined “raw” information outside the realm of traditional SQL disciplines

That “raw” information is what big data is all about. New applications can now capture a variety of data, rather than just sample it and shunting to another database.

The newer applications are becoming increasingly more capable in tuning into the stream of data that emanates from other systems. The new sources of input, no longer restricted to just business transactions, result in a bigger variety of business intelligence.

Of course, moving outside the narrower, more disciplined realm of traditional data gathering and analysis poses new challenges in managing big data. Computer replica clusters running on Hadoop solve that by allowing local analysis while new real-time data is continually fed to the online data warehouse.

The foregoing solution is where real-time applications can run on an incredibly scalable database by applying dynamic rules in an endless variety of combinations. Returning to traditional non-big-data databases, analytics were the questions asked in advance. With big data, the analytics can be part of a continuous loop based on “rules within rules” for better and quicker business decisions.

Whatever your big data storage and analytics needs, we have the software you need to manage it easily. Contact us for your best and most flexible cluster management options.

Photo Credit: thethreesisters via Compfight cc