Watch this space: Big Data – Big Problems?

Posted on: December 9, 2011 by Paul Albada Jelgersma

When you run out of space in your cupboard, you go out and buy a new cupboard. You might even choose a similar model so it looks good in your bedroom or kitchen.

If you run out of floor-space in your house, the problem is a bit more complex from a financial point of view – but the solution is similar.

I think we had, for many years, the same expectation in IT. If we ran out of storage, we would buy additional storage. Well, it seems we need to wake up and face the problem, because the solution is not that simple anymore.

In an soon to be published whitepaper from the Atos Scientific Community I read:

“[…] several major changes in the IT world have dramatically increased [data storage and processing needs] rate of growth. […] Computer capabilities have not increased fast enough to meet these new requirements. When data is counted in terabytes or petabytes, traditional data and computing models can no longer cope.”

This problem forces us to have a different view on storage and database technologies. Traditional databases that use a relational model cannot process the data quick enough and adding additional computing power and memory is not the solution.

The issue is luckily addressed by storage and database vendors – they coined the term “Big Data” and are developing new solutions to make sure we can cope with the rapid increase in the information we want to be available online.

Unfortunately the impact of these new technologies is big (no pun intended) and there is limited experience in the way the technology is applied successfully and sustainable.

Some vendors are looking towards changes in hardware and provide dedicated storage-boxes that are hardwired to handle large databases or large data-files. Others are looking to provide solutions using new database software.

Most of the software developers and vendors that are facing big data issues are reconsidering the ‘traditional’ relational database model and are bringing new ‘NoSQL’ database models into view.

Based on the amount of marketing and buzz , this ‘NoSQL’ seems to be the next best thing to go with for these type of solutions. So, do we really need all of this stuff? The Scientific Community whitepaper claims:

“In most situations, using NoSQL solutions instead of RDBMS (relational database management systems - paj) does not make sense in cases where the limits of the database have not been reached. Although, given the current exponential growth of data storage requirements, these limits are increasingly likely to be reached in the future. Unless RDBMS evolves quickly to include more advanced data distribution features, NoSQL solutions will become more and more important.”

The specialists have spoken – it is important. We need to care and we need to take action. Additional problem is that the field is evolving quickly, good solutions are provided by small companies and will soon become part of large providers through acquisitions or other business activities. I also expect some patent-conflicts (do we not love those?) and maybe some bad choices leading to loss of data.

My recommendation is you start looking for areas in your organization where this challenge will become a problem very soon. Ask your systems administrator about the time they need to do backups of databases and restore times. Ask your system developers if they foresee issues with your next generation document management or transaction processing system. And while you are at it – ask your business analyst about the data they need to create meaningful business intelligence reports (and how much time it takes to create them). This will give you a good overview of your Big Data improvement areas. Do not ask your vendor before having done an internal assessment. You do not want to be stuck with the wrong technology.

I will let you know when the whitepaper is published so you can download it for your benefit.

Share this blog article

About Paul Albada Jelgersma
Global Head Atos IoT Solutions and member of the Scientific Community
Paul has over 30 years of experience in running IT and business programs; including managing large teams of IT and other professionals. Paul is a founding member of the Atos Scientific Community. In his day job, he focusses on developing and delivering Internet of Things Solutions for our customers. Paul likes to work with Virtual Teams in the new way of work. He dislikes email as a chat and document-management tool. He occasionally writes articles and publishes at his own blog. Paul is married and has 2 children. He lives near Amsterdam, The Netherlands. He likes to travel, moderate hiking and watch Netflix series.

Follow or contact Paul