Issues and Solutions to the Management of Big Data

Issues and Solutions to the Management of Big Data

By Mark Anstead | Business intelligence | 20 Oct 2018


Over the last few years with the explosion of technology, we’ve seen websites like Facebook and Dropbox, which are platforms that stores massive amounts of data.  However, with this, hackers have found a habit of breaking into these servers that might have extremely confidential data and information.  The problem is that there are new technologies being developed outside of the typical firewall that prevents hacking. 

This technology, blockchain, has helped to create more secure platforms is still in its infancy, but it’s interesting to see how companies can use it to help with analyzing different information and data sets. On top of that, companies are able to use artificial intelligence where they are easily able to process this information and move into the stage to where you are able to report it. 

Issues with the Management of Big Data

Over the last decade, we’ve seen an exponential growth in the technology world. With that, we’ve had topics arise with big data and ethical issues stemming from that.  So exactly how can data be used to create problems? Well, the two big problems that have come up are with privacy of our data, and the security of it.  Part of the issues we are seeing is how this data is distributed on servers (centralized manner), and the laws that haven’t really been made to keep our privacy, and feeling of anonymity. 

Some of the issues we’re going to encounter more and more going forward is also how well we are able to scale our systems to handle the amount of data being stored since it comprises of “raw, structured, semi-structured, and unstructured data” (Agrawal & Nyamful).  The issue isn’t so much the different types of data, but just the massive amount of storage required by our traditional systems.  In many cases, there are just massive data centers that different organizations access to store information and process/distribute large amounts of data.  Typically all the computers on these networks are interconnected, allowing them to share information. 

When looking at Big Data, we’re gaining a massive amount of breakthroughs through different applications and services, but with that, we’re taking a big risk with our privacy.  This is a fine line right now with the government since they don’t really know exactly how to regulate different privacy rules (Marr).  Finally earlier this year, the General Data Protection Regulation (GDPR) has the goal to give users more control over their data.  It was actually interesting to go on different websites that stores massive amounts of user data when this passed to have little blurbs talking about how the website is going to protect and keep your data private.  The issue is the majority of people don’t actually realize that this data has monetary value, and are willingly giving their data to many of these companies. 

The other concern mentioned earlier is with security.  Even after the GDPR releasing the announcement to try to secure users data, there was a massive breach on one of the most active websites, Facebook. Originally, they believed the hackers breached 50 million accounts, but more recently they have determined it was 30 million- still a pretty massive amount of users.  With this massive hack, “14 million had their names, contact details and sensitive information such as their gender, relationship status and recent location check-ins exposed” (Rodriguez).

It is absolutely crazy how these type of hacks continue to occur. Hackers are always staying one step ahead of the current state of protection for servers, which is why it is baffling that companies don’t look to more advanced features to protect their users.  One of the more famous hacks of recent years was that of Dropbox.  According to reports, more than 68 million accounts lost their credentials and were stolen by a hacker (Newman).  While Dropbox was able to take some of the basic steps required to protect their users afterwards, like doing a mass password reset, let’s hope that the company has learned their lesson to how vital this can be to the success of their organization and people’s personal information.

Decentralized Cloud Storage

When looking at the security of data, it is far easier for hackers to break into servers that are built into centralized servers as opposed to a decentralized one.  In order to protect the information of individuals online, businesses need to into platforms that enable them to store information and data.  There are many platforms that people can use that are actually comparable to AWS (Amazon Web Services) or Dropbox, like IPFS, FileCoin, Sia, Storj, and MaidSafe.  These platforms all store these files and data using cryptography, making it virtually impossible for anyone to hack into the information that is stored.

In addition to that, when this information is stored across a network of servers in a decentralized manner, this actually raises the difficulty level for hackers to an even higher level! They’d need to hack into each and every “node” in the network to uncover the information stored across these servers.  This allows the customer to still access their stored files with ease because they just need to enter in a key.


It’s incredibly interesting to see massive enterprises begin to integrate some of these emerging technologies as we are really starting to see some of the many use cases of them. The big question to find out going forward is how scalable some of these technologies are, and the expense to keep all of the data from many of these automated vehicles that we are having unrolled.  There are various emerging technologies like Hyperledger, Hashgraph, and Tangle that will be extremely interesting to see how they end up developing.

These are going to be incredible use cases for businesses and corporations for their massive amounts of data assuming that these technological breakthroughs continue to shape up like their predecessor, the blockchain.  This behavior in the market is very similar to that of 20 years ago when the internet was truly in its infancy. Developers and IT specialists from all around the world work together to help build what the Internet is today! Hopefully this same type of outcome will occur with some of these emerging technologies, but only time will tell!


Agrawal, R., & Nyamful, C. (2016, March). Challenges of big data storage and management. Retrieved from ResearchGate website:

Blenkinsop, C. (2018, September 27). TD Bank Partners with Startup to Add Blockchain and AI Solutions to Investment Products. Retrieved from CoinTelegraph website:

Lannquist, A. (2018, January 19). Blockchain in Enterprise: How Companies are Using Blockchain Today. Retrieved from

Lovells, H. (2017, September). A guide to blockchain and data protection. Retrieved from

Marr, B., Mr. (2017, June 17). 3 Massive Big Data Problems Everyone Should Know About. Retrieved from Forbes website:

Newman, L. H. (2016, August 31). HACK BRIEF: 4-YEAR-OLD DROPBOX HACK EXPOSED 68 MILLION PEOPLE'S DATA. Retrieved from

Qureshi, A. (n.d.). What Does The Rise of Blockchain Technology Mean For Big Data? Retrieved from Smart Data Collective website:

Ray, S. (2017, November 5). Blockchains versus Traditional Databases. Retrieved from Hackernoon website:

Redactor. (2018, April 3). How Blockchain will Improve Big Data. Retrieved from Medium website:

Rodriguez, S. (2018, October 12). Facebook says hackers were able to access millions of phone numbers and email addresses. Retrieved from

Mark Anstead
Mark Anstead

Co-Founder of Numio

Business intelligence
Business intelligence

Impacts of Business Intelligence with blockchain

Send a $0.01 microtip in crypto to the author, and earn yourself as you read!

20% to author / 80% to me.
We pay the tips from our rewards pool.