CIO and big data challenges they need to solve

Chief information officers play a crucial role in keeping your business abreast of the latest technological trends. By monitoring the latest innovations, they can identify the most-significant challenges you’ll face when growing your organisation; a recent challenge faced by businesses is the introduction of big data.

In this article, we spoke to industry experts working in the field of big data, and they gave us their take on the challenges faced by CIOs, and what they need to know when implementing the technology.

big data challenges: differences

When introducing big data into your business, you’ll soon begin to realise the endless possibilities that can be opened up by the technology. Each aspect has a strong set of traits, allowing it to be utilised for a specific task that will aid business growth and open up new areas of expansion.

To harness the power that big data can offer, and to gain the most intrinsic value out of the information, your first plan of action should be to create a data management strategy, as this will also allow you to stay inline with your business goals.

When creating your plan of action, one crucial piece of advice Jason Wisdom, CTO of Aponia, advocates you follow, is that each tool involved in big data serves a specific purpose. Although it’s becoming essential, you won’t use big data for every data-related task in your business.

“The most common misassignment of big data involves Hadoop,” says Jason. “Due to marketing campaigns, large companies spent seven or eight figures on Hadoop installations, then expected it to be a miracle pill that would revolutionise everything. The reality is that Hadoop was, and is, very good for certain things, but not good at all for other things.”

He points out that when using big data as a tool in your business, it’s crucial that you realise that each technology is a different tool on a tool belt. “No one big data tool is best for every situation,” adds Jason. “Hadoop is good for certain things, cloud storage such as Amazon S3 for others and an MPP database for other tasks, whereas sometimes a traditional, relational database is best.”

Big data challenges: Hadoop

As a CIO, finding the right tool to assist in data analysis is a crucial part of the role, and Apache Hadoop is one of the most praised tools across the entire field of big data.

A piece of open-source software, it provides your business with a means to store, process and share large amounts of data, while also formulating the numerous technologies associated with Apache all in one place.

But despite the positive advocation, including the fact it’s highly scalable and can prevent the change in data formats, it’s a tool that’s still growing, and something that Randal Scott King, Managing Partner at Brilliant Data, advises you steer away from. King believes it isn’t as matured and well-developed as the data housing software you’ll be familiar with.

Taking a more in-depth look at what Hadoop can provide, he says: “Sqoop is a command line only. Hue is graphical, but bogs down anything over 1G. And Oozie can only execute commands based on time, not the occurrence of an event.” It’s because of this that he recommends you supplement Hadoop with tools like Talend and Streamset, both of which are available in open source and commercial versions.

Big data challenges: business transformation

It’s no secret that introducing big data into your business can offer insights that would not have been previously possible. The information you accumulate can provide you with the evidence to make strategic business decisions, adapt your goals, or change the direction of your company.

As a CIO, a key part of your role is to take full advantage of the statistics and analytics offered by the technology, as this can be a new engine to power your organisation’s growth. Working as a catalyst, big data can act as the primary cog in the transformational process, and it’s something that David Chou, CIO at Children’s Mercy Kansas City, says you must use to your advantage.

“Assist your organisation to allow decisions based on the data and not just on past experiences or gut intuition,” says David. “CIOs can get caught up with trying to design the perfect solution, rather than keep things simple.”

He advises your business creates a solution that is easy to use by everyone, and will allow access to the data for everyone in your enterprise. “Stop focusing on setting up perfect data governance and unleash the tools that can solve basic business problems,” David adds.

Data governance is, of course, a crucial part of data implementation, as is ensuring you follow the right data privacy laws and rules. But, when used correctly, the technology as a whole can open your business up to growth it never thought possible.

big data challenges: new technology

Trying to gain a detailed insight into the data your business collects can be a tenuous task, and as a CIO, you may be completely unaware of what the data indicates if you lack the relevant experience in statistical analysis.

Every piece of raw data you collect will feature undiscovered patterns and snippets of information that will optimise business growth. But, if you’re yet to commit to a big data strategy, then you may lack the relevant staff and technological prowess to perform the analysis immediately.

On the other hand, if your business has fully embraced big data, your storage silos and data warehouses will be bursting with information ready for you to unlock its full potential.

To aid this task and transform your data into actionable information, Phil Husbands, Founder of Saltare, recommends introducing business intelligence software, and advises it’s a much more straightforward process than many would have you believe.

“CIOs who are looking to improve the quality of information in their organisation, and improve the steps it takes to generate information from data, need not be distracted by all the technology out there,” says Phil. “When implementing business intelligence software to get the most out of your data, the cost does not need to be hundreds of thousands of pounds.”

“When introducing new technology,” Phil continues, “start off simple with the basic links between your organisation’s goals, what you need to achieve, the information required and then how to get and use that information.”

As a business, you may still be using software like Microsoft Excel for BI purposes, and the big piece of advice from Phil is to stick with that software if the process works. “Don’t change unnecessary variables in business unless you need to,” Phil adds. “Getting rid of Excel is a really common BI objective, which doesn’t make sense. In a CIO role, I ask them to tell me why they don’t like Excel or why they think its a problem, and they can rarely come up with a good answer.”

From experience, he continues “CIOs often feel that if they aren’t using the latest cutting-edge, most-expensive technology, then they’re doing it wrong. My advice would be no; it’s not necessarily the case. It’s possible for you to have the latest tools and still be getting things wrong, BI is not a technology problem.”

Big data challenges: data protection

Big data is booming, but as your business begins to take full advantage of what it can offer, there will also be added risks. Throughout data collection, storage, and management processes, introducing the right practices and principles to protect the information flowing into your silos will keep you one step ahead of hackers looking to infiltrate your network.

When creating your data management strategy, factoring in data security and privacy protocols should be high on the agenda. In the current climate, failure to comply with the latest rules and regulations regarding privacy can land your business in hot water, especially with the impending introduction of GDPR.

Having a clearly structured plan in place will help you to take a step closer to implementing security analytics, which should also aid the protection of your data. But as data security expert Raffael Marty comments, your business should never introduce any aspects of data security by choosing algorithms or products to solve issues; you should begin by deciding what you are trying to do with the collected data.

“Ideally this will fit into your entire security program,” says Raffy. “You need to look at the company as a whole and say ‘where are the security problems?’ and ‘how can we solve those?’, and you may not need security analytics for that.”

But, if you reach the decision that your business does need to employ security analytics, then have a clear vision of your business goals and find a solution that fits them. “There may be a certain algorithm or system that can be applied,” Raffy continues. “I have seen many companies buying off-the-shelf products just to realise they don’t know how to deploy and use it.”

He advises that you figure out what you want to do with your chosen product, and the risks you are trying to protect your organisation from. Where do you need to spend money? Where do you have to protect things? Only when you have the answers to those questions will you be able to make confident decisions about security analytics, and the practices and principles associated with data security.

Big data challenges: consent and regulations

Data is an invaluable asset that can help you unlock the potential hidden in your business. But the security and privacy of the data are essential aspects of data collection that, as a CIO, you’ll be unable to ignore. Ensuring you have the right practices should be a well-thought-out process that should be implemented from the get go, and will put you in a strong position should any issues or threats arise in that area.

Playing a pivotal role in both the security and privacy of your data is the regulatory practices your business should be following, as a failure to uphold those could cause a significant issue, especially with the introduction of GDPR.

Many businesses have reported that they won’t be fully ready when the new regulations come into place. But as a CIO, who has a close working relationship with the team handling the transition, you should have a firm grasp of the rules and regulations.

If you still feel as though your business is a little behind, following a strategic checklist will ensure you’re on track to develop new security practices and ensure you’re completely compliant.  But if a regulator approaches your business, Ivanka Menken, CEO of the Art of Services, advises that you’ll have the perfect answer if they ask: Where did you get the data and how did the data subject agree to it being collected?

“Not being able to answer this question may open up the company to legal ramifications and fines,” says Ivanka. “Especially under the new rules and regulations that come into effect from May 2018 onwards.”

You should be decisively aware if your business is ready for the introduction of GDPR, or if your business is already compliant when collecting personally identifiable information. If you’re still behind the implementation curve, a failure to demonstrate compliance can lead to a fine of up to 4% of your annual global turnover, or €20 million, so you best speed up your implementation to be ready for May 25th.

Big data challenges: the rules to follow

As a CIO you can have all the information and background knowledge when it comes to the implementation of big data, but you can still face issues that you and key members of the team will have to solve. To give you a helping hand throughout the process, we advise you focus on the following points:

  • Implement a data management strategy
  • Ensure you’re compliant with all data privacy laws
  • Make sure your business is ready for the introduction of GDPR
  • Have a clear data security strategy in place
  • Explore new analytical opportunities
  • Select the right big data technology for your business needs

Are you a CIO who has successfully introduced big data into your business? Let us know in the comments below, and don’t forget to sign up to the Churchill Frank blog.