The big data revolution shaping the role of traditional IT professionals
Big data is an ever-changing technology that, if successfully implemented, will open up new areas of discovery for your business. However, its introduction has also caused significant upheaval, forcing changes to the roles of many information technology (IT) professionals, and pressuring them to learn new skills to fulfill their role requirements.
Before the digital transformation, data was processed by a team of qualified data analysts and data scientists. While this is still the case, the expansive capacity of big data requires a deeper understanding of what it can help a business achieve, and this has lead to the recruitment of skilled professionals or the re-education of those already employed to handle the influx of information.
It’s the process to gain more value out of data that is putting the role of the generic IT professional in jeopardy and has forced businesses to undertake a significant overhaul of their current positions, as they alter their focus towards a more data-driven world. So below, we’ve listed some of the most significant challenges faced by your business and IT professionals as they make the big data switch.
The influx of more information
When implementing strategic big data operations, what you chose to do with the information will come down to fulfilling the goals set out in your data management strategy, as failure to adapt or capitalise on the new streams of data flowing into your business can make realising its value an impossible task.
It’s this diversification from managing small amounts of data to handling large amounts of data that your business will need to govern. But’s this is also one area that will provide considerable change to the role of your current professionals.
If your company has yet to begin the big data changeover, then Reza Moaiandin, Co-Founder of SALT.agency and Cyber Scanner, advises that successful transition will require your IT professionals to learn, adapt and create new techniques to assist your business expansion.
“One of the best ways to solve data growth is to move to a more scalable platform, which means that you can process and store data across a large number of servers and storage facilities in a scalable format,” says Reza. “Once you hit your limit, you can add more resources to the network, and it should all come together.”
But as Reza points out, saying it is easy but putting it into action it can be a lot harder. “Companies such as Amazon, Rackspace, and others have tried to solve this solution by creating the Cloud, but it still requires a deep degree of knowledge to be operated correctly.”
Changes happen fairly quickly in IT, so to keep abreast of the new technology and techniques, IT professionals continuously need to update their knowledge. “But more often than not, this doesn’t happen,” continues Reza. “The correct security measures aren’t always implemented, causing security vulnerabilities ranging from website installation to more complex applications, which will allow hackers to have a starting point for more severe problems.”
Increasing your business security
The crossover between big data and information technology security is becoming a common trend, and it’s something you’ll encounter as you build robust prevention against cybersecurity threats.
Adding data analytics to your security practices can be quite a significant task. However, with a recent survey by the US government only further highlighting the blossoming partnership between the two techniques, as 84% of respondents said they had used big data to block external security threats, perhaps it’s time for our business to get on board.
It’s the actionable intelligence drawn from big data that’ll allow your business to halt threats and prevent a risk from penetrating your networks. Part of the successful eradication comes from the detailed analysis of raw and historical data, and it’s this aspect that is forcing the current generation of IT professionals to focus more on big data.
However, combining the technologies can disrupt your business. Without the right experience and training, your current IT staff will be unable to handle the volume of data streaming into your business.
To prevent this issue from happening, you’ll need to introduce added training measures into your business, and for the roles of your current IT professionals to undergo changes successfully, security analytics expert Raffael Marty advises that systems need to be quickly adoptable from one industry to another.
“The industry keeps building systems for super advanced security people, but we also have to build systems that other IT professionals can understand,” says Raffy. “In a busy environment, those without relevant experience simply want issues to be fixed. However, this is one area that I’ve seen many companies have altered their focus towards.”
When working within a specific business, IT professionals will know all of the different elements that keep systems running, including the security tools. “If this is the case, professionals will have to understand everything about that tool, and what to do when ransomware attacks happen,” adds Raffy. “However, some do not understand how to stop attacks or what to do with the software itself.”
The lack of understanding on how to stop an attack sets alarm bells ringing in the security analytics industry, and should prompt your business to make big data and security analytics more accessible on a wider scale.
As you go through this process, it can place further strain on your business, but you need to remember it will be beneficial in the long run. “What we’re going to see is more a closed system,” Raffy adds. “This will increase the time between detection and response, which will make it easier for professionals not experienced in data and analytics, which in the long run will be very important.”
Ensure professionals follow the right rules and regulations
If you’re still behind the trend when it comes to big data, then your business could be missing out on crucial information. But if you’re already riding the big data train, the way in which you accumulate, analyse and visualise data will also change as the technology continues to develop.
To get the full value of your data, and to help you meet the objectives set out in your data management strategy, you should have the right practice and principles in place so the information collected can help during your decision making.
Utilising big data can be a game-changing factor. However, when first collecting, and then using the data, you should ensure you have the right to use it, and you completely adhere to data privacy laws. These new rules and regulations could be a significant changing point in the role of those who currently handle your data, placing further strain on their position as they may need additional training.
Partnering the changes to data privacy is the introduction of GDPR, which will also require a complete role change from a current member of staff. The new rules and regulations require your business to employ a data protection officer, whose role will centre around ensuring all the new privacy laws are upheld.
Following the recent incidents surrounding data privacy, it only further highlights the need for your business to successfully adapt to the use of big data, while ensuring you have the right professionals in place to safeguard your data.
But as Ivanka Menken, CEO of The Art of Service advises, it will come down to the professionals in your business to kickstart this process, and those who lack the experience to be offered relevant training in regards to how to separate personal and private data in a way that total privacy of the data guaranteed.
“An IT professional will need to know the answer to this question,” says Ivanka. “When sharing data, are appropriate procedures such as sharing agreements put in place to ensure that any personally identifiable information remains strictly confidential and protected from unauthorised disclosure?”
To answer this question, new security measures will need to be implemented, which will call for a change in the role played by your IT professionals. But, you’ll need to be aware that it won’t only come down to the big data specialists to protect the information you collect. For the entire process to be effective, it will need to be a company-wide effort, as one missing link could open up a security threat.
The move away from technology to information
As your business grows, you will continue to develop new uses for your data, which ultimately means the power you’re able to harness will exponentially increase.
In a bid to achieve the most value, there has been a positive movement towards the information side of big data rather than the technological aspect, which is something that many professionals in your business will need to adapt to.
As your IT professionals continue to work with the gathered information, Kirk Bourne, Principal Data Scientist and Executive Advisor at Booz Allen Hamilton, explains that their roles are going to have more of an emphasis on the I in their title, rather than the T.
“IT started as a conduit for the digital information,” says Kirk. “But the technology became the major focal point as it is the enabler of the generation, transmission, sharing, and exploration of digital information.”
But now as the focus shifts from the enabler to the information, businesses have begun to feel greater benefits. “It’s the digital information (big data), that delivers insights, value, benefits, opportunities, innovations, transformative power, and big ROI from IT investments to organisations. While IT professional must still deliver secure systems that can manage big data assets, the business driver is now unequivocally the data.”
Recruiting the right professionals
Introducing big data into your business will open up a myriad of new opportunities. However, when it comes down to the skills needed to manage the technology, having the right people in place who have the prerequisite knowledge to handle the system can set your business above the rest.
If you choose to go down that route, recruiting the right professional can be a crucial part of your big data strategy, and although your business may already have skilled IT staff, without the right people to manage the influx of information, it can become a significant stumbling block for your business.
This requirement for such specialist knowledge to handle big data has forced a considerable change in a number of IT roles. However, it’s these positions that are having to adapt, because as Daniel Child, Manager at Churchill Frank explains, the industry is still very candidate short.
“Big data is still very much an emerging market. I’ve found when it comes to specialist roles, finding the right candidates to fill those are like gold dust,” says Daniel. “When a client requires a candidate with a niche skill set, finding people with the right qualifications, certifications, and relevant experience can be a challenge, as many candidates don’t meet the full requirements.”
As the technology continues to develop, the fact that skilled big data professionals are so in demand is a factor your business won’t be able to ignore. However, if your business isn’t flexible in its recruitment approach for the role, and candidates with experience aren’t willing to adapt, then your business could face a significant skills shortage in this area.
“When it comes down to securing the right people for a role,” continues Dan. “I’ve found that candidates aren’t as flexible as they should be, and this negative effect will only add to the burden on current IT professionals, forcing them to change their roles to fill the gaps that are appearing.”
The change of career path for IT professionals
Big data isn’t a brand new technology, but it’s only in recent years that we have seen full utilisation, with more businesses growing and adapting to the technology.
Thanks to the technological snowball, numerous professionals who move into big data from academia often find increased success as they have previous experience working with similar concepts and actions. It’s this path that Leila Powell, Security Data Scientist at Panaseer followed, as when studying to become an astrophysicist, she discovered an interesting career path leading into data science.
“The emergence of big data has started to group together a set of cross-disciplinary skills that would add value anywhere that has lots of data for analysis,” says Leila. “Essentially, this new perspective emphasises recruiting for experts in data itself, not in the subject matter it describes, and this opened the door for people who could handle data to move across a wide range of industries, rather than following the typical path of specialising and staying put.”
The fact that big data has continued to grow in value allows more industries to consider themselves in need of experienced IT professionals who specialise in big data or data science, which in turn, opens up career options in the sector thanks to big data.
“I chose cybersecurity,” Leila adds. “In part, this was largely because I was impressed with the team I would be joining, but also because it is an issue of ever-increasing importance and provides an abundance of interesting data-driven problems to tackle.” Although, it is still early days for the use of big data in that field, “It’s an incredibly exciting time to be involved.”
Continued disruption dead ahead
Whether you’re an IT professional, who has suffered upheaval due to big data or a business who is behind on the implementation curve, the technology itself shows no signs of slowing down.
As the industry continues to develop, the successful combinations of big data with external sources will only increase, and while Jason Wisdom, CTO of Aponia, believes that big data has already caused disruption, just like the cloud, next on the list is artificial intelligence.
“Ten years ago, IT professionals carved out a niche and stuck with that for their entire career. However, big data has made career niches obsolete, nobody becomes a database administrator anymore,” says Jason. “Big data is evolving with breathtaking speed. Two years ago, Spark on Hadoop was the big advancement, with professionals focusing their careers on that. Now, Spark has moved beyond Hadoop.”
If your chosen career path leads to big data, or your business is looking for those wishing to move into the sector, it can open up a world of new opportunities. “For traditional DBAs that have been rendered irrelevant by big data, there’s the chance for those who like to learn new tricks to become architects or thought leaders within a new technology, as many of the architectural concepts will still apply, even if the dialects and syntax have changed.”
In the technological world, big data shows no signs of slowing down. So if you’re a professional working in the IT industry, the time to increase your skill set could be now. To ensure you don’t miss out a key role, or lack the experience your company requires, you must constantly be enriching your skill set. This way you will always stay ahead of the curve.
Has the introduction of big data affected your current role? Let us know in the comments below and don’t forget to sign up to the Churchill Frank blog.