70% of Big Data projects in UK fail to realise full potential
Big Data has rapidly ascended as a major technique for companies to capitalise on the potential that sits within their data. According to a new study, nearly half of UK organisations now apply big data for insights; however, the track record still isn’t great, as just 30% of organisations that use Big Data are extracting enough value. Shortage in skills and too little knowledge about the topic are the main bottlenecks for successful implementation.
Big Data by definition entails the underlying need to process a voluminous and complex amount of data in the area of terabytes if not petabytes (in other words, a large volume of data). Its use has been touted as a potential game-changer for businesses as, while traditional data-processing application software is inadequate to deal with them, the capture, storage and data analysis of such data could reveal vital statistics relating to everything from corporate fraud, to changing consumer behaviour, to supply chain inefficiencies.
Due to this, Big Data is regarded as one of the most important drivers of digital transformation, however, tapping into that potential is easier said than done. Many UK companies are still finding their way around Big Data, testing technologies, use cases and tools, as well as new operational models.
According to a new survey from digital consultancy firm Whishworks, which polled 100 Big Data strategists, architects and users, the sector leading the way in the use of Big Data is Media and Entertainment, with over 75% of the sector’s data being processed by a Big Data solution. Media and Entertainment is followed by Telecommunications and Financial Services, which have both been using Big Data successfully for over three years. Further findings in Whishworks’ ‘The State of Big Data in the UK 2018’ report also revealed that Big Data also presents significant challenges, from costs to integration into a business model.
Huge uptake
Despite the relative youth of the Big Data concept, nearly half of UK organisations have already sired Big Data projects and initiatives. Of the organisations which have already implemented Big Data in some shape or form, however, only 19% are processing the entirety of their data. Those which do leverage solutions to analyse their vast data set are mostly organisations which entered the game early on, over three years ago.
While smaller, more agile companies may be the ones already getting the most from Big Data, the allure of the concept to big business is evident from the fact the majority of organisations that have implemented Big Data initiatives, tentatively or full scale, come from the 500+ employees’ category. Companies whose scale alone suggests a high probability of a large volume of different types of data understandably believe they have a great deal to gain from Big Data, even if the vast majority have not yet developed solutions which work at the high speed required to process their information.
As an organisation becomes more confident with using Big Data, it is often the case that new and/or larger-scale initiatives (like developing new revenue streams and driving innovation) come into play. Most respondents expect to get the most value out of Big Data with regards to improving their current operations. Those quizzed by Whishworks singled out product marketing strategy, revenue and processes as the areas which will likely benefit them the most.
Data investments are also intrinsically tied to Return on Investment (ROI), as companies at the beginning of their Big Data journey hope to see faster returns via optimisations of their existing operation, reducing risk to their investments in the process.
Whether a company’s projects actually deliver on the potential of Big Data or not remains to be seen, however, as success is by no means a certainty. Thanks to the fact Big Data is a fairly new concept, the distributed processing technologies leveraged to analyse it are still rapidly evolving, and with them, the way organisations operate and compete with one another.
In such an environment it is easy to misjudge the launch of a Big Data project, or to be outflanked by nimble competitors. Skillsets are also scarce, particularly in the UK. A talent shortage due to tightening border controls and an ageing population, inflamed by a recent prediction from IDC – that the Big Data technology and services market will see record growth, reaching $58.9 billion by 2020 – provoking a surge in demand for data processing and analysis talent, mean many companies are desperately waiting for a new generation of graduates to join the workforce with more relevant qualifications.
As a result, organisations are rarely completely ready to undertake the challenge Big Data brings into their operations. A minority of 30% find their Big Data operations to be an outright success, while seven in every ten respondents of Wishworks’ poll said that their initiatives had been only somewhat successful so far. Despite the challenges, however, organisations considering a new Big Data initiative can take heart from the fact not one participant in the survey was willing to describe their efforts as “not successful”.
Room for improvement
Respondents identified a number of key areas which could lead to an improved implementation of Big Data. As previously mentioned, these were dominated by lack of skillset, at 71%. Relating to the problem of attracting new talent, firms also said their current crop of staff tended to suffer from a skills gap. 65% of companies said that limited understanding of Big Data across their organisation had hampered their efforts. Cost and data integration were also factors regarded as chief challenges.
More than half of respondents also noted that cost was a key concern, and it is easy to see why. It is difficult for firms to effectively estimate the cost of a Big Data project, and so it is easy to run out of budget fast due to wrong projections. Acquiring new hardware or paying a cloud provider could all lead to a budgeting shortfall, while a company with an on-premise solution will also need to consider the entire cost of ownership including cost of training, license renewal, maintenance, upgrades and expansion.
The pressure on firms to acquire top talent also plays into this. Big Data projects require sophisticated teams of developers, architects, data scientists and analysts who are well-heeled in a number of technologies, including Apache Hadoop, Apache Spark, NoSQL Database, Data Science, Analytics and Data Visualisation. Obtaining these skillsets does not come cheap amid a tightening market for human resources.
47% of respondents said the integration of Big Data into their business model remains an area of difficulty, partially due to the lack of an adequately trained workforce at most companies. Relating to this, 47% of respondents also noted that there was no shared data strategy or vision at their firm – something which is likely at the root of their concerns around integration.
An area which has remained a perennial concern for years, and may yet come to top the list in the wake of GDPR’s launch in May 2018, is security and governance of Big Data, currently rated at 35%. The EU’s landmark privacy legislation could see corporate entities fined as much as €20 million, or 4% of global revenues, should they be found to abuse the data of their customers, or should they fail to adequately guard it from hackers. In terms of business-related challenges, concern in this area looks likely to grow, as 41% of respondents named changing business requirements as a top issue, while a further 35% said a lack of data governance was a major worry.
Commenting on what may come next for companies looking to leverage Big Data, Edward Davies, Sales Director at Whishworks, said, “As companies gain more understanding of how they can benefit from Big Data, the rate of adoption is going to increase exponentially. In 2018, we expect to see companies to process more of their data with Big Data technologies and start implementing more complex and/or impactful use cases. It is also going to be interesting to see how different industries will evolve taking advantage of the new opportunities Big Data presents, and what new ‘disruptors’ will surface.”
source: https://www.consultancy.uk/news/16839/70-of-big-data-projects-in-uk-fail-to-realise-full-potential
