A scant nine percent of businesses have woven big data and analytics into their organization’s DNA to where’s it’s central to strategy, decision-making, execution, investments and revenue generation, all supported by an executive advocate, a new study found.
AtScale, a big data analytics specialist, conducted the three-year study of aligned professionals at 429 companies to assess the level of big data maturity. The research, which elicited about 5,600 responses on the use of big data and the cloud, revealed that the cloud is center stage, enterprises face continued self-service challenges, and cloud heavyweights Microsoft and Google are making an impact.
The study’s results showed companies fall into three categories along the big data and analytics continuum, with the most significant termed at a “transformational” stage, a slight majority at the "experimental" phase, and a third smaller group at the “strategic” step. AtScale defined the transformational stage -- that’s the nine percent mentioned above -- as those companies building specific apps to monetize big data along with a path to move from an enterprise data warehouse (EDW) to a big data lake, or a repository for huge volumes of raw data not defined until it is needed.
AtScale classified about 39 percent of the companies in the survey as in the “strategic” phase of development, having committed to a big data platform such as Hadoop, On Premise, Non-Hadoop in Cloud, such as Google BigQuery, Microsoft Azure or Amazon RedShift/EMR). IT teams in these organizations are using big data to augment their EDW, and business users can self-serve securely to the data lake.
Roughly 51 percent of the respondents are experimenting with big data, the findings showed, concentrating on data offloading and cost-saving benefits. At this point in their big data position, they focus on use cases in development with low expectations of big data’s benefits. Of note, the experimenters typically lack C-suite support for big data initiatives.
Some statistical findings from the the study:
Optimism: 95 percent of the respondents expect to do more or the same with big data in the next three months.
Strategic: 66 percent position big data as strategic/game changing to the organizations, while 34 percent regard it as experimental/tactical.
Challenges: The top big data challenges to organizations in 2018 are expected to be skills, governance, performance, security and management, in descending order. Last year, performance was at the bottom of the list and governance was third.
Hybrid environments: Only 20 percent would replace earlier platforms with big data, suggesting the need to build an agile data environment that can accommodate traditional BI platforms in concert with a modern data environment.
Maturity: In 2018, 78 percent of respondents ranked their big data maturity as “medium” or “high.” AtScale’s data, by contrast, pegged the percent of high-level maturity at a mere 12 percent, up eight percent since 2016.
Self service: 55 percent of respondents are operating in siloed, decentralized analytics teams. Roughly 34 percent are working in a centralized data science and analytics team.
Data access: The cloud has made data access harder: 59 percent of respondents have deployed big data to one degree or another in the cloud, up six percent from last year. But users’ access to data has declined with self-service access at 42 percent, a five percent slide from last year back to levels of two years ago. Also, 58 percent of respondents don’t have self-service to big data.
The cloud: About 76 percent of respondents said they would use the cloud for big data with Google BigQuery. Forty percent said would favor a cloud option over on-premise, 11 percent are planning to put Google BigQuery in production while 60 percent are mulling it over.
The top 3: The top BI tools for big data are: Tableau, Microsoft Excel and PowerBI, in that order. PowerBI moved up from seventh place last year.
AtScale’s partners in the survey included Cloudera, Hortonworks, MapR, Tableau, the Linux and Apache Foundation.
