Let's talk about data paradox...

The way data is generated and used has changed over time. Because of this, the data has been gone considered the new oil, as it can provide unimaginable insights and competitive advantages to any company that first knows how to refine this material and extract the best essence from it. But now, the ability to analyze and gain insights through data is the new oil! It's not enough just to have the data.

“Data is the new oil. It's valuable, but if unrefined it cannot reallybe used (...) so must data be broken down, analyzed for it to have value” (Humby)

As [1] mentions, In 1996 only 0.8% of the data produced was stored in digital format, different from 2007 when 94% of the data was stored in digital format.

The large amount of data generated in a very short time and in the most varied formats, instead of simply providing an advantage for those who own it, brought a great challenge to companies, becoming a burden for many. The 2020 Digital Transformation Index showed that data overload and inability (whether due to financial limitations, tools or trained professionals) are growing barriers to the transformation of companies into data-driven companies, as they make it difficult to extract useful information and relevant, which can lead to wrong conclusions and create the data paradox.

Additionally, the data paradox can arise when decision makers rely heavily on data without considering other factors such as context, intuition, or subject expertise. In 2021, a study commissioned by Dell Technologies with Forrester Consulting, called Data Paradox, was published, seeking to deepen to understand what actually prevented companies from transforming data into insights for the business and, how and if companies are prepared to the increasing deluge of data. For this, 4,036 directors/decision makers responsible for data strategies and digital transformation in several companies around the world were interviewed. And in this article, we will focus on the data presented in this research.

The search results...

Research has shown that 83% of companies face at least one of the barriers below, proving that most companies are currently disabled for digital transformations:

• Data warehouse that is not optimized;

• High storage costs;

• Outdated IT infrastructure;

• Manual processes that don’t meet business needs.

Companies are spending money and effort to evolve technologically while trying to compete in the market. More than half (55%) failed to come close to achieving their digital transformation goals.

The survey showed that the large amount of data already is or has become a burden for many companies, where 67% say that data collection is much faster than their own ability to process, analyze and use this data. And this scenario worsened with the pandemic, where there was a considerable increase in data generated.

It is possible to understand a little of this difficulty of companies through the numbers below:

“Knowledge is knowing that a tomato is a fruit. Wisdom is knowing not to use a tomato in a fruit salad.” (Charliton Albert)

Just having a large amount of data does not mean a competitive market advantage if companies do not have the capacity (technological or human) to extract information and be able to draw customer profiles and predict future scenarios through prediction algorithms.

But these are not the only problems faced by companies. Most stated that they constantly need more data on the other hand the use is still very low.

And the consequences of this paradox are clear... Most say their teams are overloaded and have too much data to be able to ensure security and compliance requirements.

Further complicating this equation is the problem most companies face which are data and business silos and only 17% of them are executing initiatives to promote the democratization of data.

Another problem is the level of technical knowledge of the teams. Most companies report that their teams have insufficient knowledge in the field of data science. Even though this is one of the main current problems of the majority of companies, very few are taking any action to change the internal scenario, either through hiring professionals with specific skills or incentives for training and development of the necessary skills in professionals.

To get a more realistic view of data usage scenarios within the company, all respondents were classified by separating them into 4 groups.

• DATA NOVICES - score poorly in both technology & process and culture & skills categories ;

• DATA TECHNICIANS- focus their efforts on technology and process, to the exclusion of culture and skills development;

• DATA ENTHUSIASTS - emphasize culture and skills; less on developing a technology backbone;

• DATA CHAMPIONS - score highly in both technology & process and culture & skills categories;

Here in Brazil, we are very similar, 48% of companies are in the initial stage, that is, they are new to data. But the bright side, for Brazil, is that 23% of companies are data champions, surpassing the global average.

For the survey 88% of companies are neglecting their technology and processes, culture and skill, or both.

Despite the worrying present, the survey identified a promising future as most companies have clear goals for the next 3 years that will make them more agile and adaptable to technological changes.

• 66% plan to deploy machine learning to automate anomaly detection ;

• 57% intend to move to a data as-a-service model ;

• 57% plan to improve the data lakes that they have.

The conclusion

The conclusion that the research brings us is that companies need access to cutting-edge technology to overcome internal and external data silos, processes to search for anomaly data and make meaningful discoveries, and data-driven skills and culture to work with data In real time to predict the future.

Reference

[1] MARQUESONE, Rosangela. Big Data: Técnicas e tecnologias para extração de valor dos dados. Editora Casa do Código, 2016.

[2] Data paradox - https://www.delltechnologies.com/asset/en-ca/solutions/infrastructure-solutions/industry-market/data-paradox-research-findings.pdf