Denodo Senior Vice President and CMO Ravi Shankar discusses the concept of logical data fabrics and its effectiveness in addressing modern data sprawl.
By Ravi Shankar
Business organizations have had to accelerate and strategize to identify, capture and optimize the use of data in business decision-making in tandem with the rise of the digital economy. The current global health crisis has proven to be a catalyst in putting data front and center in the way businesses operate today, and processes and systems are fast streamlining and becoming data-dependent, data-driven.
However, with valuable data existing in varied formats and quality levels, the rise of data sprawl – the overwhelming amount and variety of data produced – has posed a real challenge for many organizations. That is, organizations often struggle to efficiently convert and make full potential use of their data, stuck in disparate silos, and turning it into intelligible insights. According to a recent IDC report, more than 80% of IT leaders reported data sprawl as one of the most critical problems they face today.
Nonetheless, data management has evolved significantly since the 1970s, when we first stored and accessed data by interpreting it in different spindles and punch cards. Here, I introduce the concept of logical data fabrics as the modern solution to data sprawl. The technique that allows data to be more efficiently converted and used for business purposes.
Computer scientists have created centralized repositories for storing data, from the concept of data warehouses in the 1990s to data lakes in the past decade. Mainly, for business organizations to achieve a unified view of information across applications and databases.
This approach, while attempting to go against data gravity – which pulls the data towards its sources and away from central repository – work only for some period of time. Computer scientists realized that data quickly moved apart into decentralized components as new data continuously sprouted in other locations.
Rather than forcefully consolidating the data into a central physical repository, a new data management paradigm called logical data fabric has emerged. Much akin to the weaving of textile fibers to produce cloth, logical data fabric knits a virtual view of data across applications by leaving data at the sources where they are created, while allowing a unified view of all enterprise data.
What makes the data fabric successful is that data virtualization forms its core technology and many of its capabilities are automated through Artificial Intelligence (AI) and Machine Learning (ML).
The concept of a data fabric has increased in importance as organizations increasingly store data in multiple cloud-based platforms, which can add to existing on-premises data silo problems.
Specifically, logical data fabric enables organizations to stop ‘collecting’ their data into a central repository and start ‘connecting’ to the data at the sources through data virtualization. Regardless of the data’s location – on-premises or in the cloud; format – structured or unstructured; or latency – data in motion or data in rest, data fabrics free up the data to be innovated at its sources while bringing it together in a virtual fashion for the benefits of data discovery, management, and governance. It hence operates according to the concept of data gravity.
This way, logical data fabric greatly improves the efficacy of business data users. For one, without needing to move data from its source into a temporary repository, IT teams no longer need to program ETL scripts for the purposes of converting data before loading it into target systems. Data virtualization can perform transformations on the fly, which saves on storage costs.
Furthermore, data virtualization’s low- to no-code approach significantly reduces the manpower and effort involved to technologically develop the unified views that organizations demand.
More importantly, it serves as a virtual catalogue for all data assets within the enterprise, including the sources of data origins, its format, and its relationship and association with other data assets. Through this catalogue, business users are not required to access different systems to perform actions such as data discovery or documenting business definitions for the purposes of data governance.
In addition, logical data fabric possesses powerful data preparation capabilities to normalize data formats and simplify them for business consumption. Business users will then be able to easily access the data within their favorite analytical, operational, web, or mobile applications.
As mentioned above, logical data fabric is already embracing AI and ML to automate some routine tasks, and this is set to intensify moving forward. At most business organizations, data is constantly evolving, new data sources are routinely added, and new forms of data are being innovated. AI and ML analyze changing data patterns and automatically integrate new data for unified views, delivering them in the most appropriate formats to business users. The incorporation of AI and ML technologies allow organizations to better understand consumption behaviors of certain users and share the availability of new data sets for further analysis and collaboration.
There is no doubt that the benefits of AI and ML include raising efficiency and productivity levels among data scientists working on related projects. These data fabrics will enable them to optimize data models for deeper analytics and insights that benefit the organizations, thereby improving their performance.
The logical data fabric is no longer just a concept. Their adoption has been gaining speed, thanks to the tremendous benefits outlined above. By adhering to the principles of data gravity, providing business friendly views of enterprise data in its entirety, and automating using AI and ML, data fabrics will be one of the hottest data management trends in 2020 and beyond.
(Ed. Ravi Shankar is responsible for Denodo’s global marketing efforts, including product marketing, demand generation, communications, and partner marketing. Shankar says he has an MBA from the Haas School of Business at the University of California, Berkeley.)