What is data intelligence? 

Put another way, data intelligence helps organizations answer core questions about their data, including:

• What data does the organization have? Why does this data exist?

• Where did the data come from and where does it reside? 

• Who is using data? How are they using it—and how should they use it for best results?

• How are distinct datasets related to each other?

Data intelligence answers these questions by using an interconnected set of processes and tools to automate and streamline metadata management, data discovery, data governance, quality assurance, data analysis and other activities.

As much as 68% of enterprise data is never analyzed, according to the IBM Data Differentiator. Because of the sheer amount of data at their fingertips, organizations can struggle to apply quality controls and enforce governance policies. Users can’t always find the right data for their work—and might not even know when it exists. 

Data intelligence emerged to address this problem by uniting existing tools—such as data catalogs, data lineage solutions, data marketplaces, artificial intelligence (AI) and machine learning (ML)—in a single, comprehensive process.  

This unified process gives organizations more insight into their data and how to get the most value from it. In this way, DI enables self-service analytics and supports key initiatives such as business intelligence and generative AI.

Data management is a broad discipline that oversees the entire data lifecycle from creation to disposal. Whereas data management is concerned with the practicalities of collecting, storing and processing data, data intelligence is about understanding that data.

Data intelligence complements data management by giving organizations the insights that they need to make more informed choices about capturing, securing, cleaning and sharing data.

Since the dawn of Web 2.0 and the rise of cloud computing, organizations have been collecting more data (customer data, operational data, transactional data) from more data sources (web apps, business systems, Internet of Things devices). The birth of generative AI has only increased the value—and amount—of all this data.

Managing this data—tracking how it is used and how it changes, storing it securely, facilitating access, keeping it clean and up to date—can be difficult. If data is not properly managed, it can be hard for data consumers to find the data they need, much less derive actionable insights from it.

Organizations have long had the capabilities to manage data—data lineage tools to map end-to-end data lifecycles, governance tools to define use policies, data profiling and cleaning tools and so on. However, these capabilities were often fragmented, scattered across disparate products and functions.  

The primary innovation of the data intelligence discipline is to bring these tools together with advanced AI and ML technologies, either in a single platform or in a tightly integrated data stack.  

According to IDC, many of the current data intelligence platforms evolved from data catalog tools. Since 2020, vendors have increasingly bundled their catalogs with complementary solutions, such as data lineage tools and data marketplaces, or built these functions directly into their catalogs.¹

Data intelligence is a developing field, with different vendors and practitioners presenting their own takes on the discipline. However, most agree that data intelligence includes five core functions:

• Metadata management
• Data lineage
• Data governance
• Data quality
• Data integration

Metadata is information about a data point or dataset, such as file author or size. Metadata management is foundational to data intelligence initiatives because well-managed metadata helps users easily navigate complex data systems.

Metadata management helps organize, label, filter and sort datasets so users get a full picture of the data available to them and can quickly retrieve the information they need.

Active metadata management is particularly important to data intelligence. Whereas traditional metadata management is largely manual, active metadata management uses AI and ML to automate metadata processing.

As data is transformed and used, its metadata can change. Active metadata management tracks these changes, automatically updates metadata and uses metadata to generate recommendations and alerts. In this way, it can streamline data discovery, improve confidence in data and enable data protection and governance at scale.

Data lineage is the process of tracking the flow of data over time. It provides a clear understanding of where data originated, how it has changed and its ultimate destination within the data pipeline.

Data lineage helps users understand how data changes throughout its lifecycle, making enterprise data more reliable. It also helps organizations detect errors, identify dependencies and anticipate how changes to a dataset might affect broader enterprise operations and IT systems.

Data governance helps ensure data integrity and data security by defining and implementing policies, standards and procedures for data collection, ownership, storage, processing and use.     

Data governance helps maintain safe, high-quality data that is easily accessible and compliant with relevant rules and regulations. In data intelligence efforts, governance policies help users understand how they can and should use data.  

For example, governance policies can prevent data scientists from feeding sensitive customer data to AI models in violation of data privacy laws.  

Data quality tools and practices help ensure a dataset’s accuracy, completeness, validity, consistency, uniqueness, timeliness and fitness for purpose. Data quality efforts build users’ trust in the conclusions and insights that they draw from enterprise data.

DI initiatives often include master data management (MDM) as well. Master data is an organization’s core data on key business entities, such as customers, products and locations. MDM ensures that this data is clean and consistent through validation, merging, deduplication and enrichment.  

Data integration is the process of combining and harmonizing data from multiple sources to facilitate its use for analytical, operational and decision-making purposes. Integration can involve standardizing data formats, transforming data into more usable formats and bringing together data from disparate sources in shared data lakes, data warehouses or data lakehouses.  

Data integration streamlines data access and data sharing, making it easier for data consumers to retrieve the data they need and collaborate with one another.

Some vendors offer data intelligence platforms that combine various features and functions in a single solution. Others offer integrated portfolios of complementary solutions. In either case, the fundamental tech tools behind most data intelligence initiatives include: 

• Data catalogs
• Data lineage tools
• Data marketplaces and product hubs 
• Artificial intelligence and machine learning
• Data lakes, warehouses and lakehouses

A data catalog uses metadata to create a detailed, searchable inventory of all data assets in an organization. This makes it easy for data consumers to discover the most appropriate data for any analytical or business purpose. 

Beyond inventorying data, many modern data catalogs feature capabilities such as:

• Data governance mechanisms, including the ability to set and enforce data usage and data privacy policies, such as by automatically redacting sensitive information.

• Active metadata management, by using AI and ML to automatically generate metadata and update records as data changes.

• Business glossaries, which allow organizations to create standard definitions and frameworks for key terms, concepts and core entities across the organization.

• Data quality controls, such as data profiling, cleansing, validation and quality metrics.

Data lineage tools automatically map data flows, transformations and dependencies, offering key insights into data lifecycles. Data lineage solutions allow organizations to see where data comes from, how it moves through the enterprise IT ecosystem, how it changes and how data consumers use it.

Data marketplaces, also called data product hubs, are digital platforms where users can access and share data products.  

Data products are prepackaged, preprocessed, readily consumable sets of data or data-related assets that people can use to support BI, analytics and data science efforts. Examples of data products include curated datasets, analytics dashboards, machine learning models, specialized applications and data visualizations.

Data marketplaces centralize and streamline the creation, curation, management and sharing of data products. Data marketplaces help ensure data quality and compliance with integrated governance frameworks. They also break down data silos by automating data product delivery and enabling large-scale sharing of data products from disparate sources.

AI and ML tools, new generative AI applications and large language models (LLMs), help elevate data intelligence practices beyond traditional data management. Whether as stand-alone solutions or built into other tools, AI and ML can automate data and metadata enrichment, streamline data mining and enable advanced AI data management.

For example, an integrated LLM can automatically generate and update metadata in a data catalog, providing more user-friendly explanations to make data more accessible to more stakeholders. Natural language interfaces powered by LLMs let users query datasets and surface data insights without needing to use structured query language (SQL) or other specialized languages.

AI tools can also help enforce governance policies and quality controls, such as by discovering and classifying sensitive data or identifying duplicate datasets.  

Data lakes, data warehouses and data lakehouses are data management and storage solutions with different features and functions.

• Data warehouses aggregate, clean and prepare data so it can be used for business intelligence and data analytics efforts.  

• Data lakes store large amounts of raw data at a low cost.  

• Data lakehouses combine the flexible data storage of a lake and the high-performance analytics capabilities of a warehouse into one solution.

Warehouses, lakes and lakehouses support data integration efforts by enabling organizations to bring data from different sources together in centralized sores. They also make it easier to access and use that data for analytics, BI, AI, ML and data science applications.

Data intelligence helps organizations:

• Understand their data through comprehensive data catalogs, data lineage tools and active metadata management. 

• Facilitate data access through searchable data catalogs, integrated data stores and centralized data product hubs. 

• Ensure data quality through automatically updated metadata, data profiling and cleansing.

• Guide data usage through defined governance policies and data product hubs that host curated assets for specific uses.

As a result, organizations can reap benefits such as:  

Data intelligence promotes data literacy and enables self-service analytics by giving users the insights they need to understand and use enterprise data. Stakeholders at all levels and in all roles can use data to make more informed decisions.

The IBM Data Differentiator reports that 82% of enterprises experience data silos that stymie key workflows. Data intelligence helps eradicate these silos and reduce data infrastructure complexity through centralized, unified data catalogs and marketplaces.

Users throughout the organization can find the right data for their purposes, streamlining operational efficiency and boosting collaboration.

According to Gartner, organizations lose an average of USD 12.9 million due to poor data quality.² Through data lineage, data profiling and governance efforts, data intelligence maintains high levels of data quality so organizations can get more value from their data.

Data intelligence integrates governance frameworks into key data access points, such as data catalogs and data marketplaces. This helps ensure that data consumers use data only for authorized purposes, protecting against hacking, theft, misuse and noncompliance. Governance is especially important for highly regulated industries such as finance and healthcare.

According to the IBM Institute for Business Value, 72% of top-performing CEOs agree that having the most advanced generative AI tools gives an organization a competitive advantage. And advanced generative AI requires massive amounts of high-quality, readily accessible data.

Data intelligence helps improve data quality, facilitate access and enforce governance policies to ensure that data is used only for the right purposes, a core part of responsible AI. 

One particular use case for data intelligence is in the realm of AI model intelligence. Model intelligence is the practice of understanding, managing and governing the lifecycles of the various AI and ML models in an organization’s portfolio.

Rather than relying on a single model, many organizations today use various models for different ends. Data intelligence initiatives give organizations the transparency that they need to select the right data for the right models for the right reasons.  

Specifically, data intelligence can help organizations select the right data in terms of both governance—is this data authorized for use in this model?—and fitness—is this data accurate and relevant enough for this model?

Moreover, many vendors are incorporating model management functions into their data intelligence offerings. For example, some data catalogs are introducing model catalog features, allowing them to inventory an organization’s AI and ML models the same way they inventory enterprise data. 

Data intelligence is a way of understanding the data that an organization has—its defining features, how to access it and how to use it. Data analytics, data science and business intelligence are ways of using that data.

• Data analytics extracts actionable insights from data to make better decisions. Data analytics can take many forms, such as predictive analytics—using data to make predictions about the future—and prescriptive analytics—using data to determine what to do next.

• Data science is a specialized discipline combining math, statistics, programming, advanced analytics, AI, ML and subject matter expertise.  

• Business intelligence (BI) refers to the tools and techniques people use to collect, manage and analyze enterprise data to inform business operations.

Data intelligence facilitates data analytics, data science and BI by helping users better understand and use their organizations’ datasets. When users know what kind of data the organization has and what it can be used for, they can more easily connect with the right datasets for their purposes.  

For example, data scientists can find high-quality, compliant data to train machine learning algorithms; BI users can find curated datasets tailored to their specific domains.