Egeria open source standard enhances hybrid cloud metadata and data governance initiatives

IBM Watson Knowledge Catalog — a true hybrid cloud data catalog

By | 4 minute read | September 23, 2020

Standardization has changed the way we work with technology. Whether TCP/IP, USB or XML, open standards have given us seemingly endless ways to use technology more effectively. When you’ve got it, standardization is a wonderful thing; when you don’t, things can unravel quickly. An example that resonates with many of us is the lack of global standards for electrical outputs. There are currently at least 15 different types of power outlets around the world. This has led to the travel adapter being an essential piece of additional equipment we must pack for our business trip or vacation.

The lack of global standards creates significant costs not only for the traveler who needs to match the right plug to the right socket, but for manufacturers who must build and supply a variety of products to meet the specifications of each country where their products are sold. Hotels must try to accommodate the forgetful traveler by installing multiple types of power outlets in their rooms. What has occurred however is the introduction of a universal adapter — providing an interface between the power outlet of the country I visit and the plug of the electronic device. Now wherever I travel, I just need to make sure I have my universal interface and I can be confident that I will have all I need to power up my electronic devices.

Now let’s make a leap to the world of metadata. Metadata underpins everything we do with data, analytics and AI. It identifies a wealth of important information, including data content, quality, sources and how it can be used. Similar to power outlets, there is no country- or vendor-specific standard as to how that metadata should look. Each product or vendor has its own proprietary way to describe metadata used by its software platforms. That’s generally okay as long as the platform remains “in country” on a single cloud platform. However, what happens if I want to work with data in other “countries” or on a different cloud? All of a sudden, I need to understand all these different formats of metadata before I can integrate and understand the data.

Standardization is particularly important for data catalogs and metadata management systems. The purpose of these tools is to provide a central index of metadata to govern data as well as allow data citizens to discover and use data more easily. For a typical enterprise, data modernization strategies are underpinned by successful data catalog strategy, and for that to work those catalogs need to be able to understand data wherever it resides across the entire data landscape — from on-premises to private cloud to any public cloud. A lack of standards for metadata quickly becomes a problem for data catalog vendors and business leaders who attempt to implement those data catalogs. What we need is a metadata adapter, something that bridges the format of the metadata on one cloud — the power outlet — and the format that is used by the data catalog — the plug.

At IBM we believe that it is essential for our clients to be able to operate in a multicloud ecosystem, across data they hold on premises, on private cloud or on any public cloud. We believe that clients should be able to govern data wherever it resides and serve the needs of the data citizens to support their analytics and data science projects. We call this the hybrid cloud. It is therefore imperative that there is a universal adapter for metadata. This universal adapter needs to be open so that all vendors can contribute and ensure that the metadata they hold in their own proprietary formats can be used via the universal adapter without “vendor lock-in.”

For this reason, IBM® embraces the Egeria Open Metadata and Governance project as our worldwide travel adapter for metadata. Egeria provides open APIs, event formats, types and integration logic to help organizations share data management and governance responsibilities across the entire enterprise without restricting data to a single format, platform, or vendor product. Our unified data and AI platform IBM Cloud Pak for Data and its capabilities such as IBM Watson Knowledge Catalog will catalog data from other Egeria compliant metadata repositories. Clients can now share metadata easily, and avoid being locked into proprietary formats requiring additional, expensive integrations. In the multicloud era, data modernization is a strategic imperative – and this starts with a data catalog that supports open standards.

Learn how IBM Cloud Pak for Data and IBM Watson Knowledge Catalog can support your data modernization initiatives across multiple clouds today.

Watson Knowledge Catalog expands governance capabilities to include data quality solutions. The end-to-end data catalog was named a Leader by the 2020 Gartner Magic Quadrant for Data Quality Solutions. Read the report.

Related reading: IBM’s open source strategy champions AI trust and transparency