Integration with off-platform assets

Develop an integration to observe assets managed by tools that are not part of watsonx.data integration (for example, Databricks, AWS Glue, or Apache Airflow).

If your system already generates and emits events in the OpenLineage format, you can integrate directly by using the OpenLineage API. For more information, see OpenLineage integration for off-platform assets.

Another example of an integration with off-platform assets is integration with Apache Airflow to observe DAGs. DAGs are equivalent to jobs in watsonx.data integration. You can either integrate it manually by creating an API payload or by installing the OpenLineage Airflow Provider. For more information about Apache Airflow integration, see Integrating with Apache Airflow.

You create alerts in the same way for jobs that are managed by watsonx.data integration and external tools. For more information about how to create alerts, see Data observability alerts.