Adding and configuring an Airflow integration

Adding and configuring an Airflow integration consists of the following phases: adding an Airflow integration, connecting the integration, and adding alerts. All of the phases are performed in the Databand UI but connecting the integration is also partially performed in the Airflow UI.

Adding an Airflow integration

To add an Airflow integration:

  1. Go to the Databand UI navigation menu and click Integrations.
  2. Click Add integration.
  3. Select Airflow as the integration type.
  4. In the Set up integration step, complete the details for the Airflow integration. The step consists of required and optional fields. Make sure that you provide the Integration name (it needs to be unique) and the Airflow URL. For Standard Apache Airflow getting the URL is rather straightforward. For Astronomer, Amazon Managed Workflows for Apache Airflow, Google Cloud Composer, see Special considerations for managed Airflow providers.
  5. Optional: Click the Get notifications checkbox and provide up to 3 emails. These emails receive notifications if a problem occurs with your integration.

The Advanced settings section is completed by the Databand administrators.

Connecting an Airflow integration

This step provides a ready-to-use JSON file and instructions on how to use it to enable communication from Airflow to Databand.

Save the JSON file for future reference.

  1. Create a new connection in the Airflow UI. To do so, in the Databand UI click the Airflow Connections page hyperlink, to open the Airflow UI.
  2. Copy the Key value from the Databand UI and paste in into the Airflow UI in the Connection Id field.
  3. In the Airflow UI, select the Connection Type: HTTP.
  4. In the Airflow UI Extra field, paste the JSON configuration and click Save.
  5. Go back to the Databand UI and select the I applied this configuration in Airflow checkbox.
  6. Before you click Connect, make sure that the Databand Airflow monitor DAG is enabled.
  7. Click Connect to confirm the connection was set up properly.

If the integration was created successfully, you can see a confirmation window. The integration is also displayed on the list of integrations in the Integrations window. Its status is set to running. If the integration wasn't created, the system displays the message that the monitor is not working properly because of the failed connection.

Adding alerts (optional)

You can configure automatic alerts on your pipelines. A stand-alone alert definition is created for each alert type selected, and can be later edited by the user. The alert definition is applied against the entire Airflow cluster and all off of its DAGs. Automatic alerts are created with severity High.