How to Recover from an Out of Disk Space Exception

As IBM Manta Data Lineage persists a lot of metadata, the disk (or disks) Manta Data Lineage utilizes can run out of free space. Depending on which state Manta Data Lineage is in when this happens, this situation can have different impacts and should be resolved accordingly. This article summarizes the different situations that can occur and how it is possible to recover from them.

Instructions

This article presumes that it is desirable to lose as little metadata as possible while recovering from the exception. In some situations, though, the most straightforward approach could be to truncate all the metadata—this can be helpful, for example, in a testing environment.

Truncate All Metadata

• Manta Flow Server—Data lineage repository

  • If Manta Flow Server is running:

    • Execute cleanRepositoryScenario through Manta Orchestration API Scripts

  • If Manta Flow Server is not running (and it is not possible to start it up due to the out-of-space exception):

    • Delete the folders:

      • <MANTA_HOME>/server/manta-dataflow-server-dir/data/sourcecode

      • <MANTA_HOME>/server/manta-dataflow-server-dir/data/neo4j

• Manta Flow CLI

  • Delete all content from the folders:

    • <MANTA_HOME>/cli/temp/

    • <MANTA_HOME>/cli/output/

    • <MANTA_HOME>/cli/log/

• Manta Admin UI

  • Stop the service (Manta Utilityservice)

  • Delete all files matching the following patterns

    • <MANTA_HOME>/serviceutility/manta-admin-ui-dir/data/process_monitor*

    • <MANTA_HOME>/serviceutility/manta-admin-ui-dir/data/manta_logs*

By performing these steps, a significant amount of disk space should be freed. After that, it should be possible to start both Manta Data Lineage services and continue using Manta Data Lineage.

Keep All Valid Metadata

If it is not desirable to truncate all the metadata (e.g., in a production environment), it is still possible to recover from the situation. The recovery scenario is, however, a bit different for the different states that Manta Data Lineage can be in when the out of space exception occurs. The sub-chapters below describe what actions should be performed to get Manta Data Lineage into a consistent state again.

One of the steps is to “provide more disk space.” This can be done in several ways such as by enlarging the disks that Manta Data Lineage is utilizing, saving some disk space by deleting unnecessary files (e.g., old Manta Data Lineage logs or files not related to Manta Data Lineage), or redirecting a metadata persistent storage entity (e.g., the data lineage repository or the folder containing scripts) to a different disk drive. This is for the Manta Data Lineage administrator to decide. This step will further be referred to simply as “providing more disk space.”

Exception Occurs While Manta Data Lineage Is “Idle”

In this scenario, no Manta Data Lineage scenario/workflow is running, and Manta Data Lineage is thus in a stable state. That does not mean it is completely “idle.” Users can still access Manta Flow Viewer and visualize the data lineage there. The recovery scenario consists of the following steps:

1. Stop Manta Data Lineage services.

2. Provide more disk space.

3. Start Manta Data Lineage services.

Exception Occurs during Extraction

If an exception occurs during the extraction (meaning that the data lineage analysis has not started yet and a new revision of the Manta Data Lineage repository has not been created yet), the recovery scenario is fairly straightforward.

1. Stop Manta Data Lineage services.

2. Provide more disk space.

3. Start Manta Data Lineage services.

4. Re-run the extraction (end any consequent scenario that was initially expected to be run).

Exception Occurs during Analysis

If the exception occurs during the data lineage analysis (meaning that a new revision of the Manta Data Lineage repository has been created, but not committed yet), the Manta Data Lineage repository ends up in an inconsistent state and the latest (opened) revision has to be rolled back / committed so that the analysis can be executed again.

1. Stop Manta Data Lineage services.

2. Provide more disk space.

3. Start Manta Data Lineage services.

4. Resolve the Manta Data Lineage repository revision inconsistency.

   1. Roll back the latest revision by running rollbackRevisionScenario through Manta Orchestration API Scripts. This can be a time-consuming operation, but the benefit is that the “incomplete” revision will never be accessible in Manta Flow Viewer.

   2. Commit the latest revision by running commitRevisionScenario through Manta Orchestration API Scripts. Once the revision is committed, it is available to end-users in Manta Flow Viewer. As the metadata in this revision is probably incomplete, this is most probably not desirable.

5. Re-run the analysis (and any consequent scenario that was initially expected to be run).

Exception Occurs during Export

The recovery scenario for the situation where the error occurs during export is very similar to the recovery scenario for the extraction phase. There is one difference, however. Some integrations are split into two phases—the export and the upload. If this is the case and if all the export scenarios are finished successfully (and it is the upload phase during which the error occurred), it’s not necessary to re-run the export, it is sufficient to re-run the upload scenarios.

In general, the recovery scenario consists of the following steps.

1. Stop Manta Data Lineage services.

2. Provide more disk space.

3. Start Manta Data Lineage services.

4. Re-run the export/upload.

Related Articles

• How to Measure Disk IOPS Values