Restoring corrupted H2 database

Note: To prevent corruption of H2 databases, upgrade to IBM Automatic Data Lineage R42.8 or later.

Problem

If you’re working with Automatic Data Lineage R42.0 - R42.7, embedded H2 databases might become irreversibly corrupted for the following services:

If you want to work with these services, you must restore the corrupted databases.

Keycloak

Symptoms

If the H2 databases are corrupted, Keycloak is down and you can find entries similar to the following one in the logs.

Example:

2024-09-10 08:28:21,147 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': IO Exception: "/apps/mantaflow/keycloak/data/h2/keycloakdb.mv.db" [90028-230]
2024-09-10 08:28:21,151 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null


Solution

  1. Stop and restart Keycloack.

    If you can still the error message in the logs, procced to step 2.

  2. Stop all Automatic Data Lineage services.

    • Linux: Use your service management scripts or commands.
    • Windows: Use the Windows Services Manager or relevant scripts.

  3. Change directories to the keycloack.

    • Linux:

       cd $MANTA_HOME/keycloak

    • Windows:

       cd %MANTA_HOME%\keycloak

  4. Create a back up version of the data folder.

    • Linux:

       cp -r data data_backup

    • Windows:

       xcopy data data_backup /E /I

  5. Delete the old data folder.

    • Linux:

       rm -rf data

    • Windows:

       rmdir /S /Q data

  6. Create a new data folder.

    • Linux:

       mkdir data

    • Windows:

       mkdir data

  7. Copy the contents from data_backup to data.
    The working environment from which you copy the data folder must be compatible with the environment you're restoring.

    • Linux:

       cp -r /path/to/working/keycloak/data/* data/

    • Windows:

       xcopy C:\path\to\working\keycloak\data\* data\ /E /I

  8. Copy clientsecret from the same working environment and replace it in all of the available oidc.properties files in the following directories.

    • Linux:

      • $MANTA_HOME/configurationservice/manta-configuration-service-dir/conf/oidc.properties
      • $MANTA_HOME/serviceutility/manta-admin-ui-dir/conf/oidc.properties
      • $MANTA_HOME/openmantadesigner/manta-openmantadesigner-dir/conf/oidc.properties
      • $MANTA_HOME/server/manta-dataflow-server-dir/conf/oidc.properties
      • $MANTA_HOME/cli/platform/conf/oidc.properties

    • Windows:

      • %MANTA_HOME%\configurationservice\manta-configuration-service-dir\conf\oidc.properties
      • %MANTA_HOME%\serviceutility\manta-admin-ui-dir\conf\oidc.properties
      • %MANTA_HOME%\openmantadesigner\manta-openmantadesigner-dir\conf\oidc.properties
      • %MANTA_HOME%\server\manta-dataflow-server-dir\conf\oidc.properties
      • %MANTA_HOME%\cli\platform\conf\oidc.properties

  9. Start all Automatic Data Lineage services and verify that Keycloack service works by running a diagnostic scenario in Manta Admin UI Process Manager.

DataFlow server

Symptoms

If the H2 databases such as usageStatsDb or scriptMetadataDb are corrupted, DataFlow Server is down and you can find error messages in the logs.

Solution

  1. Stop and restart DataFlow Server.

    If you can still the error message in the logs, procced to step 2.

  2. Stop all Automatic Data Lineage services.

    • Linux: Use your service management scripts or commands.
    • Windows: Use the Windows Services Manager or relevant scripts.

  3. Change directories to the manta-dataflow-server-dir.

    • Linux:

       cd $MANTA_HOME/server/manta-dataflow-server-dir/data

    • Windows:

       cd %MANTA_HOME%\server\manta-dataflow-server-dir\data

  4. Back up the corresponding corrupted database.
    In this example, file names that start with usageStatsDB and scriptMetadataDb.

    • Linux:

       mv usageStatsDb usageStatsDb_backup
 mv scriptMetadataDb scriptMetadataDb_backup

    • Windows:

       rename usageStatsDb usageStatsDb_backup
 rename scriptMetadataDb scriptMetadataDb_backup

  5. Delete the corresponding corrupted database.
    In this example, file names that start with usageStatsDB and scriptMetadataDb.

    Delete only the corrupted database folders. If you delete the entire data folder, you might lose of all the previously scanned lineage data.

  6. Start all Automatic Data Lineage services and verify that DataFlow Server works by running a diagnostic scenario in Manta Admin UI Process Manager.

Manta Admin GUI

Symptoms

If the H2 databases are corrupted, Manta Admin UI is down and you can find entries similar to the following one in the logs.

Example:

Cause: org.h2.jdbc.JdbcSQLNonTransientException: General error: "org.h2.mvstore.MVStoreException: Reading from file split:30:/opt/mantaflow/serviceutility/manta-admin-ui-dir/data/manta_logs.mv.db failed at 7374124993 (length 28695310336), read 0, remaining 1536 [2.2.224/1]"; SQL statement:

In the example, manta_logs.mv.db is the corrupted database . Dropping logs doesn’t affect the repository. You can no longer extract the logs from the Log Viewer, but you can retrieve them manually by accessing the log folders.

For other corrupted databases, such as process-monitor-h2.mv.trace.db, if you drop the Process Manager, you mightt lose all workflows.

Solution

  1. Stop and restart Manta Admin UI.

    If you can still the error message in the logs, procced to step 2.

  2. Stop all Automatic Data Lineage services.

    • Linux: Use your service management scripts or commands.
    • Windows: Use the Windows Services Manager or relevant scripts.

  3. Change directories to serviceutility/manta-admin-ui-dir/data.

    • Linux:

       cd $MANTA_HOME/serviceutility/manta-admin-ui-dir/data

    • Windows:

       cd %MANTA_HOME%\serviceutility\manta-admin-ui-dir\data

  4. Back up the corrupted database.
    In the example, it's manta_logs.mv.db.

    • Linux:
       mv manta_logs.mv.db manta_logs.mv.db_backup
    • Windows:
       rename manta_logs.mv.db manta_logs.mv.db_backup

  5. Delete the corresponding corrupted database files.
    In the example, it's manta_logs.mv.db.

    Delete only the corrupted database folders. If you delete the entire data folder, you might lose data.

    Depending on which database you delete, different features might be affected. Before you delete a database, assess the potential impact. For example, if you delete manta_logs.mv.db, you can't export logs directly from the Manta Admin UI Log Viewer anymore. Instead, you can extract them manually from $MANTA_HOME/cli/log/ on Linux or %MANTA_HOME%\cli\log\ on Windows.

    • Linux:

       rm manta_logs.mv.db

    • Windows:

       del manta_logs.mv.db

  6. Start all Automatic Data Lineage services and verify that Manta Admin UI service works by running a diagnostic scenario in Manta Admin UI Process Manager.