Manta Flow Server Repository Management

The metadata repository of the Manta Flow Server application is stored in its installation directory. It is shared by all modules, including visualization, import, export, and query. It can be filled, cleaned, or queried by Manta Flow client applications.

Back Up and Restore

Physical Copy

To back up the metadata repository:

• Ensure that no application is using Manta Flow Server.

• Stop the Manta Flow Server application by either running the bin/shutdown.(bat|sh) script or stopping its installed service.

• Locate the database directory: manta-dataflow-server-dir/data/neo4j

• Copy the database directory to a backup location.

• Start Manta Flow Server again by either running bin/startup.(bat|sh) or starting its installed service.

• Ensure that the Manta Flow Viewer application is running on its configured URL.

To restore a previous backup of a metadata repository:

• Ensure that no application is using Manta Flow Server.

• Stop the Manta Flow Server application by either running the bin/shutdown.(bat|sh) script or stopping its installed service.

• Locate the database directory: manta-dataflow-server-dir/data/neo4j

• Remove the database directory.

• Copy the database directory from your backup location back to the original directory.

• Start Manta Flow Server again by either running bin/startup.(bat|sh) or starting its installed service.

• Ensure that the Manta Flow Viewer application is running on its configured URL.

Physically copying the repository usually goes fast, but it depends on the OS and the version of Manta Flow.

Dump

The metadata repository can be exported and imported in binary format, which is independent of the OS and the version of Manta Flow. This dump applies to the whole repository including all versions and all technical metadata.

Dumps are only accessible to users with the role ROLE_MERGER.

Warning: Importing a dump will erase the old data in the target repository!

Links for the dump:

• Export — <manta-server-url>/dump/export

• Import — <manta-server-url>/dump/import

Dump size is affected by the number and content of the revisions in the repository. To reduce the dump size, delete and prune revisions that are no longer needed.

REST API

It is also possible to export and import dumps through the REST API.

• Export — <manta-server-url>/api/dump/export

  • Method: GET

  • The dump is returned in response as Content-Disposition: attachment

    Curl request example

    curl --location --request GET 'http://localhost:8080/manta-dataflow-server/api/dump/export' \
    --header 'Authorization: Bearer <place your token here>'
    

• Import — <manta-server-url>/api/dump/import

  • Method: POST

  • Content-Type: multipart/form-data

  • The dump is sent in an attachment named file

  • Returns JSON with an isOK flag and an error message upon failure

    Curl request example

    curl --location --request POST 'http://localhost:8080/manta-dataflow-server/api/dump/import' \
    --header 'Content-Type: multipart/form-data' \
    --header 'Authorization: Bearer <place your token here>' \
    --form 'file=@"/C:/path/to/the/dump_sample.zip"'
    

See API Token-Based Authentication for instructions on how to obtain the authorization token. A postman collection sample is available here.

ServerDump.postman_collection.json

Maintenance Links

There are several links to maintain the repository.

• <manta-server-url>/api/truncate — permanently erases all data in the repository so that it is impossible to restore without a backup

• <manta-server-url>/api/inspect — collects information about the last revision made and displays the numbers for each type

• <manta-server-url>/api/revision/rollback-current-revision — rolls back the newest revision, which must be uncommitted

• <manta-server-url>/api/revision/delete-last — deletes the last committed revision (fails if there is an uncommitted revision)

Location of Persistent Files

The configuration Configurations > Server > Common > Repository Configuration in Manta Admin UI contains the locations of all repository files and directories that should persist across Manta Flow Server reboots.

Session Policy

It is also possible to adjust the behavior of Manta Viewer sessions.

The file for changing the default configuration is <MANTA_SERVER_HOME>/manta-dataflow-server-dir/conf. It is possible to change the default 30-minute session timeout by adding the following entry.

server.servlet.session.timeout=<timeout in minutes>m

Version Control System

The metadata repository supports the version control system. This system can theoretically hold an unlimited number of revisions, but each revision takes up space on the disk. The maximum number of revisions is maintained by the prune feature, which deletes older revisions that exceed the configured amount.

It is possible to turn off the version control system. One reason to do so could be to only use the Manta Flow Server for exporting to third-party applications. In this case, the client workflow must be changed.

Revision System

Two types of revisions exist.

1. Major revisions — Work like revisions in older versions of Automatic Data Lineage. Each major revision is identified by an integer.

2. Minor revisions — Enable the incremental update of repository metadata. Loaded metadata is merged with metadata from the previous revision. Each minor revision is identified by a six-digit float number.

Basic API

• POST <manta-server-url>/api/revision/createMajor — creates a new major revision

• POST <manta-server-url>/api/revision/createMinor — creates a new minor revision

• GET <manta-server-url>/api/revision/commit/<revision> — commits the revision <revision>

• POST <manta-server-url>/api/revision/prune-preserve — prunes old revisions; request parameters:

  • revisionCount — count of revisions to be preserved

  • majorsOnly — if true (by default), only preserved revisions are counted over major revisions; if false, all revisions are counted