Contribute in GitHub:
Edit online
Manta Flow Usage: Preparing Scanner Inputs
Many technologies allow the automated extraction of metadata information. However, when this is not possible, Manta Flow supports manually-supplied metadata files. This is a generic article explaining the methods. See the technology-specific details and input structures in the documentation for individual scanners described in the section on manual inputs.
Default Metadata Input
The MANTA_DIR_HOME directory structure is detailed in
Manta Flow Scanner (Client)
Configuration; for example, mantaflow/cli/.
Input metadata files may be copied into the static directories for each technology and configured connection ID. A few basic examples are provided below.
The input directory contains directories with the defined technology-specific connection ID and subdirectory structure.
-
example:
<MANTA_DIR_HOME>/input/mssql/${mssql.dictionary.id}[/database_name[/schema_name]] -
example:
<MANTA_DIR_HOME>/input/oracle/${oracle.dictionary.id}[/schema_name] -
example:
<MANTA_DIR_HOME>/input/ifpc/${ifpc.extractor.repository}{/workflows|ifpcServiceSettings.prm|ifpcConnectionDefinition.prm}] -
example:
<MANTA_DIR_HOME>/input/datastage/${datastage.extractor.server} -
example:
<MANTA_DIR_HOME>/input/talend/${talend.system.id}
API Metadata Input
POST manta-admin-gui/public/process-manager/v1/executions
Metadata may also be supplied at the time of executing a workflow in Admin UI on the Process Manager screen (Workflow Execution Modal Window) or by calling the Orchestration
API
manta-admin-gui/public/process-manager/v1/executions method (
Manta
Orchestration API). This provides a method of automating the upload of a ZIP file and executing a workflow to process the contained metadata. When processing metadata dynamically using a ZIP file upload, the contents will contain the same
directory structure and files as would be supplied to a static input directory.
When providing metadata files using Admin UI or API, the contents of the ZIP are processed and analyzed in the lineage database. This allows Manta Flow to ingest multiple uploads with metadata without overwriting files in the default input directories for each technology. Metadata files uploaded to the server are processed for the specific execution only and automatically deleted when the workflow completes.
The ZIP file must contain a folder named “input” with the defined technology-specific subdirectories containing metadata files.
-
example:
input/mssql/${mssql.dictionary.id}[/database_name[/schema_name]] -
example:
input/oracle/${oracle.dictionary.id}[/schema_name] -
example:
input/ifpc/${ifpc.extractor.repository}{/workflows|ifpcServiceSettings.prm|ifpcConnectionDefinition.prm}] -
example:
input/datastage/${datastage.extractor.server} -
example:
input/talend/${talend.system.id}
__MACOSX folder, .DS_Store files) created by MacOS. As of R42.4, such files and folders are silently ignored by IBM Manta Data Lineage).
See Execute a Workflow via Orchestration API and Provide an Input File for more details and examples of how to execute a workflow via Orchestration API and provide an input file.