Filesystem Resource Configuration

A filesystem data file is represented in a dataflow graph as a File node in an appropriate Directory node hierarchy and, finally, in a Server node.
For example: C:/data/file.txt will be represented as localhost[Server] → C: [Directory] → data [Directory] → file.txt [File].

A cloud storage (Amazon S3) data file is represented in the dataflow graph as a File node server in either:

• A Directory hierarchy in a Bucket or

• Directly in a Bucket

Examples:

• A file located at s3://my-bucket/myFile.txt will be represented as my-bucket [Bucket] → myFile.txt [File].

• A file located at http://my-bucket.s3.amazonaws.com/myDir/myFile.txt will be represented as my-bucket [Bucket] → myDir [Directory] → myFile.txt [File].

File Path Mapping Configuration

This configuration allows the mapping of filesystem paths to provide a more user-friendly representation and/or to ensure that the same file can be correctly referenced from different technologies, even if the technology itself references the file by a different file path (for example, if the hostnames are different due to the network configuration). The configuration is common to all source systems.

The mapping is configured in a standard CSV file with the following columns.

The configuration can be changed in Configuration > CLI > Common > File Path Mapping in Manta Admin UI, as seen in the screenshot below.

Mapping Rules

• The first matching record is used

• Any condition that is left empty in the configuration is always met

• The record matches if all four source conditions are met:

  • Source Technology

    • Identification of the technology that referenced the file (e.g., ORACLE, DATASTAGE—see below)

  • Source Connection ID

    • Identification of the connection where the file is used (the same as that defined when creating the connection)

    • Case sensitive

  • Source Hostname

    • Hostname of the file as used in the analyzed technology

    • Case insensitive

  • Source Path Prefix

    • Beginning of the file path to be matched and replaced (when mapping file paths)

    • Regular expression

    • Cannot be used only to remove a prefix, but can be used only to add a prefix

    • Case sensitivity is controlled by the filepath.lowercase property

    • The directory separator is a platform dependent / or \\. As this is evaluated as a regular expression, use . or [/\\] which covers both possible directory separators. Leading directory separator (e.g. /data/upload) is mandatory.

• Non-empty target values from the matched record are used to override the original attributes

  • Target Resource

  • Target Hostname

  • Target Path Prefix

    • Replaces the section of the path matched by the source path prefix

Source Technology Values

Examples