MySQL Resource Configuration

MySQL Scanner is in the MVP version, currently supporting only Extractor, Dictionary Mapping, and Dataflow Query Service scenarios.

Note that MySQL scanner requires Manta Agent. Read Manta Flow Agent Configuration for Extraction for more details.

Source System Properties

This configuration can be set up by creating a new connection on Admin UI / Connections tab or editing an existing connection in Admin UI / Connections / Databases / MySQL / specific connection. New connections can also be created via Manta Orchestration API.

Granularity of the IBM Automatic Data Lineage connection for MySQL is one MySQL, MariaDB, or SingleStoreDB server. Use filter on databases and schemas to limit the scope of analysis as needed. Use of multiple connections against a single MySQL, MariaDB, or SingleStoreDB server may lead to within-system lineage not be connected properly.

Property name	Description	Example
mysql.dictionary.id	Name of a resource representing this MySQL server known as the dictionary ID, used as an output subdirectory name for extracted DDL files and the database dictionary.	dwh
mysql.subdialect	The server type - use `MYSQL` for MySQL, use `MARIADB` for MariaDB, use `SINGLESTORE` for SingleStoreDB.	MYSQL
mysql.url	JDBC connection string for the database server. The JDBC driver URL format needs to match the actual JDBC driver used, see `mysql.jdbcDriverClass` property. See documentation for respective JDBC URL Syntax MariaDB: https://mariadb.com/kb/en/about-mariadb-connector-j/#connection-strings MySQL: https://dev.mysql.com/doc/connector-j/en/connector-j-reference-jdbc-url-format.html SingleStore: https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#connection-string-format	jdbc:mariadb://<host>:<port> jdbc:mysql://<host>:<port> jdbc:singlestore://<host>:<port>
mysql.username	User name for connection to the database server.	manta_user
mysql.password	User password for connection to the database server	manta_password
mysql.extractedSchemas	Comma separated list of schemas to be extracted; each part is evaluated as a regular expression	sch1,sch2
mysql.excludedSchemas	Comma separated list of schemas excluded from the extraction; each part is evaluated as a regular expression	sch3,sch4
mysql.extraction.method (as of R42.1)	Set to Agent:default when the desired extraction method is the default Manta Extractor Agent, set to Agent:{remote_agent_name} when a remote Agent is the desired extraction method, set to Git:{git.dictionary.id} when the Git ingest method is the desired extraction method. For more information on setting up a remote extractor Agent please refer to the Manta Flow Agent Configuration for Extraction documentation. For additional details on configuring a Git ingest method, please refer to the Manta Flow Agent Configuration for Extraction:Git Source documentation.	default Git agent
mysql.ddl.encoding	Encoding of automatically extracted DDL scripts. See Encodings for applicable values.	utf8
mysql.script.encoding	Encoding of manually provided SQL scripts performed on this database server. See Encodings for applicable values.	utf8

Common Scanner Properties

This configuration is common for all MySQL source systems and for all MySQL scenarios, and is configured in Admin UI / Configuration / CLI / MySQL / MySQL Common. It can also be overridden on the individual connection level.

Property name	Description	Example
mysql.dictionary.dir	Directory with data dictionaries extracted from MySQL	${manta.dir.temp}/mysql
filepath.lowercase	Whether paths to files should be lowercase (false for case-sensitive file systems, true otherwise); default = false	false true
mysql.ddl.output	Directory for automatically extracted MySQL DDL scripts (for the extraction phase)	${manta.dir.temp}/mysql/${mysql.dictionary.id}/ddl
mysql.ddl.input	Directory with automatically extracted MySQL DDL scripts (for the analysis phase)	${manta.ddl.output}
mysql.script.input	Directory with manually provided MySQL, MariaDB, or SingleStoreDB SQL scripts which are performed on a given database server (for the analysis phase)	${manta.dir.input}/mysql/${mysql.dictionary.id}/
mysql.script.replace	Path to the CSV file with the replacements to be applied to the provided MySQL, MariaDB, or SingleStoreDB SQL scripts; see Placeholder Replacement in Input Scripts for details about the replacement file format	${manta.dir.input}/mysql/${mysql.dictionary.id}/replace.csv
mysql.script.replace.regex	Flag specifying whether replacements for MySQL, MariaDB, or SingleStoreDB SQL scripts in the provided CSV file specified in `mysql.script.replace` should be interpreted as regular expressions (true) or simple text (false)	false true
mysql.analyze.parallelCount	Number of parallel threads which will analyze DDL and MySQL, MariaDB, or SingleStoreDB SQL scripts (currently unused)	4
mysql.dictionary.mappingFile	Path to automatically generated mappings for MySQL, MariaDB, or SingleStoreDB servers	${manta.dir.temp}/mysql/mysqlDictionaryMantaMapping.csv
mysql.dictionary.mappingManualFile	Path to mappings provided manually for MySQL, MariaDB, or SingleStoreDB servers	${manta.dir.scenario}/conf/mysqlDictionaryMantaMappingManual.csv
mysql.extractExtendedAttributes.enabled	Extract extended attributes like primary key and unique and referential integrity constraints of columns; default = false	false true
mysql.extractDatabaseGeneratedDdls.enabled	Extract the DDL definition of database entities instead of letting Automatic Data Lineage generating them; this is time consuming operation and its usage will affect extractor performance; default = true	false true
mysql.extractor.ssl.enabled	Use SSL-protected connection; default = false	false true
mysql.extractor.ssl.mode	SSL mode for the connection `disable`: Do not use SSL/TLS (default) `trust`: Only use SSL/TLS for encryption. Do not perform certificate or hostname verification. This mode is not safe for production applications. `verify-ca`: Use SSL/TLS for encryption and perform certificates verification, but do not perform hostname verification. `verify-full`: Use SSL/TLS for encryption, certificate verification, and hostname verification.	disable trust verify-ca verify-full
mysql.driverClassName	Class name for the JDBC driver used. Provide the class name appropriate for the your JDBC driver based on the documentation. Setting in this property must be reflected in your JDBC Connection URL as well.	com.mysql.cj.jdbc.Driver org.mariadb.jdbc.Driver com.singlestore.jdbc.Driver

Manual Mapping Properties

It is possible to manually configure mappings for MySQL databases. This is necessary for non-extracted schemas in the Automatic Data Lineage extraction phase and complicated usage of database links.

Each mapping has its own row with the following parts separated by semicolons.

Property name	Description	Example
Dictionary ID	Name of a resource representing this MySQL server known as the dictionary ID, used as an output subdirectory name for extracted DDL files and the database dictionary	dwh
Sub-dialect	The server type - use `MYSQL` for MySQL, use `MARIADB` for MariaDB, use `SINGLESTORE` for SingleStoreDB.	MYSQL
Hostname / Endpoint	Name of the host where the database is running	localhost 127.0.0.1
Port	Port on which the database listens	3306
Connection ID	External MySQL connection ID in third-party tools; can be left empty if it is the same as the host name	dwh
Included databases/schemas	List of databases/schemas (database and schema is the same thing in MySQL) to extract, separated by commas; each part is evaluated as a regular expression	sch1,sch2
Excluded databases/schemas	List of databases/schemas (database and schema is the same thing in MySQL) to exclude from extraction, separated by commas; each part is evaluated as a regular expression	sch3,sch4