Basic Log Analysis

Problem

Identification of the most common errors and the steps to resolve them. A more detailed analysis of the text logs produced by IBM Automatic Data Lineage scanners and aggregated statistics of the errors to make them easier to focus on along with suggestions on how to resolve the issue.

Solution

All logs from lineage analyses are available in the mantaflow/cli/log/ directory. Use the following Linux shell script to filter the logs to get a basic understanding of the common errors in the logs. Follow the related articles to get more detailed information on error messages for particular technologies. On Windows, you can run it after installing a Linux subsystem or downloading the grep and sed command line utilities for Windows.

1. Save this script in mantaflow/cli/log/.

   filter_logs.sh

   #! /bin/bash
   # version: 2020-02-11
   
   # Common Errors
   grep -l " ERROR " *.log | grep -vE "Dataflow|Extractor.*properties" > _general_ERROR.txt
   grep -l "OutOfMemoryError" *.log > _general_OutOfMemory.txt
   grep "Caused by: " *.log | sort | uniq -c | sort -nr > _general_exceptions.txt
   grep "Connect to localhost.*refused" *.log | sort | uniq -c | sort -nr > _general_MantaServerDown.txt
   grep "Cannot create a new revision without committing the previous one." *RevisionScenario.log > _general_OpenRevision.txt
   grep "Library .*exists in more versions" *.log | sed "s/.*MantaVersionHelper //g" | sort | uniq > _general_incorrectPatchSet.txt
   
   # ifpc
   grep "Connection with shortcut" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*ConnectionException: //g" | sort | uniq -c | sort -nr > _ifpc_unknownConnectionShortcuts.txt
   grep "Connection variable .* is not blank" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*ExternalAnalyzerImpl//g" | sed "s/in transformation [^ ]* //g" | sort | uniq -c | sort -nr > _ifpc_unresolvedConnectionVariable.txt
   grep "Connection variable .* is not blank" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*ExternalAnalyzerImpl Connection variable //g" | sed "s/is not blank.*//g" | sort | uniq -c | sort -nr > _ifpc_unresolvedConnectionVariable_Grouped.txt
   grep "was replaced to an empty string in" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*IfpcDataflowException: //g" | sort | uniq -c | sort -nr > _ifpc_emptyParameter.txt
   grep "Unable to find parameter file" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*ParameterException: //g" | sed "s/ when searching for parameter .*//g" | sort | uniq -c | sort -nr > _ifpc_missingParameterFile.txt
   grep "Creating deduced database" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*TeradataResolver.DEDUCTION//g" | sed "s/ for name.*//g" | sort | uniq -c | sort -nr > _ifpc_missingTeradataDatabase.txt
   grep -E "There is no mapping for the Connection|Cannot find any matching system for Connection" ifpcDataflowMasterScenario_*.properties.log | sed "s/.*ReaderProvider -* *//g" | sed "s/.*DataflowQueryServiceImpl -* *//g" | sed "s/userName.*//g" |sort | uniq -c | sort -nr > _ifpc_noConnectionMapping.txt
   grep "does not exist in the connection setting file" ifpcDataflowMasterScenario_*.properties.log | sort | uniq -c | sort -nr  > _ifpc_connectionNotFound.txt
   
   # ssas
   grep -E "There is no mapping for the Connection|Cannot find any matching system for Connection" ssasDataflowMasterScenario_*.properties.log | sed "s/.*ReaderProvider -* *//g" | sed "s/.*DataflowQueryServiceImpl -* *//g" | sort | uniq -c | sort -nr > _ssas_noConnectionMapping.txt
   
   # talend
   grep "talend.*Unknown component" log.txt | sed "s/.* Unknown component with typename //g" | sort  | uniq -c | sort -nr > _talend_UnknownComponents.log
   
   # ssis
   grep -E "There is no mapping for the Connection|Cannot find any matching system for Connection" ssisDataflowMasterScenario_*.properties.log | sed "s/.*ReaderProvider -* *//g" | sed "s/.*DataflowQueryServiceImpl -* *//g" | sort | uniq -c | sort -nr > _ssis_noConnectionMapping.txt
   grep "Object not found in dictionary" ssisDataflowMasterScenario_*.properties.log | sed "s/.*DatabaseConnectorImpl -* *Object not found in dictionary: \(.*\)\.[^.]*$/\1/g" | sort | uniq -c | sort -nr > _ssis_ObjectNotFoundInDictionary_Grouped.txt
   grep "Object not found in dictionary" ssisDataflowMasterScenario_*.properties.log | sed "s/.*DatabaseConnectorImpl -* *Object not found in dictionary: \([^ ]*\) *$/\1/g" | sort | uniq -c | sort -nr > _ssis_ObjectNotFoundInDictionary.txt
   grep "Could not find .* connection manager" ssisDataflowMasterScenario_*.properties.log | sed "s/.*Analyzer //g" | sed "s/ referenced in.*//g" | sort | uniq -c | sort -nr > _ssis_CouldNotFindConnectionManager_grouped.txt
   grep "Could not find .* connection manager" ssisDataflowMasterScenario_*.properties.log | sed "s/.*Analyzer //g" | sort | uniq -c | sort -nr > _ssis_CouldNotFindConnectionManager.txt
   grep "No connection string set for" ssisDataflowMasterScenario_*.properties.log | sed "s/.*Impl //g" | sed "s/. Lineage may be inaccurate or missing for.*//g" | sort | uniq -c | sort -nr > _ssis_NoConnectionStringSet_Grouped.txt
   grep "No connection string set for" ssisDataflowMasterScenario_*.properties.log | sed "s/.*Impl //g" | sort | uniq -c | sort -nr > _ssis_NoConnectionStringSet.txt
   
   # netezza
   grep " using deduction to DATABASE" netezzaNzplsqlDataflowMasterScenario_*.properties.log | sed "s/.*to DATABASE//g" | sort | uniq -c | sort -nr > _netezza_DeducedDatabase.txt
   
   # teradata
   grep "Creating deduced database object with normalized name" teradataBteqDataflowMasterScenario_*.properties.log | sed "s/.*Creating deduced database object with normalized name//g" | sed "s/ for name.*//g" | sort | uniq -c | sort -nr > _teradataBteq_DeducedDatabase.txt
   grep "Creating deduced database object with normalized name" teradataDdlDataflowMasterScenario_*.properties.log | sed "s/.*Creating deduced database object with normalized name//g" | sed "s/ for name.*//g" | sort | uniq -c | sort -nr > _teradataDdl_DeducedDatabase.txt
   
   # mssql
   grep "Cannot open database " mssqlExtractorMasterScenario_*.properties.log | sed "s/.*Cannot open database //g" | sed "s/ requested by the login.*//g" | sort | uniq > _mssql_missingPriviledgeToDatabase.txt
   grep "The SELECT permission was denied on the object" mssqlExtractorMasterScenario_*.properties.log | sort | uniq > _mssql_missingPriviledgeSelect.txt
   grep "Could not find body of .* check that the database user has VIEW DEFINITION privilege" mssqlExtractorMasterScenario_*.properties.log > _mssql_missingPriviledgeViewDefinition.txt
   
   # oracle
   grep "have sufficient rights to the object or schema." oracleExtractorMasterScenario_*.properties.log > _oracle_insufficientPrivileges.txt
   grep -E "There is no mapping for the Connection|Cannot find any matching system for Connection" oracleDdlDataflowMasterScenario_*.properties.log | sed "s/.*ReaderProvider -* *//g" | sed "s/.*DataflowQueryServiceImpl -* *//g" | sort | uniq -c | sort -nr > _oracle_noConnectionMapping.txt
   

2. Run the script and investigate the files generated using the table below. After fixing the issue, rerun the Automatic Data Lineage lineage analysis as well as this log filter.

Related Articles

• Server Start Failure Because of Inability to Set Localhost

• Oracle Database Links

• Explanation of Teradata Connector Log Errors

• Explanation of Microsoft SSIS Connector Log Errors

• Explanation of Oracle Connector Log Errors