Troubleshooting common core services

Use these workarounds to troubleshoot problems that you might encounter with Flight service or other issues in common core services.

Can't load large data sets if the notebook environment doesn't have enough memory

You receive an error that states the notebook kernel is restarting because the kernel died while running the generated code that loads data from a connection to your notebook. You receive a "FlightTimedOutError" when you run the generated code. This timeout error occurs only if the generated code uses Flight service.

Cause

The reason is that you are trying to load a large amount of data to your notebook in a runtime environment with limited memory.

When you load large data sets into a specific data structure in a notebook, you typically need more memory than the size of the source data on disk. The amount of memory depends directly on the target data structure and the type of data. Additional memory is required for intermediate data structures that are created during the data transfer process from the source, depending on the transfer mechanism, such as Flight service, JDBC, or others.

Resolving the problem

Make sure that you are running your notebook in an environment runtime that is large enough to load the data sets you want to work with. If the provided environments aren't large enough, you can create a custom environment template.

Can't run generated code from deprecated functionality to load data for an Informix connection

You receive a "DatabaseError: Execution failed on sql: ... unable to rollback" when running the code that is generated to load data from an Informix database when using the deprecated functionality and you try to query a table with an upper-case name.

Cause

The Informix database is configured for case-sensitive identifiers and you need to update the generated code.

Resolving the problem

If you can, change to using the code that is generated using the Flight service to load data from an Informix connection. If you want to continue using the code that is generated by the deprecated functionality, you need to edit the generated code as follows:

  • For Python, add the connection property 'DELIMIDENT=Y' to the connection and surround the upper-case identifier with double-quotes (""). Replace the following lines:

    Informix_connection = jaydebeapi.connect('com.informix.jdbc.IfxDriver',
'{}://{}:{}/{}:user={};password={};'.format('jdbc:informix-sqli',
Informix_metadata['host'],
Informix_metadata['port'],
Informix_metadata['database'],
Informix_metadata['username'],
Informix_metadata['password']),
[Informix_metadata['username'],
Informix_metadata['password']])

query = 'SELECT * FROM informix.FVT_EMPLOYEE'

    With:

    Informix_connection = jaydebeapi.connect('com.informix.jdbc.IfxDriver',
'{}://{}:{}/{}:user={};password={};DELIMIDENT=Y;'.format('jdbc:informix-sqli',
Informix_metadata['host'],
Informix_metadata['port'],
Informix_metadata['database'],
Informix_metadata['username'],
Informix_metadata['password']),
[Informix_metadata['username'],
Informix_metadata['password']])

query = 'SELECT * FROM informix."FVT_EMPLOYEE"'

  • For Spark with Python, replace:

    data_df_0 = sparkSession.read.format('jdbc') \
  .option('url', 'jdbc:informix-sqli://{}:{}/{}'.format(Informix_metadata['host'],Informix_metadata['port'],Informix_metadata['database'])) \
  .option('dbtable', 'informix.FVT_EMPLOYEE') \
  .option('user', Informix_metadata['username']) \
  .option('password', Informix_metadata['password']).load()
data_df_0.show(5)

    With:

    data_df_0 = sparkSession.read.format('jdbc') \
  .option('url', 'jdbc:informix-sqli://{}:{}/{}'.format(Informix_metadata['host'],Informix_metadata['port'],Informix_metadata['database'])) \
  .option('dbtable', 'informix."FVT_EMPLOYEE"') \
  .option('DELIMIDENT', 'Y') \
  .option('user', Informix_metadata['username']) \
  .option('password', Informix_metadata['password']).load()
data_df_0.show(5)

  • For R, add the connection property 'DELIMIDENT=Y' to the connection and surround all upper case names with double-quotes (""). Replace the following lines:

    paste("jdbc:informix-sqli://", Informix_credentials[][["host"]], ":", Informix_credentials[][["port"]],
"/", Informix_credentials[][["database"]],
":user=", Informix_credentials[][["username"]],
";password=", Informix_credentials[][["password"]], ";", sep=""),
...
query <- "SELECT * FROM myschema.MY_TABLE"

    With:

    paste("jdbc:informix-sqli://", Informix_credentials[][["host"]], ":", Informix_credentials[][["port"]],
"/", Informix_credentials[][["database"]],
":user=", Informix_credentials[][["username"]],
";password=", Informix_credentials[][["password"]],";DELIMIDENT=Y", ";", sep=""),
...
query <- "SELECT * FROM myschema.\"MY_TABLE\""