Errors when you run a query or virtualize objects and the scheduler is not available

When you run a query or virtualize objects in Data Virtualization and the scheduler is not available, you might receive an error or the process hangs or stops working.

Symptoms

For example, you might see the following SQL5197N SQL error code in the user interface when you run a query or virtualize objects:

SQL5197N  The statement failed because of a communication error with a Big SQL component. 
Big SQL component name: "SCHEDULER". Reason code: "1"

You might also see requestScanMetadata errors in the log files.

Causes

These errors occur if the scheduler service is unavailable.

The Db2® Big SQL scheduler is a service that helps in the execution of queries. It communicates with all of the components in Data Virtualization. The scheduler provides access to the metadata for Hadoop tables that are involved in a scan. This metadata includes table descriptors with details such as column types, column lengths, partition information, and input format. It evenly distributes the scan of a Hadoop table in your cluster. The scheduler also does partition elimination. It uses system resources efficiently to maximize throughput and minimize response time.

The scheduler process can become unavailable because of memory constraints. When the scheduler runs out of memory, Data Virtualization stops running in the pod where the scheduler process is running.

Diagnosing the problem

If virtualization or a query fails, check the db2diag.log in the head pod (c-db2u-dv-db2u-0) for any requestScanMetadata errors. The virtualization might fail due to insufficient memory allocation to the hurricane pod, as shown in the following log file example:

2023-03-29-17.45.56.043984+000 I14758539E704         LEVEL: Error
PID     : 107690               TID : 140318766786304 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000            DB   : BIGSQL
APPHDL  : 0-14563              APPID: 192.168.18.141.51440.230329174517
AUTHID  : SVCCP4DSLSPROD       HOSTNAME: c-db2u-dv-db2u-0
EDUID   : 122915               EDUNAME: db2agent (BIGSQL) 0
FUNCTION: DB2 UDB, base sys utilities, sqeBigSqlSchedulerInternal::requestScanMetadata, probe:5003
MESSAGE : BigSql Scheduler Exception Occurred
DATA #1 : signed integer, 4 bytes
-5105
DATA #2 : String, 98 bytes
Incomplete-table: Front-End-Exception.  schemaName=cp4dschema tableName=table_csv.

Resolving the problem

Make sure that you allocate sufficient memory for the cache or else the scheduler service might hang and then fail to respond to db2 requests to scan metadata.

The memory requirement should scale with the volume of file metadata to be cached. Each deployment is different. Incrementally increase memory allocation until a sufficient amount is reached. The memory limit must be increased as a percentage of the actual memory that is allocated to the hurricane pod, which is statically assigned to the scheduler process when it starts. The minimum memory requirement for the hurricane pod is 2.5Gi. For more information, see Preparing to install Data Virtualization.

To increase the memory, run the following command:
```
oc edit bigsql db2u-dv
```
Edit the section under podConfig > hurricane > resource.
For example, the following configuration would increase the memory limit to twice the default.
```
resource:
  hurricane:
    limits:
      cpu: 8
      memory: 5120Mi
   requests:
      cpu: 1
      memory: 2560Mi
```