Validation hangs when you run a query after you scale worker pods in Data Virtualization

After you scale down Data Virtualization, a query might hang. In the c-db2u-dv-db2u-0 pod in the log file, you might see a message such as Error while waiting for connect() completion with node 2.

Symptoms

You might see an error message that is similar to the following example.
277027 2021-11-11-17.47.06.533694+000 I17342851E403         LEVEL: Event
 277028 PID     : 292527               TID : 140515467060992 PROC : db2sysc 0
 277029 INSTANCE: db2inst1             NODE : 000
 277030 HOSTNAME: c-db2u-dv-db2u-0
 277031 EDUID   : 17                   EDUNAME: db2fcms-0 0
 277032 FUNCTION: DB2 UDB, fast comm manager, sqkfSendConduit::CheckForFailoverConnectRetry, probe:25
 277033 START   : Retry connect to node 2 after relocation.
 277034
 277035 2021-11-11-17.47.06.542472+000 E17343255E425         LEVEL: Error
 277036 PID     : 292527               TID : 140515467060992 PROC : db2sysc 0
 277037 INSTANCE: db2inst1             NODE : 000
 277038 HOSTNAME: c-db2u-dv-db2u-0
 277039 EDUID   : 17                   EDUNAME: db2fcms-0 0
 277040 FUNCTION: DB2 UDB, fast comm manager, sqkfTcpLink::closeConn, probe:25
 277041 DATA #1 : <preformatted>
 277042 Link info: node 2; type 6; state 3; session 4; activated 1; sent data 0
 277043
 277044 2021-11-11-17.47.09.492778+000 E17343681E512         LEVEL: Error (OS)
 277045 PID     : 292527               TID : 140515467060992 PROC : db2sysc 0
 277046 INSTANCE: db2inst1             NODE : 000
 277047 HOSTNAME: c-db2u-dv-db2u-0
 277048 EDUID   : 17                   EDUNAME: db2fcms-0 0
 277049 FUNCTION: DB2 UDB, oper system services, sqloPdbQuerySocketErrorStatus, probe:15
 277050 MESSAGE : ZRC=0x810F0077=-2129723273=SQLO_COMM_ERR_EHOSTUNREACH
 277051           "No route to host"
 277052 CALLED  : OS, -, getsockopt                       OSERR: EHOSTUNREACH (113)
 277053
 277054 2021-11-11-17.47.09.498432+000 I17344194E2996        LEVEL: Error
 277055 PID     : 292527               TID : 140515467060992 PROC : db2sysc 0
 277056 INSTANCE: db2inst1             NODE : 000
 277057 HOSTNAME: c-db2u-dv-db2u-0
 277058 EDUID   : 17                   EDUNAME: db2fcms-0 0
 277059 FUNCTION: DB2 UDB, fast comm manager, sqkfTcpServices::identifyUntrustedIncomingEvent, probe:41
 277060 MESSAGE : Error while waiting for connect() completion with node 2. rc = -2129723273. status = 3; port = 60000
 277061 DATA #1 : Hexdump, 488 bytes

Resolving the problem

  1. Change to the project where Data Virtualization pods are installed.
    oc project ${PROJECT_CPD_INST_OPERANDS}
  2. Log in to the Data Virtualization head pod.
    oc rsh c-db2u-dv-db2u-0 bash
  3. Switch to the Data Virtualization database instance owner db2inst1.
    su - db2inst1
  4. Run the following commands to restart the bigsql pod.
    bigsql stop
    bigsql start