Troubleshooting
Problem
Scheduled job Exception with "Communications link failure due to underlying exception" on the customTableDataUpload_80, customTableDataUpload_104, customTableDataUpload_105 or customTableDataUpload_106 trigger.
Symptom
Scheduled Job exceptions similar to:
>>>
2011-11-23 11:41:08.0 customTableDataUpload_105 trigger:
customTableJobGroup.customTableDataUpload_105 29 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
NESTED EXCEPTION ** 1
<<<<
These exceptions commonly occur every 5 minutes for one or more of the customTableDataUpload_80, customTableDataUpload_104, customTableDataUpload_105 or customTableDataUpload_106 jobs
Cause
This exception is due to one or more of the following reasons:
- the CUSTOM database table ENTERPRISE_NO_TRAFFIC has reached the 4GB limit;
- the CUSTOM database has reached the 4GB limit;
- inter-appliance connection timeout was exceeded
- the appliance listed in the <hostname> is a new aggregator.
- port closed in firewall.
Environment
Guardium v8.2 Central Manager
Diagnosing The Problem
customTableJobGroup.customTableDataUpload_80 9 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_104 20 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_105 29 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
AND/OR
customTableJobGroup.customTableDataUpload_106 2 total inserts. Error
occurred while upload on datasource: <collector/aggregator hostname> :
Communications link failure due to underlying exception: ** BEGIN
Resolving The Problem
Follow these steps to resolve the issue (ALL steps must be performed on the central manager ONLY - however, patch installation should be performed on all appliances):
1. Install the latest GPU (v82p100 or later). This GPU can be downloaded from FixCentral (http://www.ibm.com/support/fixcentral)
2. Restart the appliance after installing the latest GPU (this is only required if the appliance does not restart after installing the GPU).
3. Increase the inter-appliance connection timeout to at least 250 seconds. This can be completed by using the "store timeout db_connection 250000" - note this is in milliseconds. Verify it is set correctly by using "show timeout db_connection" - note the response is displayed in seconds.
4.In the GUI go to <Tools>, <Report Builder>, <Customer Table Builder> and:
- for customTableDataUpload_80 exceptions: select "STAP_INFO" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_104 exceptions: select "CM_SNIFFER_BUFFER_USAGE" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_105 exceptions: select "Enterprise No Traffic" and click purge. Schedule a purge to run daily for all data older than 1 day.
- for customTableDataUpload_106 exceptions: select "ENTERPRISE_STAPS_CHANGED" and click purge. Schedule a purge to run daily for all data older than 1 day.
5.In the GUI go to <Tools>, <Report Builder>, <Customer Table Builder> and:
- for customTableDataUpload_80 exceptions: select "STAP_INFO" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
- for customTableDataUpload_104 exceptions: select "CM_SNIFFER_BUFFER_USAGE" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
- for customTableDataUpload_105 exceptions: select "Enterprise No Traffic" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 20 minutes.
- for customTableDataUpload_106 exceptions: select "ENTERPRISE_STAPS_CHANGED" and click "Upload Data", click "Modify Schedule". Define a schedule to run no more frequently than once every 15 minutes.
6. If the appliance in the exception is a new aggregator. Unregister this aggregator from the central manager, run a manual aggregation using "Run Once Now". Once the aggregation has completed, re-register it back with the central manager.
7. Check that all ports are open as required. See this article:
https://www-01.ibm.com/support/docview.wss?uid=swg21973188
Especially check that 3306 is open in both directions.
8. Open a PMR to have someone from L2 Guardium support ensure the CUSTOM database and/or the four tables mentioned above are NOT full.
Note: In Step 4 above, define a purge schedule that will work for you but will NOT fill the CUSTOM database.
Note: In Step 5 above, these are only recommended schedule times, but do not run the data uploads more frequently than suggested.
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21605273