IBM Support

MustGather: IBM WebSphere Edge Load Balancer (Dispatcher component)

Troubleshooting


Problem

Collecting data for Dispatcher problems with the WebSphere® Application Server Load Balancer process. Before engaging IBM® support, review this information to familiarize yourself with the troubleshooting process. 
Be aware there are two Edge Load Balancer (LB) products:
  • Edge Load Balancer for IPV4 - short name is LLB for legacy Load Balancer (kernel intrusive) also known by its process name "ibmlb" (formerly "ibmnd"). This product is out of service.
  • Edge Load Balancer for IPV4 and IPV6 - short name is ULB for User space Load Balancer (minimal kernel-dependent code) also known by its process name "ibmlb2".

Resolving The Problem

If you already contacted support, continue on to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server Products.

Load Balancer MustGather information
If high availability is configured, there is a primary and a standby load balancer. When running with high availability configured, Load Balancer data should be collected from both load balancers.
If the reported problem involves communication with the backend servers, data should be collected from the backend servers as well as the load balancer.
For problems involving forwarded traffic,  use a client workstation on the same subnet as the Load Balancer cluster. In addition to the requested data, after attempting to access the cluster from the client workstation, run the server collection tool on the client workstation.
Prior to data collection:
If possible, the Load Balancer should be running and servers configured to respond to traffic when data is collected. 
  • If a Load Balancer is stopping unexpectedly or the server.log file contains entries which start with the keyword "Exception", ensure your operating system is configured to produce an application core file. Review information about core file creation to ensure necessary data is collected.
  • Increase default logging prior to loading the Load Balancer configuration.
    dscontrol set loglevel 5
    dscontrol set logsize unlimited
  • For high availability problems, increase the reach log level and log size.
    dscontrol manager reach set loglevel 5
    dscontrol manager reach set logsize unlimited
  • For advisor problems, increase the advisor log level and log size.
    dscontrol advisor loglevel protocol port 5
    dscontrol advisor logsize protocol port unlimited
Data collection:
Download the Load Balancer problem determination tool (lbpd) to collect the necessary data. If using a unix operating system, set execute permission on the tool after transfer to the load balancer machine. The Load Balancer problem determination tool will create the lbpmr.jar file in the 'PD' directory which is created in the ULB home directory.
If the reported problem is related to forwarding traffic or the backend servers, download and run the server tool on each server. If using a unix operating system, set execute permission on the tool after transfer to the server workstation. The tool will create a text file (txt) called the hostname of the workstation in the local directory.
Follow instructions to send diagnostic information to IBM support.

Core file creation
The Load Balancer may crash when a programming error occurs. The executor is responsible for processing all packets and is most likely to experience a crash. Configure your operating system to produce a core file to ensure necessary data is available if a problem occurs.
  • If the Load Balancer experiences a fatal programming error on AIX, a system core will be produced by default. If the system core is not created, ensure the defined dump device has sufficient space.
  • For Linux operating system, user limits must be sufficient size.
    ulimit -c unlimited
    • To configure RedHat servers to produce core files, follow the instructions provided by RedHat. If the abrt service is used, edit /etc/abrt/abrt-action-save-package-data.conf and change the OpenGPGCheck and OpenUnpackaged settings.
      OpenGPGCheck=no
      OpenUnpackaged=yes

      Restart the abrtd service after changes are saved.
    • To configure Ubuntu servers to produce core files, the apport software must be configured.
    • To configure SLES servers, follow instructions provided by SUSE.
  • Microsoft Windows does not produce core files by default. Settings must be modified in the registry and the systems needs to be rebooted for the changes to take effect.  The application name to use when creating the new registry key is ibmlb2.exe.
Additional data may be requested based upon the reported problem. Follow the instructions provide if additional data is requested.
  • Network traces help identify discarded or delayed traffic. AIX users must review the network trace material provided before collecting any network traces.
  • Executor tracing may be requested for unexpected program behavior.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"Edge Component","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"9.0.0.0;8.5.5;8.5;8.0;7.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
12 May 2021

UID

swg21140797