IBM Support

MustGather: Core Security problems for WebSphere traditional and Liberty

Troubleshooting


Problem

This document describes the process for collecting data for problems with the IBM WebSphere® Application Server traditional and WebSphere Liberty core security components. The core security component includes authentication, authorization, user/group search, membership/members, user/group role assignment, SSO (except Web SSO), and so on. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.


Avoid delay:
For Web Single Sign-on (SSO) component issues, such as SAML, OpenID, OpenID Connect or OAuth, see:
MustGather: Web Single Sign-on problems with WebSphere Application Server

For SPNEGO issues, see:
MustGather: Problems with SPNEGO

Resolving The Problem


  • Read first and MustGathers


    For a listing of all technotes, downloads, and educational materials specific to the Security component, search the WebSphere Application Server support site.
  • Exchange data with IBM Support

    To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:


  • Core security trace specifications
    • WebSphere traditional
      Avoid trouble: The trace strings for WebSphere traditional must be entered as one line with no breaks or spaces.

      Core security trace string:

      *=info:com.ibm.ws.security.*=all:com.ibm.websphere.security.*=all:com.ibm.websphere.wim.*=all:com.ibm.wsspi.wim.*=all:com.ibm.ws.wim.*=all


       
      • If the problem is related to authentication to an Enterprise JavaBean™, append the following to the trace specification:

        :SASRas=all:ORBRas=all

      • If the problem is related to security domains, append the following to the trace specification:

        :SecurityDomain=all

      • Only if requested by IBM Support, collect communication (COMM) traces by setting the following in the Generic JVM arguments for the JVM being traced:

        -Dcom.ibm.CORBA.Debug=true
        -Dcom.ibm.CORBA.CommTrace=true

    • Liberty
      Core security trace string:

      com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all

  • Collect data for WebSphere traditional
    • This section is for collecting data for WEBSPHERE TRADITIONAL. If you want to collect data for Liberty, see the Collect data for Liberty section below.

    • Diagnostic questions

      Please answer the following questions about the problem

      1. What activities were taking place when the problem was noticed? (Starting a server, enabling security, a user logging on, changing security configuration, managing users or groups, etc.)
      2. Are you able to consistently make the problem happen or is there no apparent trigger or timing?
      3. If you can consistently make the problem happen, what steps do you take?
      4. On which JVM(s) are you seeing the problem?
        • For WebSphere traditional: deployment manager, node agent, server1, cluster member, etc.
      5. Are there messages you can point us to in the SystemOut.log, or something else that has a timestamp of the problem?
        • If not, please provide your best estimate of the date/time and where (what JVM) the problem occurred.
      6. Have you used the WebSphere Application Server Support Portal to research this problem?
      7. Have you taken any steps to try and resolve this issue?
        • If so what steps did you take?
    • Step-by-Step
      • Before you collect data, be sure to answer the Diagnostic questions in the section above.
      • You may choose to follow this step-by-step document or you can watch the video in the Video section below.
      • Security issues on WebSphere may be difficult to troubleshoot. Please make sure to collect all the information described below. When all the information for your issue is ready, follow the instructions on Exchanging information with IBM Technical Support for problem determination to send the information and files that you collected.

      SET UP WEBSPHERE TRADITIONAL FOR CORE SECURITY TRACING

      1. Enable trace core security trace
        • If the Administrative Console is not functioning:
        • If you can use your Administrative Console:
          1. In the Administrative Console, expand Troubleshooting > Logs and Trace > server_name > Diagnostic trace service
          2. Configure the trace file specifications
            1. Under Trace Output, select File
            2. Set Maximum File Size and Maximum Number of Historical Files
              • Security traces are quite large. You will need to change the Maximum Number of Historical Files and Maximum File Size to a sufficient number to capture the problem. Set the Maximum Number of Historical Files to at least 20 and the Maximum File Size to at least 20MB.
            3. Click Apply
          3. Set the trace specification
            1. Click Change log level details
            2. If there is a trace specification in the box, delete it
            3. Enter your trace string:
              • Start with this core security trace string:

                *=info:com.ibm.ws.security.*=all:com.ibm.websphere.security.*=all:
                com.ibm.websphere.wim.*=all:com.ibm.wsspi.wim.*=all:com.ibm.ws.wim.*=all

              • If the problem is related to authentication to an Enterprise JavaBean™, append the following to the trace string:

                :SASRas=all:ORBRas=all

              • If the problem is related to security domains, append the following to the trace string:

                :SecurityDomain=all

            4. Click Apply
          4. Click Save
      2. Enable COMM trace (only if requested)


       

      COLLECT WEBSPHERE TRADITIONAL CORE SECURITY TRACES

      Avoid delay: Unless otherwise instructed, it is important that WebSphere core security traces be gathered from server startup.

      For each WebSphere server that you are tracing:
      1. Stop the server.
      2. Backup and clear the logs and FFDC directories.
      3. Start the server
      4. Reproduce the problem, making note of time when the problem occurs

      Remember to 'undo' the tracing after the issue has been resolved.

       

      GATHER WEBSPHERE TRADITIONAL CORE SECURITY DATA TO SEND TO IBM

      Avoid delay: All of the following data is required for proper problem determination of most issues. Do not send a subset of this data unless you were instructed to do so by IBM support.
       
      File to send
      Instructions
      Diagnostic questions Answer the Diagnostic questions in the section above.
      A collector jar

      Note: You need to run the collector tool on each <PROFILE_ROOT> in which you enabled trace.
       
      From a temporary directory, run the Collector Tool, collector.sh or collector.bat, which is located in the <PROFILE_ROOT>/bin directory.

      If there is a message about the collector tool being deprecated, ignore it; this is the tool IBM support needs you to run.
      waspolicies If you are using multiple security domains please send us all the files/directories under:

      <PROFILE_ROOT>/config/waspolicies

      See the information in Exchanging information with IBM Technical Support for problem determination to send this diagnostic information to IBM support.
       
    • Video

      This section is for collecting data for WEBSPHERE TRADITIONAL. If you want to collect data for Liberty, see the Collect data for Liberty section below.

      • Before you collect data, be sure to answer the Diagnostic questions in the section above.
      • You may choose to watch this video or follow the step-by-step instructions in the in the Step-by-Step section above.
      • Security issues on WebSphere may be difficult to troubleshoot. Please make sure to collect all the information described in the video. When all the information for your issue is ready, follow the instructions on Exchanging information with IBM Technical Support for problem determination to send the information and files that you collected.
      • The following video goes over the necessary steps to collect data for a core security problem on WebSphere traditional:
  • Collect data for Liberty
    This section is for collecting data for LIBERTY. If you want to collect data for WebSphere traditional, see the Collect data for WebSphere traditional section above.
     

    SET UP LIBERTY FOR CORE SECURITY TRACING

    1. Set up the Liberty server for core security tracing
    2. Verify that your tracing is working as intended
      1. Stop the Liberty Server
      2. Delete any existing logs files found under the logs directory:
        <LIBERTY_HOME>/usr/servers/<serverName>/logs
      3. Restart the Liberty Server and review the logs to confirm that they are recent.
      4. Verify that the new Liberty trace setting has been picked up by reviewing the upper part of the trace.log file.
     

    COLLECT LIBERTY CORE SECURITY TRACES

    Avoid trouble: It is important that core security traces be gathered from Liberty server startup.
    1. Stop the Liberty server
    2. Restart the Liberty server
    3. Reproduce the problem, making note any information that may be useful when processing the trace, such as:
      • Relevant user/group names used
      • Exact URL strings accessed
      • General time stamps
     

    GATHER LIBERTY DATA TO SEND TO IBM SUPPORT

    • Use the "dump" command to generate a .zip file containing the logs and config files which can be sent to support.
      • For Windows platforms, run:
        <LIBERTY_HOME>\bin\server.bat dump <serverName>

        For UNIX platforms, run:
        <LIBERTY_HOME>/bin/server dump <serverName>
      • Collect the resulting dump zip file with date & time. These files can be found under the following path:
        <LIBERTY_HOME>/usr/servers/<serverName>

        File name example:
        (myserver.dump-17.03.20_22.20.57.zip)
    • Collect the java.security file from your JDK. This file can be found under the following path:
      JAVA_HOME\lib\security\java.security

    Avoid delay: Make sure that traces that you send to IBM support covers a time frame where your issue occurs.

    See the information in Exchanging information with IBM Technical Support for problem determination to send the trace file(s) and supplemental information to IBM support.


Note:

This document uses the term WebSphere traditional to refer to WebSphere Application Server v9.0 traditional, WebSphere Application Server v8.5 full profile, WebSphere Application Server v8.0 and earlier, WebSphere classic, traditional WebSphere, traditional WAS and tWAS.
 

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Security","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5;8.0;7.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
10 November 2020

UID

swg21470063