IBM Support

Important: A critical issue has been identified in /opt/qradar/bin/qchange_netsetup (IJ31239)

News


Abstract

QRadar® development identified a defect in the product where running the qchange_netsetup utility can cause critical configuration issues on appliances. Administrators with QRadar® V7.4.1 or earlier are instructed to confirm information in qradar_netsetup.log before you complete any network changes that use the /opt/qradar/bin/qchange_netsetup utility.

Content

Change list


  • 16 April 2021: Updated the hostname validation section to add 7.4.2 Fix Pack 2 as it was missing from the list of software version that can block and upgrade of an appliance if the hostname includes capital letters.
  • 15 April 2021: Added information for users on the changes deployed in the 14 April 2021 weekly auto update for qchange_netsetup in the section:
    + Auto update and enabling qchange_netsetup.
  • 31 March 2021: Added APAR IJ31239 for users to subscribe to this issue.
  • 16 March 2021: Corrected a navigation error in the technical note.
  • 11 March 2021: Added procedures for network changes or software upgrades for administrators. These procedures must be followed when a network configuration update is completed with qchange_netsetup or when you receive an invalid hostname error when you pretest your appliances before an upgrade.

    Note: This change list is updated to provide dates to users monitoring this software issue.

Urgency

Critical: QRadar® development identified a defect in the network component /opt/qradar/bin/qchange_netsetup where a hostname issue can cause a critical error, impacting the appliance configuration. This technical note provides procedures for administrators who need to complete network changes or complete upgrades where a hostname change is required. As of 14 April 2021, qchange_netsetup a QRadar® weekly auto update prevents users from running qchange_netsetup. This flash notice was updated to inform administrators on QRadar® V7.4.1 or earlier of this change and includes instructions on how to re-enable qchange_netsetup.

Users on affected versions are alerted when they attempt to run qchange_netsetup with the following message:

The qchange_netsetup script execution was disabled.
For more information, see [APAR](https://www.ibm.com/support/pages/apar/IJ31239).

Affected products


Administrators must alert their teams to validate the logs for qchange_netsetup utility on the following versions:
  • QRadar® 7.4.1 all fix pack (patch) versions
  • QRadar® 7.4.0 all fix pack (patch) versions
  • QRadar® 7.3.3 all fix pack (patch) versions
  • QRadar® 7.3.2 all fix pack (patch) versions
  • QRadar® 7.3.1 all fix pack (patch) versions
  • QRadar® 7.3.0 all fix pack (patch) versions
  • QRadar® 7.2.8 all fix pack (patch) versions
Note: Administrators on QRadar® 7.4.2 (all fix pack versions) and QRadar® on Cloud Consoles do not experience the qchange_netsetup error. If you are on QRadar® V7.4.2 or later, you are not required to monitor qradar_netsetup.log before you make a network configuration change.

Auto update and enabling qchange_netsetup

The 14 April 2021 weekly auto update (build 1618430623) includes a utility that disables qchange_netsetup by default for customers to raise awareness to the host configuration issues that can occur when qchange_netsetup is run. This alert is displayed to all users on affected QRadar versions. Administrators who need to update their network configuration can enable qchange_netsetup on the appliance, but it is critical that you confirm the 'Run by' field in the logs to ensure that host configuration settings are not affected.

Users on affected versions are alerted when they attempt to run qchange_netsetup with the following message:

The qchange_netsetup script execution was disabled.
For more information, see [APAR](https://www.ibm.com/support/pages/apar/IJ31239).
To complete important network changes for an appliance, you must:
  1. Enable qchange_netsetup on the appliance.
  2. Confirm the getParentName in the qchange_netsetup.log displays 'Run by qchange_netsetup'. For example,
    Apr 12 12:03:13 qradar_netsetup.py[8823]: qradar_netsetup getParentName [INFO] Run by qchange_netsetup

WARNING: Enabling qchange_netsetup bypasses safeguards IBM included to prevent potential configuration corruption on the QRadar appliance. If you are unsure of the impact of this change or require assistance updating your network configuration for APAR IJ31329, contact QRadar Support for assistance before you continue.
Procedure

This procedure enables qchange_netsetup on all appliances. QRadar Support recommends that administrators advise all users with root access who might run qchange_netsetup of this issue to prevent corruption of the configuration settings on the QRadar appliance. Administrators on QRadar V7.4.2 (any fix pack level) are not affected by this qchange_netsetup issue.
 

  1. Use SSH to log in to the QRadar Console as the root user.
  2. To enable qchange_netsetup, enter the following command:
    /opt/qradar/support/all_servers.sh -Ck \
    "[ -f /opt/qradar/bin/qchange_netsetup.bkp.py ] && \
    /usr/bin/mv -vf /opt/qradar/bin/qchange_netsetup.bkp.py /opt/qradar/bin/qchange_netsetup.py && \
    ln -svf /opt/qradar/bin/qchange_netsetup.py /opt/qradar/bin/qchange_netsetup && \
    ln -svf /opt/qradar/bin/qchange_netsetup /usr/sbin/qchange_netsetup"
  3. Wait for the command prompt to return.

    Results
    The qchange_nsetsetup utility is enabled on all appliances affected by IJ31329, you must review the next section How to update your network configuration with qchange_netsetup to ensure you safely run qchange_netsetup on your appliance. Administrators on affected versions can avoid APAR IJ31239 by upgrading to QRadar 7.4.2 (any fix pack level). To find the latest QRadar versions, see the QRadar 101 Software page.

How to update your network configuration with qchange_netsetup

Administrators can use SSH to tail the qradar_netsetup.log file to confirm the network change does not corrupt the host configuration. In this procedure, the administrators can use an SSH session to tail qradar_netsetup.log and their console keyboard, crash cart, or IMM to run the qchange_netsetup command. QRadar Support advises users to review the qradar_netsetup.log for all IP address changes, hostname changes, or DNS changes on any appliance at QRadar version 7.4.1 or earlier.

Procedure

  1. Use SSH to log in to the QRadar host that requires a network change as a root user.
    Note: If a network change is required on a managed host, your SSH sessions must originate from the QRadar Console for security purposes.
  2. Navigate to /var/log/setup-<your version>.
    Note: If you have multiple setup folders, navigate to the folder with the newest date in the folder name.
  3. To tail the qradar_netsetup.log, type:
    tail -f qradar_netsetup.log | grep -i 'run by'
  4. Switch to your console keyboard, remote management interface (IMM), or crash cart.
  5. Type the following command: /opt/qradar/bin/qchange_netsetup
  6. When qchange_netsetup starts, wait at the Internet Protocol Setup screen. Do NOT continue.
    image 8717
    Figure 1: Administrators must wait at this screen and verify the qradar_netsetup.log output from the tail command.
  7. In your SSH session, review the output and confirm the Run by field displayed qchange_netsetup logs.
    • Correct:  Mar 10 12:03:13 qradar_netsetup.py[8823]: qradar_netsetup getParentName [INFO] Run by qchange_netsetup
    • Incorrect:   Mar 10 12:03:13 qradar_netsetup.py[8823]: qradar_netsetup getParentName [INFO] Run by -bash
      Important: Any value displayed in the Run by field that does not list qchange_netsetup is considered an error and can lead to configuration issues on the managed host.
  8. If you experience an incorrect output in the logs, use the arrow keys to select Cancel from the qchange_netsetup. Exiting qchange_netsetup restarts services on the QRadar appliance.
  9. After services restart, you might be required to reconnect your SSH session.
  10. From the console keyboard, VM console, or IMM interface, run the qchange_netsetup command again to confirm the Run by log output.
    Note: When you run qchange again, the Run by output must display [INFO] Run by qchange_netsetup.
  11. Complete your network changes.
    Note: A valid hostname is a string up to 24 characters that includes [a-z][0-9], period (.), and minus sign (-) only.

    Results
    After services restart, your network changes are complete. If you continue to experience an incorrect output in the qradar_netsetup.log, contact QRadar Support for assistance.

Validating your hostnames before you upgrade QRadar

Administrators upgrading QRadar are advised to run a pretest check to verify you do not experience an upgrade issue where a non-compliant hostname can halt the setup. This procedure can help administrators prevent upgrade issues where a deployment is partially upgraded or you experience invalid hostname errors.  If your managed host requires a hostname change, you must remove the appliance from the deployment or break the high availability (HA) pair before you can run qchange_netsetup to update your hostname.

A pretest is required as the following versions validate hostnames and can block QRadar upgrades on appliances:

  • QRadar 7.4.2 Fix Pack 2
  • QRadar 7.3.3 Fix Pack 6
  • QRadar 7.3.3 Fix Pack 7

Procedure
Mount the QRadar sfs image and pretest your upgrade before you start a Console upgrade. QRadar Support recommends that you pretest appliances several days in advance of your change window to identify appliances with incorrect hostnames. If a pretest identifies a hostname issue, you must update your network configuration with qchange_netsetup to ensure that the hostnames are lowercase or RFC4343 compliant. Administrators must monitor the qradar_netsetup.log when you update your network settings.

  1. Create the /media/updates directory by typing the following command: mkdir /media/updates
  2. Mount the QRadar sfs image by typing the following command: mount -o loop <path_to_the_QRadar_SFS> /media/updates
  3. To pretest your appliance, type: /media/updates/installer -t
  4. If an invalid hostname is detected, the following error is displayed:
    ==============================================================
    Patch: 2072
    --------------------------------------------------------------
    At least one invalid hostname entry was found in one or more deployment settings.
    Contact support to evaluate the required changes to your deployment.
    See patch log file /var/log/setup-2020.3.2.20201116214411/patches.log for more details.
     
    Abort the software update to fix the invalid hostname entries found.
    Choices:
    1) Yes, abort the software update.
  5. Type 1 to exit the pretest.
  6. Select one of the following options to remove the appliance from the deployment:
  7. Open an SSH session to the QRadar appliance.
  8. Navigate to /var/log/setup-<your version>.
    Note: If you have multiple setup folders, navigate to the folder with the newest date in the folder name.
  9. To tail the qradar_netsetup.log, type: tail -f qradar_netsetup.log | grep -i 'run by'
  10. Connect to the appliance from your console keyboard, remote management interface (IMM), or crash cart.
  11. Type the following command: /opt/qradar/bin/qchange_netsetup
  12. When qchange_netsetup starts, wait at the Internet Protocol Setup screen. Do NOT continue.
    image 8717
    Figure 2: Administrators must wait at this screen and verify the qradar_netsetup.log output from the tail command.
  13. In your SSH session, review the output and confirm the Run by field displayed qchange_netsetup logs.
    • Correct:  Mar 10 12:03:13 qradar_netsetup.py[8823]: qradar_netsetup getParentName [INFO] Run by qchange_netsetup
    • Incorrect:   Mar 10 12:03:13 qradar_netsetup.py[8823]: qradar_netsetup getParentName [INFO] Run by -bash
      Important: Any value displayed in the Run by field that does not list qchange_netsetup is considered an error and can lead to configuration issues on the managed host.
  14. If you experience an incorrect output in the logs, use the arrow keys to select Cancel from the qchange_netsetup. Exiting qchange_netsetup restarts services on the QRadar appliance.
  15. After services restart, you might be required to reconnect your SSH session.
  16. From the console keyboard, VM console, or IMM interface, run the qchange_netsetup command again to confirm the Run by log output.
    Note: When you run qchange again, the Run by should display the correct output as [INFO] Run by qchange_netsetup.
  17. Update the hostname on your appliance. 
    Note: A valid hostname is a string up to 24 characters that only include [a-z][0-9], period (.), and minus sign (-).
  18. Select one of the following options to readd and reassign the appliance from the deployment:
    Results
    After the host or high availability (HA) pair is added to the deployment, the procedure is complete. Repeat these steps to pretest other appliances in your deployment before you attempt to upgrade.  If you continue to experience an incorrect output in the qradar_netsetup.log, contact QRadar Support for assistance.

We apologize for any inconvenience that this validation causes for administrators. If you need assistance or have questions about this flash notice, you can contact QRadar Support for assistance and attach logs from the appliance that needs a network change to your case. Users who want to track this issue can subscribe to APAR IJ31239.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2.8;7.3.0;7.3.1;7.3.2;7.3.3;7.4.0;7.4.1"}]

Document Information

Modified date:
16 April 2021

UID

ibm16427875