IBM Support

QRadar: Migration from GlusterFS to Distibuted Replication Block Device on Event Collector terminates due to stale PID file

Troubleshooting


Problem

The QRadar® upgrade to version 7.4.2 requires you to run a migration script on the console. This script migrates the High Availability file system from GlusterFS to Distributed Replication Block Device on all Event Collectors in your deployment:
  
/opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin
In some scenarios, the script terminates because the /var/run/glusterfs_migration.pid file exists (from a previous execution of the script) and it no longer points to a valid path on the /proc file-system because the older PID is not valid.

Symptom

The glusterfs_migration_manager script displays an error:
IOError: [Errno 2] No such file or directory: '/proc/82258/cmdline'
Failed to execute script glusterfs_migration_manager
NOTE: More information about the /proc file-system can be found here.

Cause

This error occurs when the previous run of the migration script has terminated unsuccessfully and has left behind a /var/run/glusterfs_migration.pid file that points to the older PID of the script (in this case 82258):
cat /var/run/glusterfs_migration.pid
82258

Environment

QRadar Event Collectors (stand-alone or in a High Availability setup).

Diagnosing The Problem

On the QRadar console, check the log file /var/log/remove_glusterfs.log for these messages:
[WARNING] During migration to DRBD, event collection does not occur.
[WARNING] QRadar uses /store during the migration to DRBD. All other processes that use /store are terminated during migration.
[WARNING] The migration to DRBD restricts software updates to 7.4.2 or higher.
Do you want to continue [yes/no]? yes
Traceback (most recent call last):
  File "glusterfs_migration_manager.py", line 699, in <module>
  File "glusterfs_migration_manager.py", line 682, in main
  File "glusterfs_migration_manager.py", line 556, in start_migration
  File "glusterfs_migration_manager.py", line 414, in create_migration_pid
IOError: [Errno 2] No such file or directory: '/proc/82258/cmdline'
Failed to execute script glusterfs_migration_manager

Resolving The Problem

  1. Download the newest version of the migration script.
    1. Navigate to IBM Fix Central.
    2. In the script section, type gluster.
    3. Create a working directory to save the file.
        
      mkdir -p /store/IBM_Support
    4. Download the file and copy it to the previously created directory in the Console.
    5. Navigate to the directory and unzip the file.
        
      cd /store/IBM_Support
unzip glusterfs_migration_manager-<script_version>.zip
  2. Assign executable permissions to the bin file.
      
    chmod +x glusterfs_migration_manager-<script_version>.bin
  3. Copy the bin file to the /opt/qradar/ha/bin/ directory in the Console:
      
    cp -pfv glusterfs_migration_manager-<script_version>.bin /opt/qradar/ha/bin/
  4. Copy the same file to the /opt/qradar/ha/bin/ directory in the EC:
      
    scp glusterfs_migration_manager-<script_version>.bin <Event Collector IP>:/opt/qradar/ha/bin/
  5. On the Console, delete the file /var/run/glusterfs_migration.pid:
      
    rm /var/run/glusterfs_migration.pid
  6. Run the migration script in the Console in pre-check mode:
      
    /opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin -p
  7. Run the migration script again; the script should now run successfully:
      
    /opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin -m
  8. The migration should now run successfully. Once it finishes, check the status of the migration:
      
    /opt/qradar/ha/bin/glusterfs_migration_manager-<script_version>.bin -s

Document Location

Worldwide

The words LINSTOR®, DRBD®, LINBIT®, and the logo LINSTOR®, DRBD®, and LINBIT® are trademarks or registered trademarks of LINBIT in Austria, the United States and other countries.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtdAAA","label":"Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.2"}]

Document Information

Modified date:
30 March 2021

UID

ibm16434547

Page Feedback