Central manager redundancy

Use Central Manager Redundancy or Backup Central Manager (CM) to configure a secondary or backup CM in case the Primary CM becomes unavailable.

Central Manager redundancy supports the following:

  • Backup Central Manager - Make Primary CM link is available after the primary central manager loses connection.
  • User Layouts are retained.
  • User and roles are in the synch backup and do not rely on Portal User Sync.
  • User Group Roles Data are retained.
  • An API function make_primary_cm allows you to switch the central manager from the CLI.
  • Data is retained from Audit Process Builder processes after switching Primary Central Manager to Backup Central Manager.
  • Central Management backup includes all the definitions (reports, queries, alerts, policies, audit processes etc.), users and roles as it did before.
  • It includes the schedules for enterprise reports, distributed reports and LDAP.
  • It includes schedules for all audit processes; schedules and settings for data management processes such as archive, export, backup, and import; populate group members from query.
  • It includes settings for Alerter and Sender.
  • User's GUI customization's, custom classes and uploaded JDBC drivers are included.
Note:
  • Data, either collected data, audit results and custom tables data, is not included.
  • The Top risky users list and threat cases are not copied to the backup central manager.
  • To list status of cm_sync_file(s) on Backup CM, use the CLI command, show local_cm_sync_file. To list the value of Backup CM IP for each managed unit, use the GuardAPI command, grdapi show_backup_cm_ip (this API command can only run on a central manager).
  • Failover with Central Manager load balancing - After failover, if the new managed units connect and then disconnect right away, the correct DB_USER is not sent until the failover message is received.
  • Switching to a backup central manager interrupts communication with collectors and may generate the following message: "Central manager experienced failed data transfer from collector." The issue is visible in the Scheduled Jobs Exceptions report and should clear within 24-hours.

Perform these steps on your development or secondary servers and test. If successful, then perform these steps on your Primary or live Guardium Servers.

Install Patches on Central Manager
  1. From the now Primary CM, login as CLI.
  2. Install patches with the following CLI command, store system patch install scp
  3. This CLI command will copy the files over to your Guardium Server and give you the ability to install them.
  4. Watch these patches being installed with the following CLI command, show system patch install
  5. Wait until the patch status shows “DONE: Patch installation Succeeded.” for both patches.
Install Patches on Backup CM
  1. Login into the now Primary CM GUI as admin.
  2. Navigate to Manage > Central Management > Central Management.
  3. Click check boxe for the Backup CM managed unit ONLY on the Central Manager.
  4. Click Patch Distribution and install all of the patches that you just installed onto the Primary CM.

Example to install a patch:

  1. Click Patch Distribution.
  2. Click Install Patch Now.
  3. Wait approximately 15 minutes to be sure the patch is installed on all managed servers.
  4. To verify, login as CLI on the Backup CM and run CLI command, show system patch install, from Backup CM server.
Install Patches on all other managed servers (optional steps)
  1. Repeat the previous steps to install patches on all managed servers.
  2. Verify that all patches have been installed before going to the next procedure.
After all Patches have been installed on the CM and managed servers
  1. Login as admin onto the now Primary CM.
  2. Navigate to Manage > Central Management > Central Management and click Designate Backup CM.
  3. Select Backup CM server from the returned list of eligible Backup CM candidates.
  4. Click Apply.
  5. Wait approximate two minutes for the Backup CM to sync and the NEW Backup CM file to be created and copied to the Backup CM.
  6. Wait for two complete rounds of backups to complete (approximate 1 hour) for two Backup CM sync files that will be copied to the Backup CM and can be viewed from the Guardium Monitor tab - Aggregation Archive Log Report.
  7. Select Guardium Monitor and select Aggregation/Archive Log Report to view the progress of the creation of the Backup CM sync file.
  8. Verify the Activity Backup has started and the cm_sync_file.tgz file has been created from the Aggregation/Archive Log Report.
    1. Login as Admin from the GUI.
    2. Select Guardium Monitor tab.
    3. Select Aggregation/Archive Report.
    4. Look for Backup Types.
  9. When complete:
  10. The patches have been installed on the CM.
  11. The patches have been installed on the Backup CM.
  12. Option: The patches have been installed on all other managed units.
  13. Two Backup CM Sync files have been completed (see Aggregation/Archive Log file under Guardium Monitor Tab).
  14. The following steps outline the process to convert the now Primary CM and its managed nodes to the Backup CM.
  • IMPORTANT: Wait approximately one hour to be sure at least TWO of the Backup CM sync files supporting Backup CM have completed.
  • The backups schedule for Backup CM sync files is approximately every 30 minutes.
  • The process will run on the CM to create a backup CM file and copy that file to the directory on the Backup CM.
Start the Backup CM Process after two sync file process have completed

Shutdown the Primary CM Guardium Server

If you have no access to shutdown the Primary CM, then go directly to the Backup CM and login as Admin. Navigate to Manage > Central Management > Central Management and click Make Primary CM. Skip to the section "Steps to start the Backup CM configuration to become the Primary CM" in this document.

  1. Wait approximate five minutes and login again as admin in the GUI of the Backup CM.
  2. Once the Primary CM is shutdown completely, you can continue onto the next step

If you are logged into the Primary CM and it goes down, you get a message indicating that the connection has timed out.

Steps to start the Backup CM configuration to become the Primary CM

The secondary CM will not be responsive for approximately five minutes. Login after five minutes and the Make Primary CM link will be available. The link is available under the admin login at Manage > Central Management > Central Management.

  1. When the Primary Server goes down, you will get a message on the Backup CM “Unable to connect to Remote Manager, consider switching to (the name of the backup CM)".
  2. If you decide to switch:
    1. Login as admin
    2. Navigate to Manage > Central Management > Central Management.
    3. Click Make Primary CM (do not click the “Make Primary CM” link more than once. Also stay on this screen and do not select anything else during the running of this process. A log file will be created that you can view to see the progress and completion of this process.) Be patient as this process will take awhile to complete. There is a safeguard that if you do click this button more than once nothing will change with the current running process.
    4. Within seconds you should get a message “Are you sure you want to make this unit the primary CM? Click OK.
    5. Within a few seconds more you will get a message stating “This may take a few minutes”. The time it takes for the Backup CM to become the primary CM depends on the amount of data backed up from the Backup CM sync file and the amount of managed nodes that switch to the Backup CM which will become the Primary CM. Click OK. As soon as we click OK a log file will be created called load_secondary_cm_sync_file.log that will allow you to view the progress of the switch to the completion of the Backup CM switch process. This file can be viewed from your GUI. The following steps indicate how to view this log file.
    6. The last message will take a while to be presented to the screen. It will be the last message before the Backup CM switch has completed. The message is “GUI will restart now. Try to login again in a few minutes and the Backup CM will now become the Primary CM”. Click OK. Wait a few minutes for the Backup CM to become Primary and for all the managed nodes to complete switching over to the new Primary CM.
While the CM Backup Process is running – viewing the progress log file

From the Backup CM while the Make Primary CM process is running, you can do the following to view the progress of the Backup CM becoming the Primary CM.

Prerequisite: You will need the IP of the server you are connected to in order to view the log files.

  1. Login as CLI from your Backup CM server from a Putty.exe session
  2. From CLI run Fileserver <IP> “enter your IP number” 3600", for example: fileserver 9.70.32.122 3600
  3. From the GUI, enter the value: http://yourserver.x.x.x.com (will display in the CLI screen after entering the command, example: http://joe.server.guardium.com (the server name will be the Backup CM server).
    Fileserver Window on the UI will open to select file – Select Sqlguard logs
  4. Select the file: load_secondary_cm_sync_file.log. (The file will display in a list of files from Step #3.) This will allow you to view the progress of the Backup CM becoming the Primary CM. Locate log file for viewing
    CM Backup Process is complete when you see this line in the load_secondary_cm_sync_file.log
    Import CM sync info - DONE
  5. Wait approximately 10 minutes for all the Managed units to become available to the New Primary CM.
After the Backup CM becomes the Primary and all Managed nodes are now managed by the Backup CM server

You can now bring up the old CM server. Once it is up and running, perform the following steps to add it as the Backup CM server.

  1. Reboot Old Primary CM.
  2. Once the Server is up, login as CLI.
  3. Delete the manager unit type, enter delete unit type manager.
  4. After it completes and you get an OK message from CLI.
  5. VERY IMPORTANT: Wait approximately five minutes for the GUI to completely restart even after the deleted unit type displays a successful message and the GUI restart message.
  6. After five minutes, log into the New Primary CM to register Old CM as a managed unit.
  7. Login as admin on New Primary CM.
  8. Navigate to Manage > Central Management > Central Management.
  9. Click Register New.
  10. Enter IP of the Old Primary CM that you just rebooted.
  11. Enter 8443 as Port.
  12. Click Save. (IMPORTANT: Be patient, do not click this button twice).
  13. Wait a minute for the Old Primary CM to become registered.
  14. Make the Old Primary CM a New Backup CM.
  15. Click Designate Backup CM.
  16. Click on Old Primary CM server.
  17. Click Apply.
  18. Old Primary CM server is NOW the New Backup CM server.
  19. Refresh Central Management screen to see the New Unit type Backup CM defined.
  20. This task is complete.
Report Data After Backup CM Process is complete

The following data is missing after the Backup CM process is completed. This is related to only the "first" switch from the Primary to the Secondary CM.

Missing Data:

  1. Audit Process Results
  2. Custom Table Data
  3. Custom Report Data
  4. VA Results
  5. Classifier Results
  6. DSD Results
  7. CAS results
  8. Datamart Data
  9. Collected Data
  10. Entitlement Data

The reports are populated again is once you run these reports again on the New Primary CM. If you switch back to the old Primary CM, the data for these reports will be presented.