Upgrading Best Practices

1 like Updated 5/2/18, 11:19 AM by JanetTaylorTags:


Important Note:  Beginning in BigFix version 9.5.3 the Device Type property in the console evaluates differently.  The values returned are completely dependent upon the type of underlying hardware on the endpoint (instead of the class of OS installed).  BigFix agents running on VM guests will always return Server for Device Type.  For other endpoints, the chassis type of the machine determines whether Device Type returns as Server, Desktop, or Laptop.  If you heavily depend on this property in your automatic groups and policy actions, it is critical that you first validate the upgrade experience in a test deployment first.  See the following dW Answer for additional information:  https://developer.ibm.com/answers/questions/304878/why-is-device-type-is-unexpectedly-reporting-serve/


Upgrading documentation



IBM BigFix upgrades are designed to be simple and easy while also providing control and flexibility for customers. Since BigFix can update itself, deploying an upgrade is typically as easy as deploying any Fixlet.

All BigFix components can be upgraded manually, but it is generally easier to upgrade the majority of components using the upgrade Fixlets. For more information about manual upgrade instructions, click Upgrading.



The following covers a number of recommended planning items to consider when upgrading the BigFix environment:

Important Notes:

The upgrade to version 9.2 Patch 5 takes longer than other upgrades. The duration is proportional to the amount of subscribed external contents, the number of client uploads on the server and the speed of server hardware. For some deployments it could take multiple hours to complete. Smaller deployments with lots of external content are most likely to experience longer upgrade time, for example 100,000 external site fixlets could take up to 2 hrs to upgrade, and 200,000 could take 3-4 hrs.

While the upgrade is running, its progress is not displayed and the installer dialog becomes unresponsive.

**Do not stop or abort the upgrade process once started.**

To check if the upgrade is still in progress, refer to the following technote or contact IBM support.

As part of this upgrade new database tables are filled in with data, so expect database space usage growth. For example, in a sample test environment with about 20,000 Fixlets, the database usage might use additional 1.5GB of disk space. Deployments with several foreign language patch sites will experience larger database growth of 10-20GB.

IBM recommends testing the server upgrade in an isolated lab environment as described below prior to completing a production upgrade.


General order of upgrade:

The following describes the recommended order that BigFix components be upgraded:

  • The Root Servers must be upgraded before Relays, which must be upgraded before Agents.

  • Root Server, Web Reports, and Consoles must match version and release and must be upgraded at the same time. For example, a 9.2.x Root Server requires a 9.2.x Console and Web Reports.

  • Root Servers, Relays, and Agents do not need to match versions and the upgrade of the different components can occur at different times.

  • Older Agents can continue to report to newer Relays or Root Servers, but they will not have all the functions that are available in the new releases.

  • For customers using Disaster Server Architecture (DSA), one DSA Server can be upgraded first to ensure that the upgrade is successful and the other DSA Servers can be upgraded afterwards. For more information, see Disaster Server Architecture.


Testing overview:

It is recommended that all BigFix upgrades go through a phase of testing prior to implementation within a production environment.  Although, our upgrades are generally fast and reliable, our in-house testing cannot account for environment-specific details.  As the size and complexity of a deployment grows, the importance of testing within the specific environment grows as well.  Below are some sample recommended high-level tests:

  • Sample upgrade of production BigFix root server and Web Reports databases: tests general upgrade process as well as provides rough estimate of upgrade duration.

  • Validation that all services start up and that the console and Web Reports load properly.

  • Validation testing of sample data and processes such as action deployment, downloads, reports, and so on.

  • Validation of additional related components, for example Security Compliance Analytics (SCA), Software Use Analysis (SUA), plugins.

  • Validation of any customizations or data extraction processes.

  • Existing functionality and potential integrations.

  • New features of interest


Testing preparation:

  1. Prepare a Lab environment that matches the configurations within Production.

  2. Important: Ensure that the Lab environment is *network-isolated* from the Production Servers. This can be achieved by using network segmentation, or by HOSTS file modifications on the Lab server.

    If using HOSTS file for isolation, ensure that at least the following point to localhost:

  • The URL/name specified in the masthead.

  • The servers configured within the ODBC System DSNs related to BigFix .

  • Any replication/DSA servers as defined in the REPLICATION_SERVERS table within BFEnterprise.

  1. Use steps similar to those listed in Server Migration to prepare a clone of the BigFix Server to be tested with the following considerations:

  • Follow a backup and restore approach rather than detaching and attaching the BigFix databases.

  • Do not stop the BigFix Server services on the Production server at the end of the process as this is not an actual migration.

  • Install the various components in the same paths as in Production.

  1. Deploy and configure Web Reports in the Lab as it is in Production.

  • Install Web Reports in the same path as in Production.

  • Web Reports instance running on Root Server versus remote/dedicated instance.

  1. Deploy and configure SUA, if applicable, in the Lab as it is in Production

  2. Deploy and configure SCA, if applicable, in the Lab as it is in Production


Testing (Sample Tests):

  1. Prepare for upgrade procedure

  • Upgrade installers folder

  • Close all console connections

  • Ensure backup and database maintenance routines are not running

  1. Upgrade Root and Console Server

  • Upgrade Root server

    • After upgrade of Root Server, run BES Diagnostics tool to check for errors

  • Upgrade Web Reports

    • Validate successful install by logging into Web Reports and performing an update to the cache and confirming the build version on the upper right corner of the web page.

  • Upgrade Console

    • Validate successful install of console application by logging into console. Verify version of the console application via the help/about menu.

  • Document any errors encountered

  1. Test Basic Functionality of Root Server and Console

  • Issue Actions as a Non-Master Operator

    • Verify action status from client

  • Issue actions as a Master Operator

    • Verify action status from client

  • Delete actions issued by operators in step 4a/4b

  • Issue Site subscriptions as Master Operator

    • Verify site subscription action is propagated and properly interpreted by client

  • Revoke/remove site subscriptions as Master Operator

    • Verify site removal subscription action is propagated and properly interpreted by client

  • Delete existing Non-Master Operator

    • Verify Non-Master Operator is no longer able to log in

  • Delete existing Master Operator

    • Verify Master Operator is no longer able to log in

  • Create/provision a Non-Master Operator

    • Assign Computer Management and Site Permission

    • Local versus LDAP (if applicable)

    • Validate by logging in

  • Create/provision a Master Operator

    • Local versus LDAP (if applicable)

    • Validate by logging in

  • Validate Client functionality

    • Verify CPU usage of Agent is at expected levels (~2% at idle by default)

    • Check Memory Usage of Agent is within expected bounds (this can vary between versions and between platforms, but can be compared against the memory usage of existing production Agents).

  • Test new installers source

    • Use Agent install source within BES installers to install Agent

      • Verify that the Agent reports in and receives new/open actions

  1. Test basic functionality of Web Reports

  • Log on to Web Reports as a ‘normal’ user

    • Verify proper application of existing permissions associated to user/role (ex: verify correct listing of computers associated to role)

    • Run some stored reports, and validate that the data returned is expected

    • Check that saved filters work as expected

  • Log on to Web Reports as an Administrator

    • Validate general Web Reports configuration


Upgrade (Core Components)


  • In general, follow and execute a backup/restore procedures for your BigFix server application and database to prepare to recover in the event the upgrade fails and the application and database need to be restored back to the current (before upgrade attempt) version (Windows | Linux).

  • Database backup – verify restoration to ensure backups are valid/useable

  • Reduce Reporting Rate – increase heartbeat (to 120+ minutes), minimum report interval (to 21600)

  • Check database consistency:

    • MSSQL: dbcc checkdb

  • Back up BigFix Server application files

    • On Windows:

      • ..\BigFix Enteprise\BES Server

    • On RHEL: 

      • /var/opt/BESServer

      • /var/opt/BESWebReportsServer

  • DSA: increase replication intervals

  • Change the following BigFix Client settings on all clients:

    • _BESClient_Report_MinimumInterval = 3600 *This setting will reduce the amount of incoming data from the endpoints to allow the system to recover more quickly and reduce potential downtime.

    • _BESClient_RelaySelect_ResistFailureIntervalSeconds = 21600 *This value represents the amount of time BES Clients will wait after its relay appears down before performing BES Relay selection. This can prevent unnecessary automatic relay selection during the migration.

  • Change the heartbeat in the BigFix Console to 6 hours: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Endpoint%20Manager/page/Console%20Preferences * This is another way to reduce the amount of incoming data from the endpoints.

  • You may want to lock client computers, see Client Locking.  Consider locking your most critical endpoints along with a set of lesser critical endpoints that are similar in build, environment, OS, and action policy. Locking will prevent actions from happening on these endpoints until you unlock them.  After the upgrade, you can then unlock the similar set first and validate.  Then, incrementally unlock the critical endpoints, least critical to most critical to validate.


Day of, Hours Before Change:


  • Send upgrade notification/communication to appropriate end users

  • Close any open BigFix Consoles 

  • Shut down Services

    • Any DSA replicas

    • Any dedicated Web Reports instances


  • Upgrade Root Server

  • Upgrade any dedicated Web Reports instances

  • Re-enable Services for the TEMA servers

  • Upgrade BigFix Consoles

  • Open one Console for login

  • Execute tests



  • Reset Client reporting frequencies to pre-upgrade value 

    • Heartbeat

    • _BESClient_Report_MinimumInterval

    • _BESClient_RelaySelect_ResistFailureIntervalSeconds

  • Reset Console Minimum Refresh Interval to pre-upgrade value

  • Decrease WebReports Cache Interval to pre-upgrade value

  • Upgrade Relay infrastructure “top-down”

  • Upgrade Endpoints

  • Unlock any computers that had been locked (see Pre-Upgrade note above)


Failed Upgrade:
  • If the upgrade failed, implement the restore procedures referenced back in the pre-upgrade steps (backup/restore procedures) to bring the BigFix server application back to the version it was at before the upgrade was attempted.
  • Ensure you have made a backup copy of the following key files before removing the BigFix server application: 
    • License.crt, License.pvk, EncryptedServerSigningKey, EncryptedClientCAKey, EncryptedPlatKey, EncryptedWebUICAKey, EncryptedAPIServerKey