Upgrading Best Practices

This page has not been liked. Updated 7/31/15, 11:57 PM by SteveHullTags:

IBM Endpoint Manager (IEM) upgrades are designed to be simple and easy while also providing control and flexibility for customers. Since IEM can update itself, deploying an upgrade is typically as easy as deploying any Fixlet.

All IEM components can be upgraded manually, but it is generally easier to upgrade the majority of components using the upgrade Fixlets. For more information about manual upgrade instructions, click Upgrading.

 

Planning:

The following covers a number of recommended planning items to consider when upgrading the IEM environment:

  • Verify that the upgrade path is supported as documented in Upgrade Paths for version 9.2.

  • Verify that the existing infrastructure is supported under the target version of IEM

    For more information, see System requirements for version 9.2.
    • IEM Server OS

    • IEM Database Engine

    • 32- versus 64-bit platforms

    • Console platforms

    • Relay platforms

    • Endpoint platforms

  • Review major changes as well as the detailed change list

    For more information, see IBM Endpoint Manager Release Notes for version 9.2.
    • Determine key benefits and features of interest.

    • Identify potential problematic areas.

    • Pay particular attention to changes in behavior and how these may impact the environment.

  • Review documented known limitations at IBM Endpoint  Manager Known Limitations for version 9.2.

  • For large deployments, run the Root Server upgrade in a Lab environment that matches the configurations within Production, to know  the expected time frame required for the upgrade. Depending on how large and complex the environment is, the upgrade process may take several minutes to over an hour.

  • Run the Root Server upgrade manually to ensure the upgrade completes properly and to see any potential status or error messages encountered during the process to facilitate troubleshooting.

  • Notify users about the IEM upgrade in plan.

Important Notes:

The upgrade to version 9.2 Patch 5 takes longer than other upgrades. The duration is proportional to the amount of subscribed external contents, the number of client uploads on the server and the speed of server hardware. For some deployments it could take multiple hours to complete. Smaller deployments with lots of external content are most likely to experience longer upgrade time, for example 100,000 external site fixlets could take up to 2 hrs to upgrade, and 200,000 could take 3-4 hrs.

While the upgrade is running, its progress is not displayed and the installer dialog becomes unresponsive.

**Do not stop or abort the upgrade process once started.**

To check if the upgrade is still in progress, refer to the following technote or contact IBM support.

As part of this upgrade new database tables are filled in with data, so expect database space usage growth. For example, in a sample test environment with about 20,000 Fixlets, the database usage might use additional 1.5GB of disk space. Deployments with several foreign language patch sites will experience larger database growth of 10-20GB.

IBM recommends testing the server upgrade in an isolated lab environment as described below prior to completing a production upgrade.

 

General order of upgrade:

The following describes the recommended order that IEM components be upgraded:

  • The Root Servers must be upgraded before Relays, which must be upgraded before Agents.

  • Root Server, Web Reports, and Consoles must match version and release and must be upgraded at the same time. For example, a 9.2.x Root Server requires a 9.2.x Console and Web Reports.

  • Root Servers, Relays, and Agents do not need to match versions and the upgrade of the different components can occur at different times.

  • Older Agents can continue to report to newer Relays or Root Servers, but they will not have all the functions that are available in the new releases.

  • For customers using Disaster Server Architecture (DSA), one DSA Server can be upgraded first to ensure that the upgrade is successful and the other DSA Servers can be upgraded afterwards. For more information, see Disaster Server Architecture.

 

Testing overview:

It is recommended that all IEM upgrades go through a phase of testing prior to implementation within a production environment.  Although, our upgrades are generally fast and reliable, our in-house testing cannot account for environment-specific details.  As the size and complexity of a deployment grows, the importance of testing within the specific environment grows as well.  Below are some sample recommended high-level tests:

 

  • Sample upgrade of production IEM root server and Web Reports databases: tests general upgrade process as well as provides rough estimate of upgrade duration.

  • Validation that all services start up and that the console and Web Reports load properly.

  • Validation testing of sample data and processes such as action deployment, downloads, reports, and so on.

  • Validation of additional related components, for example Security Compliance Analytics (SCA), Software Use Analysis (SUA), plugins.

  • Validation of any customizations or data extraction processes.

  • Existing functionality and potential integrations.

  • New features of interest

 

Testing preparation:

  1. Prepare a Lab environment that matches the configurations within Production.

  2. Important: Ensure that the Lab environment is *network-isolated* from the Production Servers. This can be achieved by using network segmentation, or by HOSTS file modifications on the Lab server.

    If using HOSTS file for isolation, ensure that at least the following point to localhost:

  • The URL/name specified in the masthead.

  • The servers configured within the ODBC System DSNs related to IEM.

  • Any replication/DSA servers as defined in the REPLICATION_SERVERS table within BFEnterprise.

  1. Use steps similar to those listed in Server Migration to prepare a clone of the IEM Server to be tested with the following considerations:

  • Follow a backup and restore approach rather than detaching and attaching the IEM databases.

  • Do not stop the IEM Server services on the Production server at the end of the process as this is not an actual migration.

  • Install the various components in the same paths as in Production.

  1. Deploy and configure Web Reports in the Lab as it is in Production.

  • Install Web Reports in the same path as in Production.

  • Web Reports instance running on Root Server versus remote/dedicated instance.

  1. Deploy and configure SUA, if applicable, in the Lab as it is in Production

  2. Deploy and configure SCA, if applicable, in the Lab as it is in Production

 

Testing (Sample Tests):

  1. Prepare for upgrade procedure

  • Upgrade installers folder

  • Close all console connections

  • Ensure backup and database maintenance routines are not running

  1. Upgrade Root and Console Server

  • Upgrade Root server

    • After upgrade of Root Server, run BES Diagnostics tool to check for errors

  • Upgrade Web Reports

    • Validate successful install by logging into Web Reports and performing an update to the cache and confirming the build version on the upper right corner of the web page.

  • Upgrade Console

    • Validate successful install of console application by logging into console. Verify version of the console application via the help/about menu.

  • Document any errors encountered

  1. Test Basic Functionality of Root Server and Console

  • Issue Actions as a Non-Master Operator

    • Verify action status from client

  • Issue actions as a Master Operator

    • Verify action status from client

  • Delete actions issued by operators in step 4a/4b

  • Issue Site subscriptions as Master Operator

    • Verify site subscription action is propagated and properly interpreted by client

  • Revoke/remove site subscriptions as Master Operator

    • Verify site removal subscription action is propagated and properly interpreted by client

  • Delete existing Non-Master Operator

    • Verify Non-Master Operator is no longer able to log in

  • Delete existing Master Operator

    • Verify Master Operator is no longer able to log in

  • Create/provision a Non-Master Operator

    • Assign Computer Management and Site Permission

    • Local versus LDAP (if applicable)

    • Validate by logging in

  • Create/provision a Master Operator

    • Local versus LDAP (if applicable)

    • Validate by logging in

  • Validate Client functionality

    • Verify CPU usage of Agent is at expected levels (~2% at idle by default)

    • Check Memory Usage of Agent is within expected bounds (this can vary between versions and between platforms, but can be compared against the memory usage of existing production Agents).

  • Test new installers source

    • Use Agent install source within BES installers to install Agent

      • Verify that the Agent reports in and receives new/open actions

  1. Test basic functionality of Web Reports

  • Log on to Web Reports as a ‘normal’ user

    • Verify proper application of existing permissions associated to user/role (ex: verify correct listing of computers associated to role)

    • Run some stored reports, and validate that the data returned is expected

    • Check that saved filters work as expected

  • Log on to Web Reports as an Administrator

    • Validate general Web Reports configuration

 

Upgrade (Core Components)

Pre-Upgrade:

  • Database backup – verify restoration to ensure backups are valid/useable

  • Reduce Reporting Rate – increase heartbeat (to 120+ minutes), minimum report interval (to 21600)

  • Check database consistency:

    • MSSQL: dbcc checkdb

  • Back up IEM Server application files

    • On Windows:

      • ..\BigFix Enteprise\BES Server

    • On RHEL: 

      • /var/opt/BESServer

      • /var/opt/BESWebReportsServer

  • DSA: increase replication intervals

  • Change the following IEM Client settings on all clients:

    • _BESClient_Report_MinimumInterval = 3600 *This setting will reduce the amount of incoming data from the endpoints to allow the system to recover more quickly and reduce potential downtime.

    • _BESClient_RelaySelect_ResistFailureIntervalSeconds = 21600 *This value represents the amount of time BES Clients will wait after its relay appears down before performing BES Relay selection. This can prevent unnecessary automatic relay selection during the migration.

  • Change the heartbeat in the IEM Console to 6 hours: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Endpoint%20Manager/page/Console%20Preferences * This is another way to reduce the amount of incoming data from the endpoints.

 

Day of, Hours Before Change:

Upgrade:

  • Send upgrade notification/communication to appropriate end users

  • Close any open IEM Consoles 

  • Shut down Services

    • Any DSA replicas

    • Any dedicated Web Reports instances

    • TEMA: SUA/SCA

  • Upgrade Root Server

  • Upgrade any dedicated Web Reports instances

  • Re-enable Services for the TEMA servers

  • Upgrade IEM Consoles

  • Open one Console for login

  • Execute tests

 

Post-Upgrade:

  • Reset Client reporting frequencies to pre-upgrade value 

    • Heartbeat

    • _BESClient_Report_MinimumInterval

    • _BESClient_RelaySelect_ResistFailureIntervalSeconds

  • Reset Console Minimum Refresh Interval to pre-upgrade value

  • Decrease WebReports Cache Interval to pre-upgrade value

  • Upgrade Relay infrastructure “top-down”

  • Upgrade Endpoints