Upgrading Orchestrator 4.0.0+ to 4.1.0 in high availability

This procedure is for upgrading Orchestrator 4.0.0+ to 4.1.0 in a high availability environment.

Important: IBM Aspera supports direct upgrades to the current General Availability (GA) version from only two GA versions prior to the current release. To upgrade to the latest version, you must be within two GA versions of the current version. Upgrading from older version requires upgrading in steps. For example, if you are four GA versions behind, upgrade to two GA versions behind (GA - 2), and then upgrade to the current GA version.

Warning: Before performing any upgrade, review the following:

Perform a full environment back up and ensure the back up is successful. In case the upgrade fails, the only reliable, short-term fix is to roll back the environment using the back up.
Test the upgrade in a test environment comparable to the production environment.
If upgrading the test environment is successful, upgrade the production environment, but do not bring the production environment back online.
Before bringing the production environment back online, the customer must test the application to determine if an immediate rollback is needed. Otherwise, customers risk losing all data generated between upgrade and rollback.

Before upgrading Orchestrator, review product release notes. Aspera strives to maintain compatibility from one release to the next, but sometimes we must make changes that affect a small number of customers. Review the release notes for all Orchestrator versions that have been released since your current version. In particular, the Breaking Changes section highlights changes that may require you to adjust your workflow, configuration, or usage. These issues may impact your new installation.

I. preliminary steps: backing up Orchestrator data and stopping the services

Before upgrading the Orchestrator server on each node of the cluster, you must back up all Orchestrator data so that you can restore them after upgrade, if needed.

See Orchestrator directory locations for more information about the directory locations referred to in this procedure.

Before you begin the upgrade, follow these backup steps for both nodes in the cluster.

Note: All commands in the following procedure must be performed as root, or an equivalent superuser.

Create a backup of MySQL data with a MySQL dump.
```
$ /opt/aspera/common/mysql/bin/mysqldump -h localhost –u root –p orchestrator > /tmp/orchestrator_db_backup_timestamp.sql
```
Important: Confirm that the dump.mysql file is created (and that it is > 0 in size) and save the file for data recovery, as needed.

Note: The generated .sql file may take several gigabytes of disk space depending on the database size, so make sure that enough space is available on the targeted partition (/tmp in the example above) before running the backup.
Run a backup of Orchestrator data.

Note: Move all backup files onto a safe server after running the backup so they don’t take up space on Orchestrator.
Back up any important configuration files which are located in the config directory, since this directory is overwritten with new data during an upgrade.
On each node, copy the following files to a safe location, such as /tmp:
```
/opt/aspera/orchestrator/config/orchestrator.yml
/opt/aspera/orchestrator/config/database.yml
```
Copy any custom files — such as aspera_logo_simple.jpg and aspera.css — to a safe location.
During the software installation phase of an upgrade, custom images, branding files and style sheets are copied from /opt/aspera/var/config/orchestrator/ to /opt/aspera/orchestrator/public/. Thus make sure that any custom files (as well as orchestrator.yml, if the default Orchestrator configuration has been changed) are placed under /opt/aspera/var/config/orchestrator/ before doing the upgrade.
```
/opt/aspera/orchestrator/public/images/aspera_logo_simple.jpg
/opt/aspera/orchestrator/public/stylesheets/aspera.css
```
Back up the var directory.
Although this directory is preserved during upgrade, it is important to take the precaution.
```
$ cd /opt/aspera/ ; tar chvfz /tmp/aspera_orig_timestamp.tar.gz var/
```
Copy or archive the shared storage that all the symlinks are pointing to.
For example, you may might create the symlink with this command:
```
ln -s /mnt/shared/orchestrator/mysql_data/ ./data
```
In that case, you would need to back up this directory:
```
/mnt/shared/orchestrator/
```
Important: You may need this directory location as a reference for restoring your configuration.

II. Upgrading the Orchestrator servers

On the machine where you will install Orchestrator, download the installation files for the new release.
Go to IBM Fix Central, https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Other%20software&product=ibm/Other+software/IBM+Aspera+Orchestrator&release=All&platform=All&function=all.
Click the link for the current release of Orchestrator, then download the installers for aspera_orchestrator and ibm_aspera_common.

On both nodes, upgrade the Orchestrator.

# rpm -i aspera-orchestrator-4.1.version-0.x86_64.rpm

Run the Orchestrator setup.
```
# orchestrator setup
```
Restart your browser (recommended) or empty the browser cache.
Confirm that the new installation of Orchestrator works on both servers by logging into the Orchestrator UI.

Reload the MySQL database using the database dump you created in "Preliminary steps: Backing up Orchestrator data".

/opt/aspera/common/mysql/bin/mysql -h 127.0.0.1 -P 4406 -u <user> -p orchestrator < /tmp/orchestrator_db_backup_timestamp.sql

Troubleshooting tip: You may see an error similar to this:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

Re-run the command with socket=, for example:

# /opt/aspera/common/mysql/bin/mysql -h localhost -u root --socket=/opt/aspera/common/mysql/var/run/mysql.sock -p orchestrator < /tmp/dump1.sql

Consult these articles for any additional required steps:
- Configuring servers for ha
- Installing Orchestrator in high availability environments.
- Configuring MySQL and Aspera services (See the articles in the section.)

Iii. post upgrade: restoring files and re-enabling services

After upgrade, if found, copy aspera.css from the backup location to the following directory on both instances:
```
lib/branding/aspera.css.erb
```
Ensure that the contents of the following files are identical.
```
/opt/aspera/orchestrator/config/database.yml  
/opt/aspera/var/config/orchestrator/database.yml
```
If the files are not identical, use the values in the following file:
```
/opt/aspera/var/config/orchestrator/database.yml
```

Iv. upgrading plugins and portlets after upgrade of Orchestrator

Important: Plugins are now upgraded automatically during the rpm upgrade. You should still review to see if there are available upgrades.

Go to Engine > Plugins and click Upgrade Available to see the newer plugins.
Each plugin contains a Revision History screen providing details about the newer plugin. After reviewing the revision history, upgrade the plugin and force maintenance. For more details on plugin installation and upgrading, see
Find out if any new plugins are not already enabled by selecting Not Enabled, and enable them if necessary.

Note: In a production system, upgrade the new plugins individually only when necessary (compare the previous version with the current released version). It is advisable to test them in pre-production first to make sure that the new plugins operate well with the existing workflows.
Click Engine > Plugins > Reload Plugins, and then click Engine > Processes > Force Maintenance to set the new code.
If the new portlets are not enabled, and they are needed, click Engine > Portlets > Enable all.
Test your workflows to make sure they function properly after the upgrade.

V. importing plugins and portlets after an upgrade of Orchestrator

When importing plugins or portlets in a high availability setup (at any time after the upgrade), do one of the following with the plugin files and portlet files:

Recommended: Import the files on both Orchestrator servers. This loads the files on both servers and ensures that the UI shows the correct version.
Import the files on one server and reload the same files on the other.

On Engine > Plugins, click Enable All.
On Engine > Processes click Force maintenance.
On Engine > Plugins, click Reload All.
On Engine > Plugins, click Disabled Unused.
This step ensures improved performance.