Upgrading the software in a File system replication environment

Any time you upgrade a primary system, you need to manually set the secondary system in the read/write mode temporarily in order to upgrade it as well. In DR environment, you need to upgrade both primary and secondary systems before resuming application again.

Before you begin

The following operations require logging in as apuser, or as a user with equivalent privileges.

Procedure

Stop the database on primary and secondary by running:
```
apstop -a
```
Create a latest snapshot from the primary.
Ensure that the afm queues are empty.
Execute apdr failover to secondary system with the command:
```
apdr failover
```
Change the role of secondary system by executing command:
```
apdr changeRole
```
Check the replication status from the new primary and ensure that all the file sets are in ready state.
```
apdr status --replqueue
```
Suspend the disaster recovery replication between the primary and the secondary system by logging in to the primary IAS and using:
```
apdr suspend
```
Output example as follows:
```
./apdr suspend
Getting APDR  status
Successfully suspended Filesystem Replication between Primary and Secondary
```
To start the database log in to both systems as apuser and run
```
apstart
```
After that you can perform all maintenance operations such as apupgrade.
Note: You can use your primary system for any database operations, but don't do any workload operations on the database that is started on secondary. When the database goes back to the secondary role, any operation done during the time DR was suspended are erased. Make sure your secondary system is upgraded to the same software level as the primary. If you try to resume replication while primary and secondary systems' versions are different, you get an error.
Important: After upgrading to IAS 1.0.22.0, the apafmdr service is stopped. As a result, you're not able to run the apdrupgrade command. A workaround for this is to run the following commands before running apdrupgrade:
```
apdr certificate --create
apdr disable
apdr enable
```
Once the upgrade procedure has been completed successfully, run the apdrupgrade script on both the primary and secondary system to upgrade DR specific changes.
```
apdrupgrade
```
After a successful upgrade on each of the systems re-enable DR by running the following commands:
```
apdr disable
apdr enable
```
When all maintenance operations on the secondary system are completed, stop the database on both systems by running:
```
apstop
```
If you're upgrading to 1.0.22.0 or higher from a version prior to 1.0.22.0, follow these steps regarding certificate authentication on one system at a time:
1. On the first system, as the apuser create a temporary directory where you will be temporarily exporting the certificate:
```
mkdir <temp directory>
```
2. Export the client certificate to the local directory you created:
```
apdr certificate --export <temporary directory path you created>
```
3. Copy this certificate to the second system in a temporary directory using any method. For example:
```
scp <path/client.pem> apuser@<host>:~/tmp/
```
4. On the second system, import the certificate you copied over:
```
apdr certificate --import <path>/client.pem
```
5. Delete the temporary copy of the certificate from your temporary directories on both systems after importing it:
```
rm <temporary directory path>/client.pem
```
6. On the second system, verify the certificate was imported correctly:
```
apdr certificate --validate
```
  In the output, look for Certificates validated on local system.
7. Restart the DR service on the second system to start it with certificate authentication:
```
apdr disable
apdr enable
```
8. Repeat substeps a-g for the second system, so that both systems can communicate in each direction.
Create a snapshot of the primary by running:
```
apdr sync -r --fsonly
```
You can move back to the old primary system by running the following command. For more information on this, see Failback to the old primary system.
```
apdr failback
```
You can now start the database on old primary by running:
```
apstart
```

If running on appliance version 1.0.19.5 or 1.0.19.6, before resuming disaster recovery, delete snapshots on both primary and secondary systems. Use the following command on BOTH primary and secondary system respectively:

apdbrollback snapshot --delete

Example output:

[root@node0101 ~]# apdbrollback snapshot --delete
Please enter yes/no if you want to proceed with deleting all existing snapshots yes
Successfully deleted snapshot for scratch on node0101-fab
Successfully deleted snapshot for local on node0101-fab
Successfully deleted snapshot for head on node0101-fab
Successfully deleted snapshot for data on node0101-fab
Successfully deleted snapshots for all filesystems

If upgrading to version 1.0.20.0 release or higher, rerun the following command on the primary system:
```
apdr setup
```
This is done in order to setup any new filesets added as a part of the upgrade.
To monitor the progress of this operation, use the following command on the primary system:
```
apdr setup -s -d
```
After the completion of the setup procedure, resume disaster recovery between the primary and the secondary system by logging in to the primary and using the following command:
```
apdr resume
```
Example output:
```
./apdr resume
Getting APDR  status
Successfully resumed Filesystem Replication between Primary and Secondary
```
You can now start the database on primary by running:
```
apstart
```
Note: Don’t start the database on the secondary system.
If you are upgrading from version 1.0.24.0 or below to version 1.0.25.0 or above, take a manual snapshot on the primary system after it is successfully brought up:
```
apdr sync -r
```

Results

After the resume operation the secondary system is back in the passive mode and starts to synchronize the operations from the primary system.