Recovering the Developer Portal after a disaster

You can restore the Developer Portal subsystem after a disaster event.

Before you begin

Important: Successful disaster recovery depends on recovery of both the Management subsystem and the Developer Portal subsystem. You must complete preparation steps for both subsystems in order to achieve disaster recovery. If you have to perform a restore, you must complete the restoration of the Management Service first, and then immediately restore the Developer Portal. Therefore, the backups of the Management and Portal must be taken at the same time, to ensure that the Portal sites are consistent with Management database.

About this task

To recover from a disaster event, you must create a new IBM API Connect installation with a running IBM API Connect Operator.

Note: Limitation for backups created on Version 10.0.2:

If restoring a Version 10.0.2 backup onto a new Version 10.0.2 deployment, performing a restore may not work if the subsystem CR name exceeds 15 characters. This limitation applies only to restoring onto Version 10.0.2.
Restoration is supported for Version 10.0.2 backups onto a Version 10.0.3.0 or later deployment when subsystem name exceeds 15 characters, as long as the correct spec.originalUID is specified upon restore. See Step 1.f.

Procedure

To prepare for the restoration, you must complete some steps before you reinstall the Developer Portal subsystem.
Warning:
- Ensure that the hostnames of the endpoints remain the same in the Developer Portal subsystem that is used for installation now as they were for the original installation. The hostnames for the endpoints cannot be changed.
1. Ensure that the Developer Portal backup subsystem secret is created in the cluster.
  
  This secret contains the credentials (username and password) for the remote storage, such as S3 or IBM Cloud Object Storage, that is used for storing the database backups. This secret must be the same secret as used for the original installation.
2. Ensure that the Developer Portal subsystem is configured for database backups.
  Configuration is done by adding a databaseBackups subsection to the spec subsection of the Developer Portal subsystem Custom Resource (CR) YAML file, portal_cr.yaml. The file is located in the helper_files archive that was extracted as part of the preparation for installation. See Backing up and restoring the Developer Portal in a Kubernetes environment.
  
  For example:
```
portalBackup:
      credentials: portal-backup-secret
      host: <backup-host>
      path: <backup-path>
      port: 22
      protocol: sftp
      schedule: ''
```
  The host and path must be the same as was specified for the original installation so that the IBM API Connect Operator can access the saved Developer Portal database backups.
3. Apply the saved YAML file that contains the Developer Portal database encryption secret into the cluster.
  Use the following command, where portal_enc_key.yaml is the local YAML file that contains the backup encryption secret. Replace <namespace> with the namespace that is being used for subsystem installation:
```
kubectl apply -f portal_enc_key.yaml -n <namespace>
```
  This command re-creates the original Developer Portal database encryption secret on the cluster, which is named as the original name of the secret.
4. Add the encryptionSecret to the Developer Portal subsystem Custom Resource (CR) YAML file, portal_cr.yaml.
  For example, add the following encryptionSecret subsection to the spec section. In this example, portal-enc-key is the name of the new created secret on the cluster that contains the original Developer Portal Database Encryption Secret. The name of the secret that is created in Step 1.a is used here:
```
encryptionSecret:
  secretName: portal-enc-key
```
5. Ensure the name spec.metadata.name of the new Developer Portal cluster CR is the same as the previous Developer Portal CR.
6. Version 10.0.3.0 or later: Add the originalUID: property to the spec of the Portal CR.
  When recreating a system, to restore a backup into it, you must specify the same spec.originalUID in the CR as was present in the system that was backed up. If the spec.originalUID in the new CR for recreating the system does not match the spec.originalUID that was present in the system that was backed up, the restore will fail.
```
spec:
  originalUID: "447ea4b3-9514-4a84-a34b-ce0b349838da"
```
  Note:
  - For Version 10.0.3.0 or later, if you do not specify spec.originalUID in the new CR, the operator automatically sets the Portal CR value of spec.originalUID to match the new CR value metadata.uid. In this case, the restore will fail because the spec.originalUID in the saved (backed-up) CR does not match spec.originalUID in the new CR.
  - The originalUID is only essential when the subsystem CR name exceeds 15 characters in length, or 10 characters limit for the API Connect Cluster CR. Recommended practice is that all backups should include the originalUID for Portal.
  - See also Preparing the Developer Portal for disaster recovery.
7. Ensure that the endpoint definitions are the same as the previous Developer Portal CR.
Apply the Developer Portal subsystem custom resource (CR) YAML file (portal_cr.yaml) to the cluster as documented as part of the original installation instructions.

You can do this step now because the secrets are re-created, and additions made to portal_cr.yaml.

Note: Wait for the Developer Portal subsystem to become fully up and running before proceeding.

Use the following command to check that the Developer Portal database backups that are stored in the configured cloud storage are now present as Developer Portal backup CRs:

Note: You might wait for the cron job at the 10th minute */10 * * * * to re-create your Developer Portal backup CRs.

kubectl get portalbackups -n <namespace>

For example:

loginabc@localhost ibm-apiconnect]$ kubectl get pb
NAME               ID                STATUS   TYPE   CR TYPE   AGE     COMMENT
portal-bup-g66gv                     Ready    site   create    5m27s   y
portal-bup-z7wgk   20201001.100021   Ready    site   record    4m24s

The AGE of each is new.

Perform a Developer Portal restore. See Backing up and restoring the Developer Portal in a Kubernetes environment
For a disaster recovery scenario, it is advised that you restore all sites and systems. portalCluster is the name of the portal cluster you want to restore. An example restore CR:
```
  apiVersion: portal.apiconnect.ibm.com/v1beta1
  kind: PortalRestore
  metadata:
    generateName: portal-restore-
  spec:
    type: all
    dryRun: false
    portalCluster: portal
    timestamp: now
    priorityList:
      - myportal-critical-site.com
      - myportal-important.com
```
When the Developer Portal restore CR is marked as Ready, the Disaster recovery is complete for the Developer Portal.
```
kubectl get portalrestore
```
Verify that you can access the restored Developer Portal sites and that you are able to log in to each one.

The restore from the disaster event is now complete.