Backing up and restoring the Developer Portal in a VMware environment

You can back up and restore your Developer Portal service in your VMware environment.

About this task

Note: You must backup both the Management and Portal subsystems at the same time, to ensure synchronicity across the services.

To complete a backup and restore of your Developer Portal, you must have a file portal-extra-values.yaml that is configured with the portalBackup subsection.

Backups can be scheduled or generated manually (on-demand). You can then list what backups are available and then use one to restore your Developer Portal if required.

The default Developer Portal backup schedule is once every 24 hours, but the schedule can be changed in the backup settings. The Developer Portal saves all system and site backups locally, and also saves them remotely based on the configured SFTP and s3 settings.

The local backups are automatically maintained so that the latest three backups of each site and of the system are kept, and older backups are removed. This maintenance means that the Developer Portal retains the latest three backups for each site and for the system however old they are, but there is no deletion of the old backups on the remote server. If a site is deleted, then all of the local backups for that site are also deleted, as otherwise the backup volume might become full of old site backups. For remote backups, you can configure a retention policy on your remote server to remove the old backup files as required.

Procedure

Configure your Developer Portal subsystem backup.
1. Create a file portal-extra-values.yaml with the following content:
```
spec:
  portalBackup:
    credentials: portal-backup-secret
    host: s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard
    path: test-bucket/restore-test
    port: 443
    protocol: objstore
    schedule: '*/5 * * * *'
```
  - All of your settings must be added in the spec: section of the file.
  - host: is the S3 endpoint with the corresponding S3 region in the format <S3endpoint>/<S3region>.
  - path: is the name of your S3 bucket to store backup data.
  - port: is the port for the protocol to connect to the host.
  - protocol: is the protocol that is used to communicate with your remote backup endpoint. Specify one of the following values:
    - sftp - for secure file transfer protocol.
    - objstore - for S3 compatible object storage.
  - schedule - is the schedule for how often automatic Developer Portal backups are run. The format for the schedule is any valid cron string.
    The timezone for backups is UTC.
  - credentials - is the name of the secret you created.
  Note: If edits to the portalBackup section are needed after installation, you can modify the portal-extra-values.yaml file and apply the changes to the Appliance cluster by using the command:
```
apicup subsys install portal_subsystem
```
2. Run the following command to add the extra values file to the subsystem:
```
apicup subsys set portal_subsystem extra-values-file path_to_portal-extra-values.yaml_file
```
3. Complete any remaining installation steps for your Developer Portal subsystem, and create an ISO file. For more information, see Deploying the Developer Portal.
4. Create your Developer Portal subsystem backup secret.
  The backup secret is a Kubernetes secret that contains your username and password for your backup database (sftp/s3). Only password-based authentication is supported for sftp and s3, not authentication based on public certificates and private keys. Password-based authentication for s3 requires that you generate an access key and secret. For example:
  - IBM (Cloud Object Storage): Service credentials.
  - AWS: Managing access keys.
  1. Enter the following command to connect to the Developer Portal subsystem by using SSH:
```
ssh ip_address -l apicadm
```
    You are logging in with the default ID of apicadm, which is the API Connect ID that has administrator privileges.
  2. Select Yes to continue connecting, and when connected:
```
sudo -i
```
  3. The secret can be created with the following command:
```
$ kubectl create secret generic portal-backup-secret --from-literal=username='<YOUR ACCESS KEY>' --from-literal=password='<YOUR ACCESS KEY SECRET>'
```

How to configure a scheduled backup.

Edit the schedule option in your portal-extra-values.yaml file to be the time that you want the backup to be taken. The format for the schedule option is any valid cron string.

spec:
  portalBackup:
    credentials: portal-backup-secret
    host: s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard
    path: test-bucket/restore-test
    port: 443
    protocol: objstore
    backupCerts: <custom-s3-server-CA-cert>
    schedule: '*/5 * * * *'

How to generate a manual backup.

You can use these steps to configure a Developer Portal with backups that use IBM's Cloud Object Storage S3 or AWS S3.

Decide which of the three types of manual backups you want to do:

all: backs up the Developer Portal system and all its sites to the remote server
system: backs up the Developer Portal system to the remote server
site: backs up a Developer Portal site to the remote server

Connect to the Developer Portal subsystem by using SSH:
```
ssh ip_address -l apicadm
```
Get necessary credentials:
```
sudo -i
```
You generate a backup manually by creating a portalBackup custom resource that is detected by the ibm-apiconnect operator that then triggers the backup process.
You can use portalbackup_cr.yaml in the helper_files directory as an example:
```
apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalBackup
metadata:
  generateName: portal-bup-
spec:
  type: all
  crType: create
  comment: "test comment"
  
```
- type: is the type of backup to be done. Valid arguments are:
  - all
  - system
  - site
- crType: Use create to trigger a backup process. When the backup is created, the operator generates another backup CR with crType: record, which represents the stored backup that you can use for a restore operation.
- comment: you can optionally add a comment to the CR before you create a backup.
- siteName: used only when you take a site backup. Valid arguments are:
  - site uuid or url - back up that site only to the remote server
  - installed - backups all installed sites to the remote server
Create the PortalBackup custom resource in the namespace of your Developer Portal to trigger the backup process:
```
kubectl create -f portalbackup_cr.yaml 
```

Optional: You can list current backups by using the command:

kubectl get portalbackup

Here is an example of some possible output:


NAME               ID            STATUS     TYPE     CR TYPE   AGE     COMMENT
p1-bup-gqz8f    20200526.142449  Complete   system   record    55m     
p1-bup-jx4vc    20200526.142458  Complete   site     record    54m     
p1-bup-st4l2                     Complete   all      create    56m     y
p1-bup-th2dm                     Complete   all      create    4h51m   y

Tip: If you want to order your portalbackups by ID, you can use the Kubernetes keyword --sort-by. For example:

kubectl get pb --sort-by=.status.backupId

Optional: You can get a detailed view of a specific backup by using the command:

kubectl get portalbackup <bup-name> -o yaml

Here is an example of some possible output:

apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalBackup
metadata:
  creationTimestamp: "2020-05-26T14:24:37Z"
  generateName: portal-bup-
  generation: 1
  name: portal-bup-st4l2
  namespace: default
  resourceVersion: "1509165"
  selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalbackups/portal-bup-st4l2
  uid: 36094280-79f6-47db-915a-1d39c8922d80
spec:
  comment: testing comment
  crType: create
  type: all
status:
  backupId: ""
  commentLeft: "y"
  conditions:
  - lastTransitionTime: "2020-05-26T14:26:44Z"
    message: |-
      2020-05-26 14:24:47: CLI task (backup:all) starting.
      2020-05-26 14:24:48: Creating a portal system backup...
      2020-05-26 14:24:49: Portal system backup created: _portal_system_backup-20200526.142449.tar.gz
      2020-05-26 14:24:49: Uploading _portal_system_backup-20200526.142449.tar.gz to s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard:22:cq-test/restore-test/ using objstore
      2020-05-26 14:24:50: Portal system backup (_portal_system_backup-20200526.142449.tar.gz) successfully uploaded
      2020-05-26 14:24:55: Making a backup for site test.com
      2020-05-26 14:26:10: A local backup was succesfully created: /var/aegir/backups/test.com-20200526.142458.tar.gz
      2020-05-26 14:26:10: Attempting to upload backup to remote backup server
      2020-05-26 14:26:10: Uploading test.com-20200526.142458.tar.gz to s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard:22:cq-test/restore-test/ using objstore
      2020-05-26 14:26:15: Site with URL test.com (test.com-20200526.142458.tar.gz) successfully uploaded
      2020-05-26 14:26:15: All installed sites have been successfully backed up
      2020-05-26 14:26:15: CLI task (backup:all) completed successfully.
    status: Complete
    type: Ready
  fullBackupName: ""
  id: 0m729hg7blf9z2b8
  message: ""

How to list available Developer Portal backups on remote server.

Connect to the portal subsystem by using SSH:
```
ssh ip_address -l apicadm
```
Get necessary credentials:
```
sudo -i
```
Run the command:

kubectl get portalbackups

Here is an example of some possible output:

kubectl get portalbackup 
  NAME               ID            STATUS     TYPE     CR TYPE   AGE     COMMENT
  p1-bup-gqz8f    20200526.142449  Complete   system   record    55m     
  p1-bup-jx4vc    20200526.142458  Complete   site     record    54m     
  p1-bup-st4l2                     Complete   all      create    56m     y
  p1-bup-th2dm                     Complete   all      create    4h51m   y

Note: You can only restore the Portal subsystem from backups where the CR TYPE is record.

Optionally, if you need more information, such as the name of the .tgz file that you can run, use command:

kubectl get portalbackups <bup-name> -o yaml

Here is an example of some possible output:

apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalBackup
metadata:
 creationTimestamp: "2020-05-26T18:21:24Z"
 generateName: portal-bup-
 generation: 1
 name: portal-bup-jx4vc
 namespace: default
 resourceVersion: "1558209"
 selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalbackups/portal-bup-2rlwk
 uid: c37bc5ab-6a78-4ef0-8a85-e00aa9eee5a6
spec:
 comment: ""
 crType: record
 siteName: ""
 type: system
status:
 backupId: "20200526.181730"
 commentLeft: ""
 conditions:
 - lastTransitionTime: "2020-05-26T18:21:24Z"
   status: Complete
   type: Ready
 fullBackupName: _portal_system_backup-20200526.181730.tar.gz
 id: ""
 message: ""

How to restore a Developer Portal subsystem.

Ensure that the backup method that you used is complete before you attempt a restore. The backup data that is stored in the remote server is used for restoring the Developer Portal subsystem data back to a previous state. Ensure that these backups are on your remote server before you attempt a restore.

There are three types of restores:

all: restores the Developer Portal system and all the sites that are found on the remote server by using the latest backup.
system: restores the Developer Portal system to the remote server
site: restores the Developer Portal site from the remote server

Connect to the portal subsystem by using SSH:
```
ssh ip_address -l apicadm
```
Get necessary credentials:
```
sudo -i
```

Decide on the options to use for your PortalRestore custom resource. The PortalRestore custom resource is detected by the ibm-apiconnectoperator that then triggers the backup process.

You can use portalrestore_cr.yaml in the helper_files directory as an example:

apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalRestore
metadata:
  generateName: portal-restore-
spec:
  type: all
  dryRun: true
  timestamp: now

type: is the type of restore to be done. Valid arguments are:
- all
- system
- site
siteName: used only when you took a site backup. Valid arguments are:
- ID, for example 20200526.142458: the backup ID of a portal backup CR. The ID must be taken from a portal backup that is of type site, and crType record.
- site-1234.tgz: restore the site by using this file name that exists on the local file system or remote server.
- myportal.com: find the latest backup for this URL (locally or remotely) and restore it.
- all: restore all sites that are found on the remote server by using the latest backup.
- site uuid or url - back up that site only to the remote server
- installed - backups all installed sites to the remote server.
systemName - Valid arguments are:
- ID, for example 20200526.142449: the backup ID of a portal backup CR. The ID must be taken from a portal backup that is of type system, and crType record.
- system_backup-1234.tgz: restores the system from the given file that exists on the pod's local file system or the remote server.
- latest: restores from latest system backup that is found from either local file system or remote server.
timestamp - Valid arguments are:
- now: use the latest backups available.
- < TIMESTAMP >: in 'YYYY-MM-DD HH:MM:SS' format, specify a timestamp to retrieve the backup from. The nearest backup, searching backwards from this timestamp, is used.
Note: From IBM® API Connect Version 10.0.1.4-ifix1-eus, the timestamp format changed to YYYYMMDD.HHMMSS. For Version 10.0.1.2-eus and earlier, the timestamp format is YYYY-MM-DD HH:MM:SS.
dryRun - Valid arguments are:
- true: runs a dry run of the command, returning the actions that will be taken. These actions can be found in the custom resource by following step 4.
- false: runs the command, restoring and replacing the system files and all sites. Any existing sites are reinstalled.

Table 1. The options that are needed for each Developer Portal restore
	`all`	`system`	`site`
`type`	X	X	X
`siteName`			X
`systemName`		X
`timestamp`	X
`dryRun`	X

Create the PortalRestore custom resource in the namespace of the Developer Portal subsystem to trigger the restore process:
```
kubectl create -f portalrestore_cr.yaml
```
Optional: You can list current restores by using the command:
```
kubectl get portalrestore
```
Here is an example of some possible output:
```
NAME                   READY     AGE
portal-restore-twjrp   Running   28s
```

Optional: You can get a detailed view of a specific restore by using the command:

kubectl get portalrestore <restore-name> -o yaml

Here is an example of some possible output:

apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalRestore
metadata:
  creationTimestamp: "2020-05-26T13:11:17Z"
  generateName: portal-restore-
  generation: 1
  name: portal-restore-8f4c5
  namespace: default
  resourceVersion: "1497597"
  selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalrestores/portal-restore-8f4c5
  uid: 5a70b657-d998-4989-b9b6-8480e986311d
spec:
  siteName: "20200526.090209"
  type: site
status:
  backupId: ""
  commentLeft: ""
  conditions:
  - lastTransitionTime: "2020-05-26T13:30:53Z"
    message: |-
      2020-05-26 13:11:25: CLI task (restore:site) starting.
      2020-05-26 13:11:36: Deleting any old site data if it exists...
      2020-05-26 13:13:18: Restoring site test.com ...
      2020-05-26 13:13:18: .
      2020-05-26 13:13:19: Patching the database dump before restoring ...
      2020-05-26 13:13:48: .
      2020-05-26 13:19:41: Triggering the drush command to restore the site. This may take some time.
      2020-05-26 13:19:48: .
      2020-05-26 13:25:40: Drush restore command completed. Setting up the site.
      2020-05-26 13:25:48: .
      2020-05-26 13:30:49: Site test.com restored from backup /var/aegir/backups/test.com-20200526.090209.tar.gz. Initializing the site...
      2020-05-26 13:30:52: Restore site completed.
      2020-05-26 13:30:52: CLI task (restore:site) completed successfully.
    status: Complete
    type: Ready
  fullBackupName: ""
  id: bybyibd99wxtmkmk
  message: ""

Troubleshooting

Connect to the portal subsystem by using SSH:
```
ssh ip_address -l apicadm
```
Get necessary credentials:
```
sudo -i
```

Check the Developer Portal backup CR or restore CR.

Use command:

kubectl get <portalbackup/portalrestore> <bup-name/restore-name> -o yaml

Here is an example of some possible output:

apiVersion: portal.apiconnect.ibm.com/v1beta1
kind: PortalRestore
metadata:
  creationTimestamp: "2020-05-26T13:10:46Z"
  generateName: portal-restore-
  generation: 1
  name: portal-restore-x2bf9
  namespace: default
  resourceVersion: "1493471"
  selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalrestores/portal-restore-x2bf9
  uid: 9f6a1e22-55ab-4595-8b98-bdfaec40d58c
spec:
  type: site
status:
  backupId: ""
  commentLeft: ""
  conditions:
  - lastTransitionTime: "2020-05-26T13:10:49Z"
    message: |
      Please provide a site to restore. To restore all sites use value 'all' in the CR
    status: Failed
    type: Ready
  fullBackupName: ""
  id: ""
  message: ""

Read the message inside the conditions of the Developer Portal backup CR or the restore CR.

Retrieving logs
Look for information in the logs of the ibm-apiconnect pod:
- Locate the ibm-apiconnect pod with:
```
kubectl get pods
```
- View the logs with:
```
kubectl logs <ibm-apiconnect-pod-NAME>
```