Backing up and restoring the Developer Portal in a Kubernetes environment

You can back up and restore your Developer Portal service in your Kubernetes environment.

• If you haven't already, configure your Developer Portal custom resource with the portalBackup subsection.
  1. Create your Developer Portal backup secret.
     The backup secret is a Kubernetes secret that contains your credentials for accessing the s3 or SFTP backup database.

     • s3

       Only password-based authentication is supported for s3, not authentication based on public certificates and private keys. Password-based authentication for s3 requires that you generate an access key and secret. For example:

       • IBM (Cloud Object Storage): Service credentials.
       • AWS: Managing access keys.
       The secret can be created with the following command:

       kubectl create secret generic portal-backup-secret --from-literal=username='<YOUR ACCESS KEY OR USER NAME>' 
           --from-literal=password='<YOUR ACCESS KEY SECRET OR PASSWORD>' -n <namespace-of-portal-subsystem>

     • SFTP

       Supported credentials types:

       • Username and password (v10.0.2.0 or later).
       • Username and SSH-key (v10.0.3.0 or later). Only OpenSSH keys are supported.¹

       Use one of the following commands to create the secret:

       • Username and password credentials

         $ kubectl create secret generic portal-backup-secret --from-literal=username='<YOUR USERNAME>' 
         --from-literal=password='<YOUR PASSWORD>' -n <namespace-of-portal-subsystem>

       • Version 10.0.3.0 or later: Username and SSH-key credentials:

         $ kubectl create secret generic portal-backup-secret --from-literal=username='<YOUR USERNAME>'
          --from-file=ssh-privatekey='<YOUR PRIVATEKEY FILE>' -n <namespace-of-portal-subsystem>

  2. Ensure that your Developer Portal custom resource is configured with the portalBackup subsection.
     • All of your settings must be added in the spec: section of the file.
     • To schedule automated backups, specify values for the schedule setting.
     An example when you use protocol: objstore.

     spec:
       portalBackup:
         credentials: portal-backup-secret
         host: s3.eu-gb.cloud-object-storage.appdomain.cloud/
         path: test-bucket/restore-test
         port: 443
         protocol: objstore
         backupCerts: <custom-s3-server-CA-cert>
         backups3URIStyle: <host-or-path>
         schedule: '0 2 * * *'

     An example when you use protocol: sftp.

     spec:
       portalBackup:
         credentials:  portal-backup-secret
         host:         sftp-service.example.com
         path:         /home/fvtuser/site-backups
         port:         22
         protocol:     sftp
         schedule:     '0 2 * * *'

     Notes about portalBackup: entry:

     • If portalBackup: entry is not present or contain nothing, just local backups will be taken if a backup CR is created. If the schedule exists, backups are taken at the scheduled time.
     • For S3 and SFTP, portalBackup: entry must contain host, credentials, protocol, and path.
     • For s3, the portalBackup: section can optionally also contain either, or both, of backupCerts and backupS3URIStyle.

     Table 1. Developer Portal backup settings

     Setting	Description
     credentials	The name of the secret you created. Supported credential types: s3 Only password-based authentication is supported for s3, not authentication based on public certificates and private keys. Password-based authentication for s3 requires that you generate an access key and secret. SFTP Username and password (v10.0.2.0 or later) Username and SSH-key (v10.0.3.0 or later) To create a secret, see step 1.
     host	s3 - host: is the S3 endpoint with the corresponding S3 region in the format <S3endpoint>, or with an optional S3 region in the format <S3endpoint>/<S3region>. sftp - the hostname of the sftp server
     path	S3 - the name of your S3 bucket to store backup data, optionally followed by a / and then any subdirectories inside the bucket. Valid examples: bucket bucket/path bucket/lots/of/paths sftp - the full absolute path of the folder on the SFTP server, beginning with /.
     port	The port for the protocol to connect to the host. S3 - Ignored for S3 (because objstore always uses port 443) SFTP - Defaults to 22 if not specified local - Ignored for local
     protocol	The protocol that is used to communicate with your remote backup endpoint. Specify one of the following values: sftp - for secure file transfer protocol. objstore - for S3 compatible object storage.
     backupCerts	(S3 storage only) backupCerts is the name of a custom certificate that contains your upstream custom S3 CA certificate. Supported beginning with 10.0.2. Note: If you use a public certificate for the S3 storage provider, it must be signed by a known certificate authority that is trusted by API Connect. Use of an untrusted authority can cause the following error during backup upload: x509: certificate signed by unknown authority This field accepts name of the Kubernetes secret containing your upstream custom S3 CA certificate. The key of the secret must be ca.crt and the value is the base64-encoded value of the CA certificate. If the server certificate that is presented by your S3/object-store endpoint is signed by a well known CA and includes any intermediate certificates in the chain, then you do not need to provide the CA certificate using this feature because the Portal subsystem already trusts the server certificate. You can test this by configuring the Portal backup to your S3/object-store server without providing the CA certificate. If you do need to provide the CA certificate then you must provide the entire chain in the ca.crt member of the secret, as follows: intermediate-certificate-1 intermediate-certificate-2 . . . intermediate-certificate-n root-certificate Where the first certificate in the ca.crt member (intermediate-certificate-1) is the issuer of the S3/object-store server certificate, and each subsequent certificate in the ca.crt member is the issuer of the preceding certificate. The root certificate in the ca.crt member must be self-signed. If there are no intermediate certificates involved, then the ca.crt member contains only the root certificate. The following sample bash script creates the Kubernetes secret: cat >customs3ca.yaml <<EOF apiVersion: v1 data: ca.crt: $(base64 <path-to-ca-certificate> | tr -d '\n' ) kind: Secret metadata: name: custom-server-ca type: generic EOF kubectl apply -f customs3ca.yaml -n <namespace>
     backups3URIStyle	(S3 storage only) Optional - backups3URIStyle indicates whether URIs to your S3 backup should use specify a host or a path value. When not specified, default to host. Supported beginning with 10.0.2. Valid values: host and path. Some custom S3 providers require URI style to be set to path. For example, MinIO supports both host and path style setup. You can create a MinIO S3 server to only accept path style client communications. Important: Contact your custom S3 administrator before configuring this field. If not properly configured, an upstream custom S3 can reject connections from the client, such as the API Connect Management subsystem.
     schedule	The schedule for how often automatic Developer Portal backups are run. The format for the schedule is any valid cron string. The timezone for backups is UTC. Valid for all backup types: s3, sftp, and local. spec: portalBackup: credentials: portal-backup-secret host: s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard path: test-bucket/restore-test port: 443 protocol: objstore schedule: '0 2 * * *'

• Reconfiguring Developer Portal backups
  You can edit the deployed Developer Portal custom resource You might want to do an edit if you did not include a portalBackup section before the installation. Or, if you made a mistake in your portalBackup, or if you'd like to change the portalBackup config.To edit your portalcluster CR:

  1. Get the name of your portalcluster CR:

     kubectl get portalcluster -n <namespace-of-portal-subsystem>
     

  2. Edit your portalcluster CR:

     kubectl edit portalcluster <name-of-portal-cr> -n <namespace-of-portal-subsystem>
     

  3. Inside the spec, add the portalBackup section as explained in step 2.
  4. Write and quit wq.
• How to generate a manual backup.

  You can use these steps to configure a Developer Portal with backups that use IBM's Cloud Object Storage S3, AWS S3, or SFTP.

  Decide which of the three types of manual backups you want to do:

  • all: backs up the Developer Portal system and all its sites to the remote server
  • system: backs up the Developer Portal system to the remote server
  • site: backs up a Developer Portal site to the remote server
  1. You generate a backup manually by creating a portalBackup custom resource that is detected by the ibm-apiconnect operator that then triggers the backup process.
     You can use portalbackup_cr.yaml in the helper_files directory as an example:

     apiVersion: portal.apiconnect.ibm.com/v1beta1
     kind: PortalBackup
     metadata:
       generateName: portal-bup-
     spec:
       type: all
       crType: create
       comment: "test comment"
       

     • type: is the type of backup to be done. Valid arguments are:
       • all
       • system
       • site
     • crType: Use create to trigger a backup process. When the backup is created, the operator generates another backup CR with crType: record, which represents the stored backup that you can use for a restore operation.
     • comment: you can optionally add a comment to the CR before you create a backup.
     • siteName: used only when you take a site backup. Valid arguments are:
       • site uuid or url - back up that site only to the remote server
       • installed - backups all installed sites to the remote server
  2. Create the PortalBackup custom resource in the namespace of your Developer Portal to trigger the backup process:

     kubectl create -f portalbackup_cr.yaml -n <ns-of-portal-subsystem>

  3. Optional: You can list current backups by using the command:

     kubectl get portalbackup -n <ns-of-portal-subsystem>

     Here is an example of some possible output:

     
     NAME               ID            STATUS     TYPE     CR TYPE   AGE     COMMENT
     p1-bup-gqz8f    20200526.142449  Ready      system   record    55m     
     p1-bup-jx4vc    20200526.142458  Ready      site     record    54m     
     p1-bup-st4l2                     Ready      all      create    56m     y
     p1-bup-th2dm                     Ready      all      create    4h51m   y

     Tip: If you want to order your portalbackups by ID, you can use the Kubernetes keyword --sort-by. For example:

     kubectl get pb -n <ns> --sort-by=.status.backupId

  4. Optional: You can get a detailed view of a specific backup by using the command:

     kubectl get portalbackup <bup-name> -n <ns-of-portal-subsystem> -o yaml

     Here is an example of some possible output:

     apiVersion: portal.apiconnect.ibm.com/v1beta1
     kind: PortalBackup
     metadata:
       creationTimestamp: "2020-10-01T10:00:16Z"
       generateName: portal-bup-
       generation: 1
       name: portal-bup-g66gv
       namespace: default
       resourceVersion: "8940"
       selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/default/portalbackups/portal-bup-g66gv
       uid: b1d5b295-b966-48f2-b6ef-dfcd1c499741
     spec:
       comment: test comment
       crType: create
       siteName: installed
       type: site
     status:
       backupId: ""
       commentLeft: "y"
       conditions:
       - lastTransitionTime: "2020-10-01T10:00:18Z"
         reason: BackupFailed
         status: "False"
         type: Error
       - lastTransitionTime: "2020-10-01T10:01:19Z"
         message: |-
           2020-10-01 10:00:18: CLI task (backup:site) starting.
           2020-10-01 10:00:20: Making a backup for site conor.com
           2020-10-01 10:00:27: A local backup was succesfully created: /var/aegir/backups/conor.com-20201001.100021.tar.gz
           2020-10-01 10:00:30: Attempting to upload backup to remote backup server
           2020-10-01 10:00:30: Uploading conor.com-20201001.100021.tar.gz to s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard:22:cq-test/ptlcq4/ using objstore
           2020-10-01 10:00:33: Site with URL conor.com (conor.com-20201001.100021.tar.gz) successfully uploaded
           2020-10-01 10:00:33: All installed sites have been successfully backed up
           2020-10-01 10:00:33: CLI task (backup:site) completed successfully.
         reason: BackupComplete
         status: "True"
         type: Ready
       - lastTransitionTime: "2020-10-01T10:01:19Z"
         reason: BackupInProgress
         status: "False"
         type: Running
       - lastTransitionTime: "2020-10-01T10:00:18Z"
         reason: BackupStatusUnknown
         status: "False"
         type: Warning
       fullBackupName: ""
       id: f8t0ux7z4zfnnjnu
       message: ""
       phase: Ready
       state: ""

• How to list available Developer Portal backups on remote server.

  Use command:

  kubectl get portalbackups -n <ns-of-portal-subsystem>

  Here is an example of some possible output:

  kubectl get portalbackup -n <ns-of-portal-subsystem>
    NAME               ID            STATUS     TYPE     CR TYPE   AGE     COMMENT
    p1-bup-gqz8f    20200526.142449  Ready      system   record    55m     
    p1-bup-jx4vc    20200526.142458  Ready      site     record    54m     
    p1-bup-st4l2                     Ready      all      create    56m     y
    p1-bup-th2dm                     Ready      all      create    4h51m   y

  Note: You can only restore the Portal subsystem from backups where the CR TYPE is record.

  Optionally, if you need more information, such as the name of the .tgz file that you can run, use command:

  kubectl get portalbackups <bup-name> -n <ns-of-portal-subsystem> -o yaml

  Here is an example of some possible output:

  apiVersion: portal.apiconnect.ibm.com/v1beta1
  kind: PortalBackup
  metadata:
    creationTimestamp: "2020-10-01T10:01:19Z"
    generateName: portal-bup-
    generation: 1
    name: portal-bup-z7wgk
    namespace: default
    resourceVersion: "8938"
    selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/default/portalbackups/portal-bup-z7wgk
    uid: 5c5899c0-4da0-4468-b7fb-1c283121a053
  spec:
    comment: ""
    crType: record
    siteName: ""
    type: site
  status:
    backupId: "20201001.100021"
    commentLeft: ""
    conditions:
    - lastTransitionTime: "2020-10-01T10:01:19Z"
      reason: BackupFailed
      status: "False"
      type: Error
    - lastTransitionTime: "2020-10-01T10:01:19Z"
      message: Record Backup CR for conor.com-20201001.100021.tar.gz
      reason: BackupComplete
      status: "True"
      type: Ready
    - lastTransitionTime: "2020-10-01T10:01:19Z"
      reason: BackupInProgress
      status: "False"
      type: Running
    - lastTransitionTime: "2020-10-01T10:01:19Z"
      reason: BackupStatusUnknown
      status: "False"
      type: Warning
    fullBackupName: conor.com-20201001.100021.tar.gz
    id: ""
    message: ""
    phase: Ready
    state: ""

• How to restore a Developer Portal subsystem.

  Ensure that the backup method that you used is complete before you attempt a restore. The backup data that is stored in the remote server is used for restoring the Developer Portal subsystem data back to a previous state. Ensure that these backups are on your remote server before you attempt a restore.

  There are three types of restores:

  • all: restores the Developer Portal system and all the sites that are found on the remote server by using the latest backup.
  • system: restores the Developer Portal system to the remote server
  • site: restores the Developer Portal site from the remote server
  1. Decide on the options to use for your PortalRestore custom resource. The PortalRestore custom resource is detected by the ibm-apiconnectoperator that then triggers the backup process.
     You can use portalrestore_cr.yaml in the helper_files directory as an example:

     apiVersion: portal.apiconnect.ibm.com/v1beta1
     kind: PortalRestore
     metadata:
       generateName: portal-restore-
     spec:
       type: all
       dryRun: true
       timestamp: now
       priorityList:
         - myportal-critical-site.com
         - myportal-important.com

     • type: is the type of restore to be done. Valid arguments are:
       • all
       • system
       • site
     • siteName: used only when you took a site backup. Valid arguments are:
       • ID, for example 20200526.142458: the backup ID of a portal backup CR. The ID must be taken from a portal backup that is of type site, and crType record.
       • site-1234.tgz: restore the site by using this file name that exists on the local file system or remote server.
       • myportal.com: find the latest backup for this URL (locally or remotely) and restore it.
       • all: restore all sites that are found on the remote server by using the latest backup.
       • site uuid or url - back up that site only to the remote server
       • installed - backups all installed sites to the remote server.
     • systemName - Valid arguments are:
       • ID, for example 20200526.142449: the backup ID of a portal backup CR. The ID must be taken from a portal backup that is of type system, and crType record.
       • system_backup-1234.tgz: restores the system from the given file that exists on the pod's local file system or the remote server.
       • latest: restores from latest system backup that is found from either local file system or remote server.
     • timestamp - Valid arguments are:
       • now: use the latest backups available.
       • < TIMESTAMP >: in 'YYYYMMDD.HHMMSS' format, specify a timestamp to retrieve the backup from. The nearest backup, searching backwards from this timestamp, is used.
       Note: From IBM® API Connect Version 10.0.3.0, the timestamp format changed to YYYYMMDD.HHMMSS. For Version 10.0.2.0 and earlier, the timestamp format is YYYY-MM-DD HH:MM:SS.
     • dryRun - Valid arguments are:
       • true: runs a dry run of the command, returning the actions that will be taken. These actions can be found in the custom resource by following step 4.
       • false: runs the command, restoring and replacing the system files and all sites. Any existing sites are reinstalled.
     • priorityList - Prioritizes the restore of any sites. The sites are listed in the order in which they are provided. They are queued for restore first, followed by any remaining sites present in the backup location. Valid arguments are:
       • myportalsite.com

       • mysecondportalsite.com

     Table 2. The options that are needed for each Developer Portal restore

     	all	system	site
     type	X	X	X
     siteName			X
     systemName		X
     timestamp	X
     dryRun	X
     priorityList	X

  2. Create the PortalRestore custom resource in the namespace of the Developer Portal subsystem to trigger the restore process:

     kubectl create -f portalrestore_cr.yaml -n <ns-of-portal-susbystem>

  3. Optional: You can list current restores by using the command:

     kubectl get portalrestore -n <ns-of-portal-susbystem>

     Here is an example of some possible output:

     
     NAME                   READY     AGE
     portal-restore-twjrp   Ready     28s

  4. Optional: You can get a detailed view of a specific restore by using the command:

     kubectl get portalrestore <restore-name> -n <ns-of-portal-susbystem> -o yaml

     Here is an example of some possible output:

     apiVersion: portal.apiconnect.ibm.com/v1beta1
     kind: PortalRestore
     metadata:
       creationTimestamp: "2020-11-11T10:14:39Z"
       generateName: portal-restore-
       generation: 1
       name: portal-restore-8f4c5
       namespace: portal
       resourceVersion: "1497597"
       selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalrestores/portal-restore-8f4c5
       uid: 5a70b657-d998-4989-b9b6-8480e986311d
     spec:
       siteName: "20200526.090209"
       type: site
     status:
       backupId: ""
       commentLeft: ""
       conditions:
       - lastTransitionTime: "2020-11-11T10:14:39Z"
         reason: RestoreFailed
         status: "False"
         type: Error
       - lastTransitionTime: "2020-11-11T10:14:39Z"
         reason: RestorePending
         status: "False"
         type: Pending
       - lastTransitionTime: "2020-11-11T14:55:32Z"
         message: |-
           2020-05-26 13:11:25: CLI task (restore:site) starting.
           2020-05-26 13:11:36: Deleting any old site data if it exists...
           2020-05-26 13:13:18: Restoring site test.com ...
           2020-05-26 13:13:18: .
           2020-05-26 13:13:19: Patching the database dump before restoring ...
           2020-05-26 13:13:48: .
           2020-05-26 13:19:41: Triggering the drush command to restore the site. This may take some time.
           2020-05-26 13:19:48: .
           2020-05-26 13:25:40: Drush restore command completed. Setting up the site.
           2020-05-26 13:25:48: .
           2020-05-26 13:30:49: Site test.com restored from backup /var/aegir/backups/test.com-20200526.090209.tar.gz. Initializing the site...
           2020-05-26 13:30:52: Restore site completed.
           2020-05-26 13:30:52: CLI task (restore:site) completed successfully.
         reason: RestoreComplete
         status: "True"
         type: Ready
       - lastTransitionTime: "2020-11-11T14:55:32Z"
         reason: RestoreInProgress
         status: "False"
         type: Running
       - lastTransitionTime: "2020-11-11T10:14:39Z"
         reason: RestoreStatusUnknown
         status: "False"
         type: Warning
       fullBackupName: ""
       id: bybyibd99wxtmkmk
       message: ""

• Troubleshooting
  1. Check the Developer Portal backup CR or restore CR.
     Use command:

     kubectl get <portalbackup/portalrestore> <bup-name/restore-name> -n <namespace> -o yaml

     Here is an example of some possible output:

     apiVersion: portal.apiconnect.ibm.com/v1beta1
     kind: PortalRestore
     metadata:
       creationTimestamp: "2020-05-26T13:10:46Z"
       generateName: portal-restore-
       generation: 1
       name: portal-restore-x2bf9
       namespace: portal
       resourceVersion: "1493471"
       selfLink: /apis/portal.apiconnect.ibm.com/v1beta1/namespaces/portal/portalrestores/portal-restore-x2bf9
       uid: 9f6a1e22-55ab-4595-8b98-bdfaec40d58c
     spec:
       type: site
     status:
       backupId: ""
       commentLeft: ""
       conditions:
       - lastTransitionTime: "2020-05-26T13:10:49Z"
         message: |
           Please provide a site to restore. To restore all sites use value 'all' in the CR
         status: Failed
         type: Ready
       fullBackupName: ""
       id: ""
       message: ""

     Read the message inside the conditions of the Developer Portal backup CR or the restore CR.

  2. Retrieving logs
     Look for information in the logs of the ibm-apiconnect pod:

     • Locate the ibm-apiconnect pod with:

       kubectl get pods -n <namespace-of-portal-subsystem>

     • View the logs with:

       kubectl logs <ibm-apiconnect-pod-NAME> -n <namespace-of-portal-subsystem>