Importing and exporting data

To export data from your source cluster, you need the following:

• The name of your Speech services custom resource for the source deployment.
• The name of the PostgreSQL datastore authentication secret that you used during installation of the Speech services of the source deployment. The default value is custom-resource-name-postgres-auth-secret.
• The name of the MinIO datastore authentication secret that you used during installation of the Speech services of the source deployment. The default value is custom-resource-name-ibm-minio-auth.
• The instance IDs of the service instances that are associated with the source deployment. After you import your data to a target deployment, you need the IDs from the source deployment to transfer ownership of the resources to the instances in the target deployment.

You can find the instance ID for a service instance in its URL under the Access information section in the Cloud Pak for Data console. For example, the following example shows two Speech services instances, one for Text to Speech and one for Speech to Text:

Clicking the name of the Speech to Text instance, speech-to-text-c248, causes the console to display the About this instance page with details about the instance. The instance ID in the following example is circled in the URL that is located under Access information:

You can use this procedure to find the instance IDs for all service instances in the source and target deployments.

From a Linux machine that has internet access, use the import_export.sh script to export your data to a local directory:

1. Change directories to the root of the directory into which you downloaded or cloned the utilities:

   cd speech-import-export

2. Run the import_export.sh script with the following options:

   ./import_export.sh export -c <custom_resource_name> \
      -o <export_directory> -v <version> \
      -p <postgres_auth_secret_name> -m <minio_auth-secret-name>
      -n ${PROJECT_CPD_INSTANCE}

   where the options and arguments provide the following information:
   • custom_resource_name is the required name of the Speech services custom resource from which the script is to export data.
   • export_directory is the required path name of the directory in which the script is to save the exported data. The script creates the directory if it does not already exist. If the directory does already exist but is not empty, the script overwrites the existing contents.
   • version is 40, which indicates version 4.0.x of Cloud Pak for Data from which you are exporting data.
   • postgres_auth_secret_name is the Kubernetes secret that is used to authenticate to the source PostgreSQL datastore. You can omit the authentication secret if is the same as the default value (custom-resource-name-postgres-auth-secret). You must provide the secret if it is different from the default value.
   • minio_auth_secret_name is the Kubernetes secret that is used to authenticate to the source MinIO datastore. You can omit the authentication secret if is the same as the default value (custom-resource-name-ibm-minio-auth). You must provide the secret if it is different from the default value.
   • ${PROJECT_CPD_INSTANCE} is the name of the project namespace in which the source Speech services are installed. If the namespace is different from the current Red Hat OpenShift context, you must provide the Red Hat OpenShift namespace in which the source Speech services are installed. Otherwise, you can omit the namespace.

The script reports on its status as it creates the export data. It shows its status for each individual datastore and then provides a final status report when the entire export operation is complete. If the script reports no errors, the export process completed successfully.

When you use it to export data, the import_export.sh script does the following:

1. Automatically downloads the following required programs when you run the script for the first time on any machine:
   • mc - The MinIO client
   • cpdbr - The Cloud Pak for Data backup and restore utility, which puts the Speech services into read-only mode
2. Waits for any asynchronous recognition jobs that are processing or queued to complete.
3. Puts the Speech services in read-only mode. This can take up to 30 minutes.
4. Uses the mc program to copy the contents of the MinIO datastore to the export_directory/minio directory.
5. Connects to a PostgreSQL pod and uses the psql utility to export a binary dump of the datastore to the export_directory/postgres directory.
6. Takes the Speech services out of read-only mode. It may take up to 30 minutes for the Speech services to be completely ready to serve requests. Read-only operations, such as speech recognition, speech synthesis, and listing information about models and voices, remain available while a service is in read-only mode. Changes to stateful data, such as creating Speech to Text asynchronous jobs, modifying Speech to Text or Text to Speech custom models, and creating new custom models for either service, are unavailable until the service exits read-only mode.

The exported data adheres to a well-defined directory structure that you must preserve for a subsequent import to succeed. A sample of the directory structure of the exported data follows:

export-directory
├── minio
│   └── stt-customization-icp
│       └── customizations
│           ├── 3755a526-8714-4cac-be0a-f9a9832d0f1b
│           │   ├── allpatches-untarred.zip
│           │   ├── data
│           │   │   └── audio1.1
│           │   ├── patch_acoustic.en-US_BroadbandModel.v2020-01-16
│           │   └── patch_acoustic.en-US_BroadbandModel.v2020-01-16.trainSnapshotDir
│           │       ├── features-000000-000000.tar.gz
│           │       ├── features.txt
│           │       └── train.snapshot
│           └── d861fc22-8a07-41a6-89b5-4543a217b17f
│               ├── allpatches-untarred.zip
│               ├── data
│               │   └── healthcare.1
│               └── patch.en-US_BroadbandModel.v2020-01-16
└── postgres
    ├── stt-async.export.dump
    ├── stt-customization.export.dump
    └── tts-customization.export.dump

If you need to move the exported data to a new machine to import it to the target deployment, you can compress the output directory for the transfer by using a lossless compression algorithm.

To import data to your target cluster, you need the following:

• The name of your Speech services custom resource for the target deployment.
• The name of the PostgreSQL datastore authentication secret that you used during installation of the Speech services of the target deployment. The default value is custom-resource-name-postgres-auth-secret.
• The name of the MinIO datastore authentication secret that you used during installation of the Speech services of the target deployment. The default value is custom-resource-name-ibm-minio-auth.

• Your target deployment must already have installed the same Red Hat OpenShift and Speech services versions as the source deployment. The target deployment must run the same Speech microservices as the source deployment.

  • For Speech to Text, the microservices can include sttRuntime, sttAsync, and sttCustomization.
  • For Text to Speech, they can include ttsRuntime and ttsCustomization.

  For example, if your source deployment ran all of these microservices, the target deployment must also run all of the microservices. Otherwise, the necessary datastores might not exist, and the import will fail.

From a Linux machine that has internet access, use the import_export.sh script to import data that you have previously used the script to export:

1. Change directories to the root of the directory into which you downloaded or cloned the utilities:

   cd speech-import-export

2. Run the import_export.sh script with the following options:

   ./import_export.sh import -c <custom_resource_name> \
      -o <import_directory> -v <version> \
      -p <postgres_auth_secret_name> -m <minio_auth-secret-name> \
      -n ${PROJECT_CPD_INSTANCE}

   where the options and arguments provide the following information:
   • custom_resource_name is the required name of the Speech services custom resource to which the script is to import data.
   • import_directory is the required path name of the directory from which the script can access the previously exported data. The directory must exist and must contain the exported data in the format in which it was created. If you compressed the export data, you must uncompress the data before you can import it.
   • version is 40, which indicates version 4.0.x of Cloud Pak for Data to which you are importing data.
   • postgres_auth_secret_name is the Kubernetes secret that is used to authenticate to the target PostgreSQL datastore. You can omit the authentication secret if is the same as the default value (custom-resource-name-postgres-auth-secret). You must provide the secret if it is different from the default value.
   • minio_auth_secret_name is the Kubernetes secret that is used to authenticate to the target MinIO datastore. You can omit the authentication secret if is the same as the default value (custom-resource-name-ibm-minio-auth). You must provide the secret if it is different from the default value.
   • ${PROJECT_CPD_INSTANCE} is the name of the project namespace in which the target Speech services are installed. If the namespace is different from the current Red Hat OpenShift context, you must provide the Red Hat OpenShift namespace in which the target Speech services are installed. Otherwise, you can omit the namespace.
3. When the import script completes, you must transfer ownership of the imported resources to make the data accessible in the target deployment. To transfer ownership, complete the procedure in Transferring resource ownership. You can transfer ownership as soon as the import script is complete. You do not need to wait for the target services to exit read-only mode to transfer ownership.

The script reports on its status as it imports data. It shows its status for each datastore and then provides a final status report when the entire import operation is complete. If the script reports no errors, the import process completed successfully.

When you use it to import data, the import_export.sh script does the following:

1. Automatically downloads the following required programs when you run the script for the first time on any machine:
   • mc - The MinIO client
   • cpdbr - The Cloud Pak for Data backup and restore utility, which puts the Speech services into read-only mode
2. Waits for any asynchronous recognition jobs that are processing or queued to complete.
3. Puts the Speech services in read-only mode. This can take up to 30 minutes.
4. Uses the mc program to copy the contents of the MinIO datastore from the import_directory/minio directory.
5. Connects to a PostgreSQL pod and uses the psql utility to import the binary dump of the datastore from the import_directory/postgres directory.
6. Takes the Speech services out of read-only mode. It may take up to 30 minutes for the Speech services to be completely ready to serve requests. Read-only operations, such as speech recognition, speech synthesis, and listing information about models and voices, remain available while a service is in read-only mode. Changes to stateful data, such as creating Speech to Text asynchronous jobs, modifying Speech to Text or Text to Speech custom models, and creating new custom models for either service, are unavailable until the service exits read-only mode.