Installing new SSH key during migration

Part 1: Add the new SSH key to all target systems

To get familiar with the sample runbook "Example: Add new SSH key to target systems" or if you have only a few target systems, then you can run the sample runbook from the UI of the original deployment. Each of the runbook parameters includes a description of what to specify and which format to use.

If you have many targets, then you can use the following description to define a batch of up to about 100 target hostnames and invoke the runbook by using the runbook API to add the new SSH key to each of those targets.

You must perform this part by using the original deployment ("OLD_INSTALL"), which has its SSH key that is already installed on all target systems. You can run the commands from any environment that has network access to your original deployment.

• If the original deployment is IBM Cloud Pak® for AIOps

  • Set the value of the CURL_BASEURL_OLD_INSTALL variable to the IBM Cloud Pak® for AIOps user interface URL and replace the path with "/aiops/api/story-manager/rba/v1". The following example assumes that the URL is "https://cpd-cp4waiops.apps.mycluster.mycompany.com/zen/#/homepage".

  • To set the value of the CURL_AUTH_TOKEN_OLD_INSTALL variable you must have an API key created first. If you need to generate an API key, follow the instructions in Generating API keys for authentication. The <username> in the command that follows is the user ID that you used to log in to the IBM Cloud Pak® for AIOps user interface when generating the API key.

  • Use CURL_OPTIONS_OLD_INSTALL to specify any options that you need to use for the curl command itself. For example, use "-k" to allow insecure SSL connections if you have not installed the deployment's certificate yet.

    export CURL_BASEURL_OLD_INSTALL="https://cpd-cp4waiops.apps.mycluster.mycompany.com/aiops/api/story-manager/rba/v1"
    export CURL_AUTH_TOKEN_OLD_INSTALL=`echo -n "<username>:<api_key_value>" | base64 -w 0`
    export CURL_AUTH_METHOD_OLD_INSTALL="ZenApiKey"
    export CURL_OPTIONS_OLD_INSTALL="-k"
    

• If the original deployment is IBM Netcool Operations Insight

  • Set the value of the CURL_BASEURL_OLD_INSTALL variable to the hostname where the IBM Netcool Operations Insight service is running, prefix it with "https://" and append "/api/v1/rba". For example, https://netcool.apps.mycluster.mycompany.com/api/v1/rba.

  • To set the value of the CURL_AUTH_TOKEN_OLD_INSTALL variable you must have an API key created first. If you need to generate an API key, follow the instructions in API Keys.

  • Use CURL_OPTIONS_OLD_INSTALL to specify any options that you need to use for the curl command itself. For example, use "-k" to allow insecure SSL connections if you have not installed the deployment's certificate yet.

    export CURL_BASEURL_OLD_INSTALL="https://netcool.apps.mycluster.mycompany.com/api/v1/rba"
    export CURL_AUTH_TOKEN_OLD_INSTALL=`echo -n "<api_key_name>:<api_key_password>" | base64 -w 0`
    export CURL_AUTH_METHOD_OLD_INSTALL="Basic"
    export CURL_OPTIONS_OLD_INSTALL="-k"
    

• Common for all deployment types

  Run the following command to verify the base URL and the authentication token:

  curl -X GET ${CURL_OPTIONS_OLD_INSTALL} -H "Authorization: ${CURL_AUTH_METHOD_OLD_INSTALL} ${CURL_AUTH_TOKEN_OLD_INSTALL}" "${CURL_BASEURL_OLD_INSTALL}/status"
  

  The previous command is expected to produce some informational data about the speed of the transfer and some JSON result data similar to the following:

  {"rbs":0,"as":0,"db":0,"connection":{"AWX":0,"IMPACT":2,"SCRIPT":0,"GITHUB":0},"errors":[],"warnings":[]}
  

  Note: Some of the connection items might have a different value than "0", and the "errors" array within the JSON result data might be nonempty (if the connection is in a failed state) - this is not a problem in this context.

  The values for the following set of variables must be defined as outlined here:

  • RUNBOOK_ID_ADD_SSH_KEY: The runbook ID of the sample runbook Example: Add new SSH key to target systems in the original deployment.
  • PARAM_TARGET_HOSTNAMES: A stringified array of hostnames, which defines the batch of target systems to be updated in this iteration.
  • PARAM_TARGET_USER: The user id to be used on the target system. This user id is the same for all target systems in the batch. If different user ids need to be used, then the batches of target hostnames need to be split accordingly.
  • PARAM_AUTHORIZED_KEYS_FILE_PATH: The relative path to the authorized keys file within the target user's home directory, or an absolute path. You can use an arbitrary file name to test the tooling (but the file needs to exist on the target). Defaults to ".ssh/authorized_keys".
  • PARAM_NEW_SSH_KEY: The complete SSH key as copied from the SSH integration of the new deployment.
  • PARAM_OPTIONS: Optional options for the SSH key. Must conform to the SSH specification (not validated by the tooling). Make sure to escape double quotes (see example below).
  • PARAM_COMMENT: The comment to be saved along with PARAM_NEW_SSH_KEY in the authorized_keys file on the target systems. Must conform to the SSH specification (not validated by the tooling). Defaults to "RunbookAutomation".
  • PARAM_VERBOSE: Set this to true to create verbose output for each of the editing actions on the target systems. Set this to false to just get a successful or failed result for the editing actions on the target systems.
  • PARAMETER_ADD_FILE_PATH: Name of the temporary file that gets generated to contain the parameters in JSON format.
  
  

  For example:

  export RUNBOOK_ID_ADD_SSH_KEY="2f2ba8ee-4ee6-4a36-b213-2be5ec65249f"
  
  export PARAM_RAW_TARGETS="doesnotexist1.myzone.mycompany.com"
  export PARAM_RAW_TARGETS=${PARAM_RAW_TARGETS}`for i in {2..100}; do echo -n ",doesnotexist${i}.myzone.mycompany.com"; done`
  export PARAM_TARGET_HOSTNAMES="[${PARAM_RAW_TARGETS}]"
  export PARAM_TARGET_USER="root"
  export PARAM_AUTHORIZED_KEYS_FILE_PATH=".ssh/authorized_keys_test"
  export PARAM_NEW_SSH_KEY="ssh-rsa testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest"
  export PARAM_OPTIONS="from=\\\"my,list,of,addresses\\\""
  export PARAM_COMMENT="RunbookAutomation@myCluster.myzone.mycompany.com"
  export PARAM_VERBOSE="true"
  
  export PARAMETER_ADD_FILE_PATH=parametersAddSSHKey.json
  
  echo "{\"parameters\":[{\"name\":\"targetHostnames\",\"value\":\"${PARAM_TARGET_HOSTNAMES}\"},{\"name\":\"targetUser\",\"value\":\"${PARAM_TARGET_USER}\"},{\"name\":\"authorizedKeysFilePath\",\"value\":\"${PARAM_AUTHORIZED_KEYS_FILE_PATH}\"},{\"name\":\"newSSHKey\",\"value\":\"${PARAM_NEW_SSH_KEY}\"},{\"name\":\"options\",\"value\":\"${PARAM_OPTIONS}\"},{\"name\":\"comment\",\"value\":\"${PARAM_COMMENT}\"},{\"name\":\"verbose\",\"value\":\"${PARAM_VERBOSE}\"}]}" > $PARAMETER_ADD_FILE_PATH
  

  Run the following command to add the new SSH key to the single batch of target systems, as defined above:

  curl -X POST ${CURL_OPTIONS_OLD_INSTALL} -s -H "Authorization: ${CURL_AUTH_METHOD_OLD_INSTALL} ${CURL_AUTH_TOKEN_OLD_INSTALL}" -H "Expect:" -H "Content-Type: application/json; charset=utf-8" -d @${PARAMETER_ADD_FILE_PATH} "${CURL_BASEURL_OLD_INSTALL}/runbooks/${RUNBOOK_ID_ADD_SSH_KEY}/execute"
  

  The output on the command line is expected to show the returned runbook instance ID and look similar to the following:

  {"runbookInstanceId":"0d8787ba91b123c7cf2cbd890800be73"}
  

  Login to the UI of the original deployment and navigate to the runbook activity that has been created. The output in runbook activity details is expected to look similar to the following:

  # Successful, with verbose output:
  exists.with.ssh.key.myzone.mycompany.com out: Changed content in .ssh/authorized_keys_test (diff from backup file):
  exists.with.ssh.key.myzone.mycompany.com out: 0a1
  exists.with.ssh.key.myzone.mycompany.com out: > from="my,list,of,addresses" ssh-rsa testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest RunbookAutomation@myCluster.myzone.mycompany.com
  exists.with.ssh.key.myzone.mycompany.com out: Done
  exists.with.ssh.key.myzone.mycompany.com status: successful
  
  # Successful, with non-verbose output:
  exists.with.ssh.key.myzone.mycompany.com status: successful
  
  # Unsuccessful, reason: file PARAM_AUTHORIZED_KEYS_FILE_PATH does not exist:
  exists.but.ssh.key.file.does.not.exist.myzone.mycompany.com out: Error: File .ssh/authorized_keys_test does not exist.
  exists.but.ssh.key.file.does.not.exist.myzone.mycompany.com status: unsuccessful
  
  # Failed, reason: old SSH key does not exist on target system
  exists.but.ssh.key.not.installed.myzone.mycompany.com error: Authentication with target machine failed
  exists.but.ssh.key.not.installed.myzone.mycompany.com status: failed
  
  # Failed, reason: target system not found
  doesnotexist1.myzone.mycompany.com error: Connection to target via TCP unsuccessful.
  doesnotexist1.myzone.mycompany.com status: failed
  

  If you have more than about 100 targets and you want to get all of your targets updated, then you must split your total list of targets into batches, and run the runbook for each batch. If you choose to write a script that loops over the set of batches, make sure to add some sleep time after you have started a batch, so the Runbook Automation service can finish the work in the background. Once you have run a few batches, you can review the actual execution time for each of them in the runbook activities view, so you can adjust the sleep time (for example, allow 30 seconds per batch).

Part 2: (Optional) Validate the new SSH key and remove the old SSH key from all target systems

You must perform this part by using the new deployment, which by now (as per part 1) has its SSH key installed on all target systems. The following steps can also be used to verify that the new key works.

Warning: Run the following steps only when the original deployment is no longer intended to run any actions on the target systems.

To get familiar with the sample runbook "Example: Remove old SSH key from target systems" or if you have only a few target systems, then you can run the sample runbook from the UI of the new deployment. Each of the runbook parameters includes a description of what to specify and which format to use.

If you want to use the runbook API to invoke the runbook, then you can run the following commands from any environment that has network access to your new deployment ("NEW_INSTALL").

The same comments on creating batches of target systems from part 1 apply to this part.

• The values for the following set of variables must be defined as outlined here:

  • CURL_BASEURL_NEW_INSTALL: Set analogously to CURL_BASEURL_OLD_INSTALL from part 1.
  • CURL_AUTH_TOKEN_NEW_INSTALL: Set analogously to CURL_AUTH_TOKEN_OLD_INSTALL from part 1.
  • CURL_AUTH_METHOD_NEW_INSTALL: Set analogously to CURL_AUTH_METHOD_OLD_INSTALL from part 1.
  • CURL_OPTIONS_NEW_INSTALL: Set analogously to CURL_OPTIONS_OLD_INSTALL from part 1.
  • RUNBOOK_ID_REMOVE_SSH_KEY: The runbook ID of the sample runbook Example: Remove old SSH key from target systems in the new deployment.
  • PARAM_TARGET_HOSTNAMES: See part 1.
  • PARAM_TARGET_USER: See part 1.
  • PARAM_AUTHORIZED_KEYS_FILE_PATH: See part 1.
  • PARAM_OLD_SSH_KEY: The complete SSH key as copied from the SSH integration of the old deployment.
  • PARAM_VERBOSE: See part 1.
  • PARAMETER_REMOVE_FILE_PATH: Name of the temporary file that gets generated to contain the parameters in JSON format.
  
  

  For example:

  # If new deployment is an IBM Cloud Pak for AIOps installation
  export CURL_BASEURL_NEW_INSTALL="https://cpd-cp4waiops.apps.mycluster.mycompany.com/aiops/api/story-manager/rba/v1"
  export CURL_AUTH_TOKEN_NEW_INSTALL=`echo -n "<username>:<api_key_value>" | base64 -w 0`
  export CURL_AUTH_METHOD_NEW_INSTALL="ZenApiKey"
  export CURL_OPTIONS_NEW_INSTALL="-k"
  
  # If new deployment is an IBM Netcool Operations Insight installation
  export CURL_BASEURL_NEW_INSTALL="https://netcool.apps.mycluster.mycompany.com/api/v1/rba"
  export CURL_AUTH_TOKEN_NEW_INSTALL=`echo -n "<username>:<api_key_value>" | base64 -w 0`
  export CURL_AUTH_METHOD_NEW_INSTALL="Basic"
  export CURL_OPTIONS_NEW_INSTALL="-k"
  
  # Verify that the new installation can be reached
  curl -X GET ${CURL_OPTIONS_NEW_INSTALL} -H "Authorization: ${CURL_AUTH_METHOD_NEW_INSTALL} ${CURL_AUTH_TOKEN_NEW_INSTALL}" "${CURL_BASEURL_NEW_INSTALL}/status"
  
  # Set other environment variables
  export RUNBOOK_ID_REMOVE_SSH_KEY="50fc56e5-55a9-429d-8c1e-7f0c59f0557f"
  
  export PARAM_RAW_TARGETS="doesnotexist1.myzone.mycompany.com"
  export PARAM_RAW_TARGETS=${PARAM_RAW_TARGETS}`for i in {2..100}; do echo -n ",doesnotexist${i}.myzone.mycompany.com"; done`
  export PARAM_TARGET_HOSTNAMES="[${PARAM_RAW_TARGETS}]"
  export PARAM_TARGET_USER="root"
  export PARAM_AUTHORIZED_KEYS_FILE_PATH=".ssh/authorized_keys_test"
  export PARAM_OLD_SSH_KEY="ssh-rsa testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest"
  export PARAM_VERBOSE="true"
  
  # Create the parameter file
  export PARAMETER_REMOVE_FILE_PATH=parametersRemoveSSHKey.json
  echo "{\"parameters\":[{\"name\":\"targetHostnames\",\"value\":\"${PARAM_TARGET_HOSTNAMES}\"},{\"name\":\"targetUser\",\"value\":\"${PARAM_TARGET_USER}\"},{\"name\":\"authorizedKeysFilePath\",\"value\":\"${PARAM_AUTHORIZED_KEYS_FILE_PATH}\"},{\"name\":\"oldSSHKey\",\"value\":\"${PARAM_OLD_SSH_KEY}\"},{\"name\":\"verbose\",\"value\":\"${PARAM_VERBOSE}\"}]}" > $PARAMETER_REMOVE_FILE_PATH
  

• Run the following command to remove the old SSH key from the single batch of target systems, as defined above:

  curl -X POST ${CURL_OPTIONS_NEW_INSTALL} -s -H "Authorization: ${CURL_AUTH_METHOD_NEW_INSTALL} ${CURL_AUTH_TOKEN_NEW_INSTALL}" -H "Expect:" -H "Content-Type: application/json; charset=utf-8" -d @${PARAMETER_REMOVE_FILE_PATH} "${CURL_BASEURL_NEW_INSTALL}/runbooks/${RUNBOOK_ID_REMOVE_SSH_KEY}/execute"