Set up Grafana to monitor statistics

Gather transfer statistics from HSTS deployment in IBM Cloud Pak® for Integration.

  1. Log in into the cluster
  2. Create a namespace
  3. Add Ceph as storage class to OCP cluster
  4. Get the IBM entitlement key
  5. Optional: Create an Aspera node credentials secret
  6. Install the operators
  7. Test and confirm that Grafana is receiving data stats

Log in into the cluster

Log in to the cluster by following the procedures in Overview of installation.

Create a namespace

In OpenShift the Namespaces are required to organize users and their applications. To create a create a new namespace, you can follow this procedure:

  1. In the Administration menu, click Namespaces and click Create Namespace.
  2. Enter the Name for your Namespace. You can use aspera since the following guide uses aspera as the namespace.
  3. Enter the Labels.
  4. Select the Default network policy according to your local policies.
  5. Click Create.
You can also run the following command to create the namespace:
kubectl create ns <namespace>

Add Ceph as storage class to OCP cluster

The Ceph is a distributed storage system that can be used to deploy clusters. To add Ceph as storage, follow this procedure:
  1. Clone the remote repository by running the following command:
    git clone --single-branch --branch v1.7.9 https://github.com/rook/rook.git
  2. Navigate to the directory where you have cloned the repository:
    cd rook/cluster/examples/kubernetes/ceph
  3. Apply the following settings to the YAML file:
    oc create -f common.yaml -f crds.yaml -f operator-openshift.yaml -f cluster.yaml -f ./csi/rbd/storageclass.yaml -f ./csi/rbd/pvc.yaml -f filesystem.yaml -f ./csi/cephfs/storageclass.yaml -f ./csi/cephfs/pvc.yaml -f toolbox.yaml

Get the IBM entitlement key

The entitlement key is required to pull the images needed by the Cloud Pak. Go to the IBM Container library and log in with your IBMid. Copy your entitlement key IBM_ENTITLEMENT_KEY and use it to create a secret for your IBM entitlement key. Run the following command:

oc create secret docker-registry ibm-entitlement-key \
 --docker-username=cp \
 --docker-password=$IBM_ENTITLEMENT_KEY \
 --docker-server=cp.icr.io \
 --namespace=aspera

Optional: Create an Aspera node credentials secret

Define your Node API credentials and use them to configure the IBM Aspera HSTS operator.

Run this command and store the secret according to your local security practices. If you copy and paste the command, make sure to edit the values NODE_USER and NODE_PASS. For example, the password value NODE_PASS is being set to uuid based on the uuidgen command:
Note: This password is not secure as it does not comply with the security best practices. This password is only an example for this documentation.
oc create secret generic asperanoded-creds --from-literal=NODE_USER=nodeuser --from-literal=NODE_PASS=`uuidgen` -n aspera
If you skip this step, a default credential with a random password is created. The default secret is saved under the key <instancename>-asperanoded-admin. You can assign your instance name to the variable $INSTANCE_NAME. To assign you instance name to the variable, run this command:
INSTANCE_NAME=`oc get IbmAsperaHsts -n aspera -o jsonpath='{.items[0].metadata.name}'`
Note: If you didn't change your instance name, the default name of the instance is quickstart.

Install the operators

  1. In the OperatorHub, search for IBM Cloud Pak for Integration. Install the operators in all namespaces by following the procedures in Installing the operators by using the Red Hat OpenShift console or Installing the operators by using the CLI.
  2. When the installation completes, go to Installed Operators:
    1. Search for IBM Cloud Pak for Integration.
    2. Click Runtime and instances > Create instance.
    3. Accept the license by setting the slider to true.
    4. Set the Storage Class, for example, to rook-cephfs.
    5. Wait until the status of the Platform UI instance becomes Ready.
    Note: This process can take up to 45 minutes.
  3. From the instance page, click the Platform UI and log in with OpenShift credentials (username and password).
  4. Create a Quick start instance by going to Operators > Installed Operators:
    1. Click Aspera HSTS Service.
    2. Accept the license by going to the License configuration: in the Accept license option, set the slider to true.
    3. Copy and paste the Aspera license from https://license.aspera.us/evals.
    4. Set the Operand version from the list of supported versions.
    Note: The Redis operator may take time to connect. During this time, the IBM Aspera HSTS shows as Pending in the Platform UI.
  5. When the status is Ready, run this command to verify that the instance can be monitored:
    oc get monitoringdashboards -n aspera

    The output should include the state -dashboard. If the output gives a different state, delete the IBM Aspera HSTS operator that is under Workloads > Pods.

    To find the IBM Aspera HSTS operator, run this command:
    oc get pods -l name=ibm-aspera-hsts-operator -n aspera

    Wait for the state to become Ready and recheck that the instance can be monitored.

  6. On the Platform UI page, go to the IBM Aspera HSTS instance and click Monitoring to launch the Grafana page.

Test and confirm that Grafana is receiving data statistics

Follow these steps to confirm that Grafana is receiving your data statistics:

  1. To get the asperanoded username and assign it to the NODE_USER variable, run this command:
    NODE_USER=`oc get secrets $INSTANCE_NAME-asperanoded-admin -n aspera -o jsonpath='{.data.user}' | sed "s/\"//g" | base64 -D`
  2. To get the asperanoded password and assign it to the NODE_PASS, run this command:
    NODE_PASS=`oc get secrets $INSTANCE_NAME-asperanoded-admin -n aspera -o jsonpath='{.data.pass}' | sed "s/\"//g" | base64 -D`
  3. To get the internet host name and access the Node API, run this command:
    NODE_HOST=`oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -n aspera -o jsonpath='{.items[*].spec.host}'`
  4. To list the access_key that are on the transfer server, run the following curl command. An access_key corresponds to the folder path on the storage where your users upload and download data:
    curl -ki -u $NODE_USER:$NODE_PASS https://$NODE_HOST/access_keys
  5. The output for the initial API call returns an empty array:
    `HTTP/1.1 200 OK
    cache: no-cache
    content-type: application/json; charset=utf-8
    date: Thu, 11 Nov 2021 02:59:07 GMT
    content-length: 5
    set-cookie: 5e8c70309ddcef6435a36900ea9a4ff5=5faa3ee8ebeeda9379f0715be1f57130; path=/; HttpOnly; Secure; SameSite=None
    cache-control: private
    [
    ]
    `
  6. To create the access key, run this command:
    curl -ki -u "$NODE_USER:$NODE_PASS" -X POST -d @- https://$NODE_HOST/access_keys <<EOF { "storage": { "type": "local", "path": "/data" } } EOF
    From the output, save the id and the secret. You will need these values for the HTTP Basic access authentication in the Node API requests:
    {
      "id" : "talIlBn3MEQ2WyK6qKhE8x4BSKkOIGAFrhNc2CRJIdI",
      "secret" : "H4yihmIyLUks7270rX4PNS02APmKRF4bbWMx0sv7Cmwbkw-tYUCwq2CYNeYoaDLFkRKViO_wzqGD_4IWkAszkA",
      "root_file_id" : "1",
      "token_verification_key" : null,
      "license" : null,
      "storage" : {
        "type" : "local",
        "path" : "/data"
      }
    }
  7. To get the host name for FASP transfers, run this command:
    oc get services -l name=tcp-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o wide -n aspera
    1. Get the EXTERNAL-IP from the output. All Node API call are authenticated using the access key and secret (ACCESS_KEY:SECRET).
      Note: If you select upload_setup as the option for the transfer, the JSON file has the transfer_spec in the output.
  8. To create a file for a transfer, run this command:
    host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/1/files -X POST -H "Content-Type: application/json" -d '{"name": "filename", "type": "file"}'
  9. Browse the file in the file system using this API command:
    host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/1/files
  10. To get the token for an upload, run this command:
    host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/upload_setup -d '{"transfer_requests":[{"transfer_request": {"paths": [{"source":"filename","destination":"anotherfilename"}], "destination_root": "/"}}]}
    The output contains the required data for a transfer. For example, a node-to-node transfer moves data from one node of a network to another.
    host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/ops/transfers -X POST -H "Content-Type: application/json" -d @- <<EOF
    {
    "paths" : [
    {
    "source" : "filename",
    "destination" : "anotherfilename"
    }
    ],
    "source_root" : "",
    "destination_root" : "/",
    "token" : "ATB3_AGsyyL30rOpCg3gtAtVJ51MN2icmWPI3ksrLKJjQHDgHrgSoHq6Wwcamp6ejT9NGWtfznvJUZhTr-s55VE6DFX_43WoA7kdt93rHUgJ6s8R5ooAAFqrIWth8NgBsjDGIA7NATR_3BTA",
    "direction" : "send",
    "target_rate_cap_kbps" : 5000000,
    "cipher" : "aes-128",
    "tags" : null,
    "rate_policy_allowed" : "fair",
    "rate_policy" : "fair",
    "target_rate_kbps" : 3000000,
    "min_rate_cap_kbps" : 0,
    "min_rate_kbps" : 0,
    "lock_min_rate" : true,
    "remote_host" : "10.17.61.151",
    "remote_user" : "xfer",
    "sshfp" : "SHA256:Xnxc5aVKRGHzrAcC4dFqB4F0tG0o/wBzEBYNuN87/WU",
    "ssh_port" : 33001,
    "fasp_port" : 33001,
    "http_fallback" : false
    }
    EOF
  11. To monitor the data, go to the Platform UI page, go to the IBM Aspera HSTS instance and click Monitoring to launch the Grafana page.