Set up Grafana to monitor statistics
Gather transfer statistics from HSTS deployment in IBM Cloud Pak® for Integration.
- Log in into the cluster
- Create a namespace
- Add Ceph as storage class to OCP cluster
- Get the IBM entitlement key
- Optional: Create an Aspera node credentials secret
- Install the operators
- Test and confirm that Grafana is receiving data stats
Log in into the cluster
Log in to the cluster by following the procedures in Overview of installation.
Create a namespace
In OpenShift the Namespaces
are required to organize users and their
applications. To create a create a new namespace, you can follow this procedure:
- In the Administration menu, click Namespaces and click Create Namespace.
- Enter the Name for your Namespace. You can use aspera since the following guide uses aspera as the namespace.
- Enter the Labels.
- Select the Default network policy according to your local policies.
- Click Create.
kubectl create ns <namespace>
Add Ceph as storage class to OCP cluster
- Clone the remote repository by running the following
command:
git clone --single-branch --branch v1.7.9 https://github.com/rook/rook.git
- Navigate to the directory where you have cloned the
repository:
cd rook/cluster/examples/kubernetes/ceph
- Apply the following settings to the YAML
file:
oc create -f common.yaml -f crds.yaml -f operator-openshift.yaml -f cluster.yaml -f ./csi/rbd/storageclass.yaml -f ./csi/rbd/pvc.yaml -f filesystem.yaml -f ./csi/cephfs/storageclass.yaml -f ./csi/cephfs/pvc.yaml -f toolbox.yaml
Get the IBM entitlement key
The entitlement key is required to pull the images needed by the Cloud Pak. Go to the IBM Container library and log in with your IBMid. Copy your entitlement key IBM_ENTITLEMENT_KEY and use it to create a secret for your IBM entitlement key. Run the following command:
oc create secret docker-registry ibm-entitlement-key \
--docker-username=cp \
--docker-password=$IBM_ENTITLEMENT_KEY \
--docker-server=cp.icr.io \
--namespace=aspera
Optional: Create an Aspera node credentials secret
Define your Node API credentials and use them to configure the IBM Aspera HSTS operator.
uuid
based on the uuidgen
command: oc create secret generic asperanoded-creds --from-literal=NODE_USER=nodeuser --from-literal=NODE_PASS=`uuidgen` -n aspera
<instancename>-asperanoded-admin.
You can assign your
instance name to the variable $INSTANCE_NAME. To assign you instance name to the
variable, run this
command:INSTANCE_NAME=`oc get IbmAsperaHsts -n aspera -o jsonpath='{.items[0].metadata.name}'`
quickstart
.Install the operators
- In the OperatorHub, search for IBM Cloud Pak for Integration. Install the operators in all namespaces by following the procedures in Installing the operators by using the Red Hat OpenShift console or Installing the operators by using the CLI.
- When the installation completes, go to Installed Operators:
- Search for IBM Cloud Pak for Integration.
- Click Runtime and instances > Create instance.
- Accept the license by setting the slider to true.
- Set the Storage Class, for example, to rook-cephfs.
- Wait until the status of the Platform UI instance becomes Ready.
Note: This process can take up to 45 minutes. - From the instance page, click the Platform UI and log in with OpenShift credentials (username and password).
- Create a Quick start instance by going to
Operators > Installed Operators:
- Click Aspera HSTS Service.
- Accept the license by going to the License configuration: in the Accept license option, set the slider to true.
- Copy and paste the Aspera license from https://license.aspera.us/evals.
- Set the Operand version from the list of supported versions.
Note: The Redis operator may take time to connect. During this time, the IBM Aspera HSTS shows as Pending in the Platform UI. - When the status is Ready, run this command to verify that the instance
can be monitored:
oc get monitoringdashboards -n aspera
The output should include the state -dashboard. If the output gives a different state, delete the IBM Aspera HSTS operator that is under Workloads > Pods.
To find the IBM Aspera HSTS operator, run this command:oc get pods -l name=ibm-aspera-hsts-operator -n aspera
Wait for the state to become Ready and recheck that the instance can be monitored.
- On the Platform UI page, go to the IBM Aspera HSTS instance and click Monitoring to launch the Grafana page.
Test and confirm that Grafana is receiving data statistics
Follow these steps to confirm that Grafana is receiving your data statistics:
- To get the
asperanoded
username and assign it to the NODE_USER variable, run this command:NODE_USER=`oc get secrets $INSTANCE_NAME-asperanoded-admin -n aspera -o jsonpath='{.data.user}' | sed "s/\"//g" | base64 -D`
- To get the
asperanoded
password and assign it to the NODE_PASS, run this command:NODE_PASS=`oc get secrets $INSTANCE_NAME-asperanoded-admin -n aspera -o jsonpath='{.data.pass}' | sed "s/\"//g" | base64 -D`
- To get the internet host name and access the Node API, run this
command:
NODE_HOST=`oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -n aspera -o jsonpath='{.items[*].spec.host}'`
- To list the
access_key
that are on the transfer server, run the following curl command. Anaccess_key
corresponds to the folder path on the storage where your users upload and download data:curl -ki -u $NODE_USER:$NODE_PASS https://$NODE_HOST/access_keys
- The output for the initial API call returns an empty array:
`HTTP/1.1 200 OK cache: no-cache content-type: application/json; charset=utf-8 date: Thu, 11 Nov 2021 02:59:07 GMT content-length: 5 set-cookie: 5e8c70309ddcef6435a36900ea9a4ff5=5faa3ee8ebeeda9379f0715be1f57130; path=/; HttpOnly; Secure; SameSite=None cache-control: private [ ] `
- To create the access key, run this
command:
From the output, save the id and the secret. You will need these values for the HTTP Basic access authentication in the Node API requests:curl -ki -u "$NODE_USER:$NODE_PASS" -X POST -d @- https://$NODE_HOST/access_keys <<EOF { "storage": { "type": "local", "path": "/data" } } EOF
{ "id" : "talIlBn3MEQ2WyK6qKhE8x4BSKkOIGAFrhNc2CRJIdI", "secret" : "H4yihmIyLUks7270rX4PNS02APmKRF4bbWMx0sv7Cmwbkw-tYUCwq2CYNeYoaDLFkRKViO_wzqGD_4IWkAszkA", "root_file_id" : "1", "token_verification_key" : null, "license" : null, "storage" : { "type" : "local", "path" : "/data" } }
- To get the host name for FASP transfers, run this command:
oc get services -l name=tcp-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o wide -n aspera
- Get the
EXTERNAL-IP
from the output. All Node API call are authenticated using the access key and secret (ACCESS_KEY:SECRET
).Note: If you selectupload_setup
as the option for the transfer, the JSON file has thetransfer_spec
in the output.
- Get the
- To create a file for a transfer, run this
command:
host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/1/files -X POST -H "Content-Type: application/json" -d '{"name": "filename", "type": "file"}'
- Browse the file in the file system using this API
command:
host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/1/files
- To get the
token
for an upload, run this command:host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/files/upload_setup -d '{"transfer_requests":[{"transfer_request": {"paths": [{"source":"filename","destination":"anotherfilename"}], "destination_root": "/"}}]}
The output contains the required data for a transfer. For example, a node-to-node transfer moves data from one node of a network to another.host=$(oc get route -l name=http-proxy,app.kubernetes.io/managed-by=ibm-aspera-hsts,cr.namespace=aspera -o jsonpath='{.items[*].spec.host}');curl -ki -u "ACCESS_KEY:SECRET" https://$host/ops/transfers -X POST -H "Content-Type: application/json" -d @- <<EOF { "paths" : [ { "source" : "filename", "destination" : "anotherfilename" } ], "source_root" : "", "destination_root" : "/", "token" : "ATB3_AGsyyL30rOpCg3gtAtVJ51MN2icmWPI3ksrLKJjQHDgHrgSoHq6Wwcamp6ejT9NGWtfznvJUZhTr-s55VE6DFX_43WoA7kdt93rHUgJ6s8R5ooAAFqrIWth8NgBsjDGIA7NATR_3BTA", "direction" : "send", "target_rate_cap_kbps" : 5000000, "cipher" : "aes-128", "tags" : null, "rate_policy_allowed" : "fair", "rate_policy" : "fair", "target_rate_kbps" : 3000000, "min_rate_cap_kbps" : 0, "min_rate_kbps" : 0, "lock_min_rate" : true, "remote_host" : "10.17.61.151", "remote_user" : "xfer", "sshfp" : "SHA256:Xnxc5aVKRGHzrAcC4dFqB4F0tG0o/wBzEBYNuN87/WU", "ssh_port" : 33001, "fasp_port" : 33001, "http_fallback" : false } EOF
- To monitor the data, go to the Platform UI page, go to the IBM Aspera HSTS instance and click Monitoring to launch the Grafana page.