Installing prepaid usage monitoring for cloud

For compute workloads that extend to the cloud, you can optimize cloud resource usage by matching cloud-use models with pay-as-you-go licensing. Known as prepaid usage, this option enables you to prepay for hours used by a host's cores (core hours) and consume those hours however you want. To audit prepaid consumption, you must install the solution for prepaid usage monitoring and configure IBM® Spectrum Conductor to upload usage metrics to IBM Cloud Private.

Before you begin

  • Install an IBM Cloud Private cluster. An existing installation of IBM Cloud Private 3.1 or higher is required to track prepaid usage for IBM Spectrum Conductor.
    In your IBM Cloud Private 3.1 environment, generate an API key:
    1. Log on to the IBM Cloud Private boot node and obtain the cluster configuration details.

      You can use either the IBM Cloud Private CLI or the management console to obtain the details. For more information, see the Accessing your cluster from the kubectl CLI topic in IBM Cloud Private documentation.

    2. Use the cloudctl command to log on to the cluster as the admin user:
      > cloudctl login -a https://my_icp_IP:8443 -u admin -p admin -n default --skip-ssl-validation

      where my_icp_IP is the IP address of the host that runs the management console.

    3. Create the spectrumserviceid and generate an API key for it:
      > cloudctl iam service-id-create spectrumserviceid-default-1 -d "SPECTRUM serviceid in default namespace, number 1"
      > cloudctl iam service-api-key-create spectrumapikey-default-1 spectrumserviceid-default-1

      An API key (similar to OiiJ1q3gjIDk2JJGxHK6xeZwU5mrIHyu2nUv38SzrQSA) is generated. Save this key to add later to the icpup.json configuration file.

  • Install an IBM Spectrum Conductor cluster (see Roadmap: Installing IBM Spectrum Conductor). Prepaid monitoring can be installed on your existing 2.3.0 cluster.
  • Download the prepaid monitoring packages from IBM Passport Advantage®. These packages specify licensing terms and extract the files to install prepaid monitoring in your cluster.

About this task

With prepaid monitoring for the cloud, CPU core and socket, GPU socket, and host count metrics are uploaded through an external load index monitor (ELIM) in IBM Spectrum Conductor to a metering service in IBM Cloud Private. You can then audit IBM Spectrum Conductor resource usage from the metering dashboard in IBM Cloud Private.

Follow this task to review the licensing terms for prepaid usage, then install and configure prepaid monitoring in your cluster.

Procedure

  1. Log on to any host in your cluster by using the installation user OS account (cluster administrator, root, or sudo-to-root permission).
  2. Run the prepaid monitoring package that you downloaded previously from Passport Advantage. This package specifies licensing terms and extracts the files to install prepaid monitoring in your cluster.
    • To extract the installation and readme files interactively, enter the command for your host operating system and follow the prompts:
      • On Linux® 64 bit:
        ./conductorprepaid2.3.0.0_x86_64.bin [--extract extract_directory]
      • On Linux for POWER®Power® LE 64 bit:
        ./conductorprepaid2.3.0.0_ppc64le.bin [--extract extract_directory]

      where extract_directory specifies the directory to extract the prepaid monitoring files. If you don't specify this option, the files are extracted by default to /opt/ibm/spectrumcomputing/conductorprepaid/.

    • To extract the installation and readme files without user interaction, first set the IBM_SPECTRUM_CONDUCTOR_LICENSE_ACCEPT environment variable to accept the licensing terms:
      export IBM_SPECTRUM_CONDUCTOR_LICENSE_ACCEPT=Y
      Then, enter the command for your host operating system:
      • On Linux 64 bit:
        ./conductorprepaid2.3.0.0_x86_64.bin --quiet [--extract extract_directory]
      • On Linux for POWER LE 64 bit:
        ./conductorprepaid2.3.0.0_ppc64le.bin --quiet [--extract extract_directory]
      where extract_directory specifies the directory to extract the installation files and readme file for prepaid monitoring. If you don't specify this option, the files are extracted by default to /opt/ibm/spectrumcomputing/conductorprepaid/.
  3. Log on to the primary host in your cluster as the cluster administrator:
    egosh user logon -u Admin
  4. Source the environment:
    • (csh) source $EGO_TOP/cshrc.platform
    • (bash) . $EGO_TOP/profile.platform
  5. Shut down the cluster (this step is required only once):
    egosh ego shutdown all
  6. Back up the following cluster configuration files:
    • $EGO_CONFDIR/ego.shared
    • $EGO_CONFDIR/ego.cluster.*
  7. Log on to each of your management hosts, create a directory (for example, /egofixes), and copy the installation package for your host operating system from the extract directory in step 2 to the /egofixes directory.
    • For Linux 64-bit hosts, copy the egocore-3.7.0_x86_64_build505989.tar.gz file.
    • For Linux for POWER LE 64-bit hosts, copy the egocore-3.7.0_ppc64le_build505989.tar.gz file.
  8. On each management host, install the package with the egoinstallfixes command:
    • For Linux 64-bit hosts, enter:
      egoinstallfixes /egofixes/egocore-3.7.0_x86_64_build505989.tar.gz --silent
    • For Linux for POWER LE 64-bit hosts, enter:
      egoinstallfixes /egofixes/egocore-3.7.0_ppc64le_build505989.tar.gz --silent

    For more information, see egoinstallfixes.

  9. Verify the installation with the pversions command:
    pversions -b 505989

    For more information, see pversions for Linuxpversions for Linux.

  10. On any management host, edit the $EGO_CONFDIR/kernel/conf/ego.shared file and add the following line to the Resource section:
    totalcores   Numeric 15   Y   (Total CPU cores in the cluster)
  11. Edit the $EGO_CONFDIR/kernel/conf/ego.cluster.cluster_name file and add the following line to the Resource Map section:
    totalcores  [all]
  12. Create a new $EGO_CONFDIR/kernel/conf/icpup.json file with the following configuration:
      {
        "icpUrl": "https://my_icp_IP:8443",
        "apiKey": "my_API_key",
        "logLevel": "LOG_INFO"
      }
    where:
    • my_icp_IP is the IP address of the host that runs the IBM Cloud Private management console.
    • my_API_key is the API key that you previously generated for the spectrumserviceid.

    To set the log level, edit the logLevel parameter to any of the following values: LOG_INFO, LOG_WARNING, LOG_ERR, or LOG_DEBUG.

  13. From the primary host, restart the cluster (this step is required only once):
    egosh ego start all

What to do next

  1. Look for upload errors in the elim.icpup.hostname log, where hostname identifies the host on which the ELIM (elim.icp.uploader) runs to upload data to IBM Cloud Private.
    1. To get the name of the host on which the elim.icp.uploader runs, open the $EGO_CONFDIR/../work/data/icpup.work file and look for the timestamp hostname values (for example, 1541451364 egomgmtnode1.example.com).
    2. Log in to this host (for example, egomgmtnode1.example.com) and check data upload status in the $EGO_TOP/kernel/log/elim.icpup.hostname log.
  2. Log on to the IBM Cloud Private console and go to Platform > Metering page to view usage data.

    The following metrics are uploaded to IBM Cloud Private:

    • Total Hosts: Total number of available and closed hosts in the cluster.
    • Total Cores: Total number of CPU cores on the reported hosts.
    • Total GPUs: Total number of GPU processors on the reported hosts.
    • Total Sockets: Total number of CPU sockets on the reported hosts.

    When prepaid usage is first enabled, CPU core and GPU socket data for the previous two days is uploaded immediately to the Metering page. Host and socket usage is uploaded from the current time. Thereafter, the system collects metrics every 5 minutes and uploads to the metering service every 15 minutes.

    For more information, see the IBM Cloud Private metering service topic in IBM Cloud Private documentation.