Using the IBM Spectrum Discover application catalog

Follow the procedure to access information on endpoints:

1. Go to IBM Spectrum Discover Documentation.
2. Choose the version of IBM Spectrum Discover that you are running.
3. Go to Table of Contents > REST API > Application management using APIs.

Run this command to query the applications that are available on dockerhub:

tcurl https://${OVA}/api/application/appcatalog/publicregistry | jq

The output that is generated contains information that is gathered from the image itself (and from dockerhub).

After you decide which application you want to run, from the query output, you can use it as a Kubernetes pod within IBM Spectrum Discover. Create a JSON-formatted file with the following information (the file that is created is named example.json):

{
  "repo_name": "ibmcom/spectrum-discover-example-application",
  "version": "1.2.3",
  "description": "Unique description about your use of this application",
  "application_name": "example",
  "my_env_var": "my_value",
  "LOG_LEVEL": "DEBUG"
}

Run the following command to start the application as a Kubernetes pod:

tcurl_json https://localhost/api/application/appcatalog/helm -d@example.json -X POST | jq

You can add environment variables to the JSON example. These environment variables can be ones that your application needs or they can be ones that can override some software development kit (SDK) values. The application SDK supports the following environment variables that can override default settings:

LOG_LEVEL - INFO (default), DEBUG

Specifies the log level for the application to run with.

MAX_POLL_INTERVAL - 86400000 (in milliseconds)(default - 1 day)

Specifies when the Kafka consumer becomes unresponsive. Set this value higher than the time it takes for the application to process up to 100 records before it sends the reply to IBM Spectrum Discover. The default allows approximately 15 minutes for each record.

PRESERVE_STAT_TIME - False (default), True

Specifies whether to preserve atime or mtime when you run the deep-inspection application. If the application processes records from Network File System (NFS), Server Message Block (SMB), or local IBM Storage Scale connections, the system preserves the exact atime or mtime (in nanoseconds).

If the application processes records from a remote IBM Storage Scale connection, the system preserves atime or mtime up to and including seconds (with no subsecond preservation). The connection user must also have write access to the files. If the connection user does not have write access to the files, the system skips restoration of the atime or mtime because of permission errors. If DEBUG is on, you can see the original atime or mtime in the logs, so you can potentially manually restore any that fail.

After you download or build your application source code, perform the following steps to run an application as a python file:

1. Issue the following command to install the required python packages:

   sudo python3 -m pip install -r requirements.txt
   
   

2. Issue the following command to install the required OS packages. If you have any NFS connections, you must install the required packages.

   sudo yum install nfs-utils

   Note: For each connection you have in the administration page in the UI, the IBM Spectrum Discover application SDK creates the following:

   • A sftp connection for each IBM Storage Scale connection
   • A local NFS mount for each NFS connection
   • A boto3 client for each COS connection
3. Define environment variables as shown.

   export SPECTRUM_DISCOVER_HOST=https://<spectrum_discover_host>
   # The IP or Fully Qualified Domain Name of your IBM Spectrum Discover instance.
   # Default: https://localhost
   

   export APPLICATION_NAME=<application_name>
   # A short but descriptive name of your application.
   # EX: exif-header-extractor-application or cos-x-amz-meta-extractor-application
   # Default: sd_sample_application
   

   export APPLICATION_USER=<application_user>
   # A dataadmin user. Ex: sdadmin
   

   export APPLICATION_USER_PASSWORD=<application_user_password>
   # The password of the above dataadmin user.
   

   export KAFKA_DIR=<directory_to_save_certificates>
   # Directory where the kafka certs will be saved.
   # Default: ./kafka
   

   export LOG_LEVEL=<ERROR WARNING INFO DEBUG>
   # Default: INFO
   

4. Start the sample application, by issuing the following command:

   sudo -E python3 ./ExampleApplication.py
   

An application by design processes each of the records one at a time. You can scale the number of replicas the pod is running to process records in parallel. You can scale up to 10 replicas based on the number of partitions available for the Kafka topics. Create a JSON-formatted file with the following information (the file that is created is named replicas.json):

{
  "replicas": 10
}

Then, run the following command to scale the replicas:

tcurl_json https://localhost/api/application/appcatalog/helm/interesting-anaconda-example-application -d@replicas.json -X PATCH

Run the following command to stop an application (no matter how many replicas you scale):

tcurl_json https://localhost/api/application/appcatalog/helm/interesting-anaconda -X DELETE | jq