IBM Support

QRadar Cloud Apps (QCA): Best practice guidance for application developers

How To


Summary

As more administrators implement QRadar Cloud Apps (QCA), there is an increase of apps into the cloud-native sphere. To assist developers, the QRadar applications team created a set of best practice guidelines in order to prevent common issues with applications that run in cloud environments. Some of these best practices are required to ensure IBM validation teams do not publish applications that contravene cloud development best practices.

Objective

QRadar® cloud application developers must implement as much of this best practice guide as possible. Developments teams that do not complete 'Required' sections might need to resubmit applications for review, extending the time to publish an application to the X-Force® App Exchange.

Environment

Applications developed for installation on QRadar Cloud Apps (QCA) appliances.

Steps

For readability, the best practice content is divided in to 5 categories: app functionality, storage, resource consumption, dependency management, and the application manifest.

Application functionality

Development teams who write apps need them to be functional when installed and work correctly when installed. This section includes best practice information on communication, infrastructure, and external library guidance for apps to operate in a QRadar Cloud App (QCA) environment.

Required

  • Do not make assumptions about application containers

    Applications that run on QCA are containerized and the infrastructure on which the application runs ought to be considered black boxes. Communication to the QRadar Console is defined in the application manifest file. Developers are expected to keep their apps self-contained with communications to QRadar Console on ports 443 for HTTP API traffic and 514 for Syslog messages. Calls to external services are not subject to these restrictions.

    Assumptions that can cause application issues:
    • Assume the app container runs on a local-link 169.254.x.x IP address. In reality, the application could be running on any IP at any time between the QRadar and QCA platforms.
    • Using /proc/meminfo as a gauge for available host memory. The use of meminfo can be dangerous as the app container is liable to be moved to another host at any time where host resources can differ.
    • Providing SSH capabilities to the host QRadar platform. Application should never access QRadar using SSH as rasises security issues by providing a means for applications to effectively break out of their contained environments. We do not support this behavior on QCA.

  • Syslog message restrictions

    RFC 3164 and RFC 5424 Syslog messages are currently supported over port 514 on the Console via TCP or UDP. All QRadar appliances listen by default on port 514 Syslog data. QRadar does not support multiline-syslog on TCP/UDP port 514 currently.

  • Avoid making changes to qpylib

    The qpylib library exists to provide helper functions when interacting with QRadar. We are required to update this library from time to time. Currently, the version within in the app takes precedence over the qpylib version on the system; however, this behavior is liable to change in the future. Therefore, it is important that developers do not make changes to qpylib within their apps as it can be overridden. The library itself is hosted open source here: https://github.com/IBM/qpylib. Developers are encouraged to raise issues to be addressed. Developers can move their changes into a separate Python module; however you are responsible for maintaining any forked code.

  • Installing extensions that are not present on the X-Force App Exchange

    The QCA platform is backed directly by the X-Force App Exchange. Only extensions that have been published to the X-Force App Exchange will be available to run on QCA. API endpoints such as /api/gui_app_framework/application_creation_task and /config/extension_management/extensions must not be invoked from within the app to install more app content as it might not be available unless it is released.

    Note: This restriction does not apply to extensions which contain multiple apps. We locate and deploy every app present within an extension on the X-Force App Exchange, so the app is available regardless of whether or not it has a top-level card provided it has been packaged as a part of an extension.

  • Do not log sensitive data

    Any sensitive data such as API tokens and passwords must not be logged within the app, including debug logs. Applications in breach of this security policy are liable to be delisted from the X-Force App Exchange immediately due to the serious security implications of clear text data. Developers can consider a placeholder value, such as '********' when you log sensitive data. Developers must also be careful with event payloads or event messages that contain sensitive data in plain text.

    Example of how administrators can obfuscate token values:

    from qpylib import qpylib
    
    endpoint = 'https://example-host/example/endpoint'
    token = 'example-password'
    
    # Exposes the token when debug is enabled
    qpylib.log('Calling endpoint {} (token={}).'.format(endpoint, token), level='debug')
    
    # Logs an obfuscated value if it is set, otherwise None when debug is enabled
    qpylib.log('Calling endpoint {} (token-{}).'.format(endpoint, '****' if token is not None else None), level=debug)
  • Ensure API calls verify certificates

    All requests made to the QRadar must have certificate verification turned on for security. The necessary certificate is provided to apps and is located either at /etc/pki/tls/certs/ca-bundle.crt or /store/consolecert.pem (QCA). Developers can use the qpylib REST functions where possible as qpylib performs certificate handling for you. Guidance has recently been put out that certificate validation will be checked as a part of app validation. Here is an example that displays incorrect validation handling and a better alternative.

    from qpylib import qpylib
    import os.path
    import requests
    
    # No certificate validation performed during the request. Causes errors in startup.log
    response = requests.get('https://qradar-host/api/endpoint', verify=False)
     
    # Determines whether to use the QCA cert or the default cert, then uses it to verify
    qca_path = '/store/consolecert.pem'
    cert_path = qca_path if os.path.isfile(qca_path) else '/etc/pki/tls/certs/ca-bundle.crt'
    response = requests.get('https://qradar-host/api/endpoint', verify=cert_path)
     
    # Handles certificate verification for you
    response = qpylib.REST('GET', '/api/endpoint')
  • Avoid using UDP when talking to on-premise resources

    There are a number of app use-cases which require access to resources running on customer's private networks, such as querying an LDAP server for user data. These types of queries are supported on QCA, however only using TCP. This requirement might change in the future, but developers cannot use UDP in these scenarios for now.

Recommended

  • Incorporate graceful shutdowns into your apps

    This is not currently easy to achieve, therefore we are only currently recommending it. Moving forward developers ought to be mindful of the effects of shutting down apps can have on the integrity of their data.

    Currently, applications support /src_deps/cleanup.sh as a means of performing shutdown operations. Anything within this script is executed before the app is shut down. For example take the following Python code that can be included in a flask app:

    import atexit
    from qpylib import qpylib
    
    def shutdown():
        """
        Function registered with atexit to perform
        shutdown operations
        """
        qpylib.log("Performing shutdown operations.")
    
    atexit.register(shutdown)

    This registers an example function that will be invoked when a signal is sent to the app. This signal can currently be sent via the cleanup.sh script:

    supervisorctl stop startflask
  • Avoid using UDP when talking to on-premise resources

    There are a number of app use-cases which require access to resources running on customer's private networks such as querying an LDAP server for user data. These types of queries are supported on QCA, however only using TCP. This restriction might change in the future; however, for now developers must avoid using UDP in these scenarios for now.

  • Be sensible when making API calls

    Traffic outbound from QCA incurs charges. Developers are expected to not export excessive amounts of data to external services, instead minimize or only export required data. This does not currently apply to API calls to the QRadar platform; however, is it possible that your app will fail validation if it is found to make excessive API calls. For example we have seen apps which process requests like so:

    import requests
    
    success = False
     
    while not success:
      response = requests.get('https://some.system/some/endpoint')
      success = (response.status_code == 200) # if some.system is down this will never return True
  • Ensure you have appropriate logging and log levels

    With the release of QCA, we can no longer presume to have backend access to any app. For this reason it is important that the logs produced by the app are meaningful for use in diagnosing issues and debugging. This often a balancing act between providing a meaningful context of what is going on within the app at any given time and avoiding too much unnecessary noise. Storing and retaining logs incurs costs, so we ask developers to be sensible with logging. Important information should be logged in a clear and concise manner, with more verbose messages reserved for the debug log level.

    All errors must be logged with the error log level. We have seen scenarios like the following in the past which should be avoided at all costs as it leads to errors being swallowed:

    from qpylib import qpylib
    
    try:
      some_function()
    except Exception as e:
      qpylib.log("An error occurred: {}".format(e), level='debug') # Incorrect level for errors
    try:
      some_function()
    except Exception as e:
      return # No logging to indicate an error
  • Handle connectivity problems

    It is inevitable that networking issues can occur during normal operations of your application. This is even more a concern with apps that run on QCA. When developing your app, you should be prepared to handle these errors gracefully. A common pitfall seen is that developers presume successful responses parsing data:

    import requests
    from flask import render_template
     
    response = requests.get('https://some.system/some/endpoint')
    data = response.json()
    field = data['field'] # If the request fails this will result in an unhandled error - crashing the page
    return render_template(some-page.html, field=field)

    Instead developers should do their best to handle these cases, and ideally return an HTML page which can render when an error occurs:

    import requests
    from flask import render_template
    from qpylib import qpylib
     
    response = requests.get('https://some.system/some/endpoint')
    if response.status_code == 200:
      data = response.json()
      field = data['field'] # If the request fails this will result in an unhandled error - crashing the page
      return render_template('some-page.html', field=field)
    else:
      qpylib.log('An error occurred while retrieving data. Request to https://some.system/some/endpoint returned {} with text {}'.format(response.status_code, response.text), level='error')
      return render_template('error-page.html', error_response=response)

    Rendering a connection error page can go a long way in improving usability and support for your app.

Storage

Required

  • Ensure stored sensitive information is encrypted

    It is a common requirement for apps to require a token for accessing the QRadar APIs or an external service. This is normally persisted under the /store directory within the application. Under no circumstances should this be stored in plain text.

    The qpylib library comes with support for encrypting and decrypting values. The following is a simple example of how encryption for tokens can be achieved:
     

    from qpylib.encdec import Encryption
     
    # Encryption object used to perform encryption. Takes a name key used to identify the encrypted value and a user parameter
    token_encryptor = Encryption({
      'name': 'api_token',
      'user': 'myappuser'
    })
     
    # Encrypts the value of the API token and stores it for the user at the provided name key
    token_encryptor.encrypt('some-api-token')
     
    # Reference the same name key and user when decrypting
    token_decryptor = Encryption({
      'name': 'api_token',
      'user': 'myappuser'
    })
     
    # API token is read from disk, decrypted and set for use
    api_token = token_decryptor.decrypt()
    print(api_token) # 'some-api-token'
  • Do not store functional data in the store log directory

    All app logs should be written to /store/log. When running in QCA this directory is regularly walked with its contents written to stdout for log collection and truncated. Therefore developers must avoid writing any data, including logs, to this directory and its sub-folders if they later expect to read from it within their app. Functional data can be written to any other sub-directory under /store and will be persisted for later use within your app.

  • Ensure file ownership is set correctly during app initialization

    If data in /store is owned by a user other than root then developers must ensure that file/folder permissions are correctly set during the initialization of the app. Ownership is configured in /src_deps/init/ordering.txt, for example:

    chown -R someuser:somegroup /store/data
    

    If the location is a directory, then ensure you include the recursive option for sub-directory ownership as defined in the example. This is required as we often perform backup and restores of app data in QCA to and from S3 storage where this metadata is lost. If the correct ownership permissions are not set, then the app might fail to function properly for users.

Recommended

  • Only persist to /store when necessary

    The /store directory is there for persistent data and should only be used as such. Each app in QCA uses mounted network-attached storage, reading and writing to this storage will take a performance hit over writing to elsewhere within the app. For anything that does not absolutely need persisted use a directory such as /tmp instead. Conversely always insure that data that should be persisted, such as configuration files, are written to /store. If this is not done then you will lose the data on app restarts.

Resource consumption

This section relates to the resource usage of an app (memory and CPU).

Required

  • Understand your memory requirements

    Developers should have an awareness of how much memory their apps will require while under load. The required memory should be gauged on what you expect when your app is running at its highest expected capacity. Failing to set a suitable memory value will result in crashes within the app. A part of this should be supplying the validation team with the necessary inputs to stress test your app to ensure its requested memory is sufficient.

    You can monitor the resource usage of an app container on your development environment. To perform this first run /opt/qradar/support/recon ps to obtain the ID of your app while it is running:

    ~# /opt/qradar/support/recon ps
    App-ID  Name                Managed Host ID Workload ID     Service Name    AB  Container Name  CDEGH   Port    IJKL
    1001    Example App         53              apps            qapp-1001       ++  qapp-1001       +++++   5000    ++++

    Using this ID you can then identify which container corresponds to your app via docker ps:

    ~# docker ps
    CONTAINER ID        IMAGE                                                                   COMMAND                  CREATED             STATUS              PORTS                     NAMES
    29fb678921d4        console.localdeployment:5000/qapp/1001:1.0.0-20200213115517             "sh /start_container…"   5 days ago          Up 5 days           0.0.0.0:328

    You can then use the container name or container ID to check the resource usage of the running app container via docker stats <identifier>:

    ~# docker stats qapp-1001-hb9aw75x
    CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
    29fb678921d4        qapp-1001-hb9aw75x   0.57%               115.5MiB / 600MiB   19.24%              37.8MB / 463MB      1.3GB / 610MB       21

  • Ensure memory is set in the application manifest

    Understanding the memory requirements of your app should enable you to accurately set the memory field within the resources section of the application manifest. By default this is assigned 200Mb but it is highly unlikely that this value is optimal. Increase or reduce it to a point where you feel the app will not have issues. For QCA apps are assigned resource units which are directly correlated with this value so it is in your best interests to have this set right. Setting an unnecessarily high memory value will increase the resource unit cost to your app and might deter an administrator from installing it for their QCA appliance.

    Memory is a field within resources, which itself is a top-level field of the application manifest. The value should be an integer representing the required memory in megabytes. For example:

    {
      "resources": {
        "memory": 400
      }
    }
  • Understanding CPU usage for your app

    With the release of QCA we are beginning to factor in the CPU consumption of apps. Previously this was not considered due to its implementation within Docker however we are now throttling in real terms the number of CPU cores available to apps on QCA. Currently, apps are receiving by default 0.1 cores when running on QCA. We have been determining internally which apps require more than this and factoring it into the resource unit cost for the app. While there are currently no concrete plans around exposing this directly to developers, it would be beneficial to begin understanding the CPU requirements for your app.

    Again docker stats can be used in this regard. Consider the example below:

    ~# docker stats qapp-1001-hb9aw75x
    CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
    29fb678921d4        qapp-1001-hb9aw75x   0.57%               115.5MiB / 600MiB   19.24%              37.8MB / 463MB      1.3GB / 610MB       21

    The 0.57% in this scenario shows the app is consuming 0.57% of a single core, well within the 0.1 (10%) of the single core we default to. A value of 100% would correspond to 1 core, 200% for 2 cores and so forth.

Dependency management

This section relates to the dependencies within apps such as pip packages and RPMs.

Recommended

  • Always use the pip and rpm directories for dependencies

    Recent work was completed to ensure dependencies included in the src_deps/pip and src_deps/rpms directories are processed at the build time of apps. This significantly reduces the startup burden incurred from installing dependencies at run time, which must be performed each time the app is restarted. Using the src_deps/init folder to install dependencies is discouraged unless it is absolutely necessary. We are pursuing the goal of 'one-click' installs for applications and minimizing boot times is key for this feature. The goal for all developers should be to have your apps ready to serve requests with 30 seconds of starting.

Application manifest

This section relates to various fields within the application manifest.

Required

  • Do not use port 443 in application services

    TCP port 443 is reserved for TLS communications into the app on QCA. For example avoid defining the following in your manifest:

    {
      "services": [{
        "port": 443
      }]
    }

    Attempting to use this port with result in your app failing to install.

  • Do not use of port 33333 in your application services

    TCP port 33333 is reserved for an internal health-check within the QCA framework.  For example avoid defining the following in your manifest:

    {
      "services": [{
        "port": 33333
      }]
    }
  • Ensure service names are unique

    When defining services in your manifest avoid duplicate service names such as the following:

    {
      "services": [{
          "name": "myservice"
        },
        {
          "name": "myservice"
        }
      ]
    }
    Note: Using duplicate names for a service results in an error code from our API if you attempt to install an app that includes identical service names.
  • Ensure service ports are unique

    When defining services in your manifest avoid duplicate ports such as the following:

    {
      "services": [{
          "port": 1234
        },
        {
          "port": 1234
        }
      ]
    }

    Note: Duplicate ports can result in an error code from our API when you attempt to install an app that reuses port numbers.

  • Ensure environment variables are valid

    When adding environment variables to the manifest, developers must ensure they match the regex '^[a-zA-Z][a-zA-Z0-9_.-]*$'.

    Examples of valid environment variables include:

    • EXAMPLE_ENV_VAR
    • EXAMPLE_ENV_VAR_
    • EXAMPLE_ENV_VAR1
    • EXAMPLE.ENV
    • EXAMPLE.ENV.
    • EXAMPLE.ENV.1
    • EXAMPLE-ENV
    • EXAMPLE-ENV-
    • EXAMPLE-ENV-1
    • example_env
    • example.env
    • example-env


    Invalid environment variable formats:

    • 1EXAMPLE_ENV
    • _EXAMPLE_ENV
    • .EXAMPLE_ENV
    • 1example_env
    • _example_env
    • .example_env


    Note: If an environment variable is not valid, an error code is returned from the QRadar API.

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
10 June 2020

UID

ibm16211837