Installing the Agile Service Manager services

The Agile Service Manager application consists of several micro-services, which are provided as containers. You can deploy these on a single server, though a multi-server deployment is required if the network and application discovery services are deployed in a production environment.

Before you begin

Hardware and software requirements are listed in the Planning section.
Important: Agile Service Manager comes with two nasm-common installation packages; one for RHEL 8 and the other for RHEL 9.
RHEL 8
1.1.19-6.el8
RHEL 9
1.1.19-6.el9
The yum installation automatically uses the higher number, so when installing on RHEL 8, you must add a filter to the yum install command to prevent the RHEL9 libraries being installed:
yum install -y --exclude=nasm-common*el9* nasm*
Updating your system:
Prerequisites
If you are updating an existing installation with the latest version of Agile Service Manager, you may already have the prerequisites in place. Before updating an installation, check that you have the correct version of the prerequisites, and that you have applied any relevant upgrade steps documented in the release note upgrade topics.
New Inventory service
From Version 1.1.18, all versions of Agile Service Manager use a new inventory service instead of the previous search and elastic search services.
If you are upgrading from Agile Service Manager Version 1.1.16 (or earlier), the installation establishes a working integrated Agile Service Manager Inventory environment while removing Search (nasm-search) automatically.
Network Discovery upgrade
After an Agile Service Manager upgrade you must configure Network Discovery Services again before running a Network Discovery Observer discovery. See the following topic for more information on Network Discovery configuration: Configuring the network discovery services.
Tip: You can use the get_package_versions.sh script to discover which Agile Service Manager packages are installed.
Remember: You must reinstall the Netcool Hybrid Deployment Option Integration Kit each time you reinstall or upgrade Agile Service Manager.
You must complete the following prerequisites before you can install Agile Service Manager.
  1. Ensure that your operating system has the latest updates applied. You can use either RHEL 8 or RHEL 9.
  2. If possible, disable SELinux before performing the Agile Service Manager installation.
    If you can disable SELinux before installing Agile Service Manager
    Edit the /etc/selinux/config file with a suitable editor and set SELINUX=disabled, before rebooting.
    Install Agile Service Manager as described.
    After you have installed Agile Service Manager, you enable SE Linux.
    If you cannot disable SELinux before installing Agile Service Manager
    Export the following environment variable to override the Agile Service Manager pre-installation check:
    export ASM_SELINUX_INSTALL=true
    Install Agile Service Manager as described in the procedure, with the additional steps in SELinux enabled.
    After you have installed Agile Service Manager, you enable SE Linux.
  3. Obtain the Netcool Agile Service Manager installation image from the Passport Advantage site, and extract it to a temporary directory. More information can be found in the download document here: https://www.ibm.com/support/pages/node/589201
    Note: You need an IBM ID to access the download document.
  4. Ensure you have IBM Dashboard Application Service Hub (DASH) 3.1.3.18 installed, and have configured access.

About this task

When you install the Agile Service Manager components, they are loaded automatically.

You install all the core applications of Agile Service Manager including the UI, but only the observers that you require. In the unlikely event that you wish to install (and start up) all available observers, you must ensure that your system meets the minimum requirements listed here: Hardware requirements Netcool Agile Service Manager Core hardware requirements 

If required, you also install the application and network discovery services. A single-server deployment of Agile Service Manager is only sufficient for a proof-of-concept deployment.
Important: For a production deployment, the application and network discovery services need to be deployed onto separate machines.

Finally, you configure communication between DASH and the Agile Service Manager UI by running the set-dash-config.sh script.

Note: The example data for software versions or directories used here may differ from your own scenario.

Procedure

Prepare the installation files

  1. Move all Agile Service Manager packages to the installation target host.
    • Copy only the observers you intend to deploy to the installation directory, or delete any unwanted observer packages after you have downloaded them. Remember that the Docker Observer is required.
    • Place the Agile Service Manager Base eAssembly and observer packages into the same directory.
      Important: To prevent unwanted observers being installed and thereby placing unnecessary strain on your infrastructure, ensure that this directory contains only the observers you wish to install.
    • For a multi-server production deployment of the application and (or) network discovery services, ensure that only the Network Discovery Observer and the Application Discovery Observer are present in the installation directory.

Install Agile Service Manager core, UI and observers

From the directory where you have placed the packages, install Agile Service Manager using the yum install command.
Non-SELinux installation
Install as directed in the next step.
SELinux disabled
Install as directed in the next step.
SELinux enabled
Before installing the complete system, follow the SELinux installation directions here.
SELinux remains enabled:

With SELinux still enabled (that is, not disabled for the Agile Service Manager installation), you first install only the nasm-common install package, which will result in a failure. You then create a SELinux policy and install nasm-common again, after which you install Agile Service Manager as directed in the next step.

  1. Run the install command for nasm-common only:
    sudo yum install nasm-common.rpm

    The installation process will fail with the following error message: bash: /bin/podman: Permission denied.

  2. You can view the error log for more details:
    grep "SELinux is preventing" /var/log/messages | tail -2
    
    Message example:
    Nov 29 06:35:43 tetrarch1 setroubleshoot[2043]: SELinux is preventing /usr/bin/bash from entrypoint access on the file /usr/bin/podman. For complete SELinux messages run: sealert -l bd7173a4-7bd1-43f7-8df0-7897e625b97e
    
  3. Run the sealert command for instructions on how to create an SELinux policy to allow access, that is:
    allow this access for now by executing:
    # ausearch -c 'bash' --raw | audit2allow -M my-bash
    # semodule -X 300 -i my-bash.pp
    
  4. Create the policy as described.
  5. Remove the nasm-common package:
    yum remove nasm-common
  6. Now that the SELinux policy is in place, install nasm-common once again:
    yum install nasm-common.rpm
    
    It should install successfully.
  7. Proceed to the next step.

  1. Run the install command.
    For a RHEL 8 installation
    yum install -y --exclude=nasm-common*el9* nasm*
    For a RHEL 9 installation
    sudo yum install nasm-*.rpm
    Inventory service: If you are upgrading from Agile Service Manager Version 1.1.16 (or earlier), this update removes Search, but not Elasticsearch.
    Tip: While it is possible to specify each individual installation image, it is recommended that you perform a wildcard (*) installation to ensure that all components are installed. Remember that you must ensure that only the observers you wish to install are present. To install an individual observer later, see the information in the following topic: Adding new observers to an existing on-prem installation
    Troubleshooting: The installation process may generate an excessive number of temporary 'docker-tar*' files in the /var/tmp/ directory, which may result in a warning about your server running out of space. (The docker-tar* files are generated regardless of whether Podman or Docker is used.)
    The workaround is to delete the temporary files from your machine:
    rm -rf /var/tmp/docker-tar*
    Yum will install Podman and all other nasm-* components as required, including all observers found in that directory. During the installation of the packages, the related images are loaded. No data can be retrieved, however, until observer jobs are defined.
  2. Remove the Elasticsearch service and data.
    1. Uninstall the Elasticsearch service.
      sudo yum remove -y nasm-elasticsearch
    2. Remove the Elasticsearch data directory.
      sudo rm -r /opt/ibm/netcool/asm/data/elasticsearch
      
  3. Required: During a first installation or during upgrades, you will be prompted to review and accept the license. You must do so after installation has completed using the following command:
    /opt/ibm/netcool/asm/bin/license-review.sh
    Note: Agile Service Manager can only start after the license has been accepted, and after all additional components have been installed, as described in the following steps.

Install application discovery and network discovery (optional)

Remember:

The deployment of the application discovery and network discovery components are not supported in a FIPS environment.

  1. Optionally, you can install the application discovery and network discovery services, either as part of a single- or a multi-server deployment.
    Single-server deployment
    Only install the application discovery and network discovery services on the same server as the other Agile Service Manager components for a non-production deployment, for example for proof-of-concept or testing purposes.
    From the directory where you placed the application discovery or network discovery packages, install one or both of these services using the yum install command, as in the following examples:
    • Install the application discovery components:
      sudo yum install nasm-db2 nasm-app-disco-discovery nasm-app-disco-primarystorage nasm-appdisco-observer
    • Install network discovery components:
      sudo yum install nasm-net-disco-config nasm-net-disco-consul nasm-net-disco-control nasm-net-disco-sidecar nasm-net-disco-worker nasm-net-disco-schema-registry nasm-net-disco-security nasm-net-disco-status nasm-net-disco-swagger-service nasm-net-disco-swagger-ui nasm-net-disco-topogram nasm-netdisco-observer nasm-net-disco-collector nasm-net-disco-sftp
    Multi-server deployment
    For any production environment, you must install the application discovery and network discovery services on separate servers.
    In the following installation scenarios, Agile Service Manager servers are identified as follows:
    Agile Service Manager core server
    <host-asmcore>
    Application discovery server
    <host-appdisco>
    Network discovery server
    <host-netdisco>
    1. To allow the application and network discovery observers to connect to their remotely deployed services, edit the files that specify the connections between components.
      For application discovery
      Edit the environment variables in the $ASM_HOME/.env file. Add the following host and port details:
      APPDISCO_SERVICE_HOST=<host-appdisco>
      APPDISCO_SERVICE_PORT=<443>
      For network discovery
      Edit the environment variables in the $ASM_HOME/.env file. Add the following host details:
      KAFKA_ROUTABLE_HOSTNAME=host-asmcore
      NETDISCO_SWAGGER_SERVICE_HOST=<host-netdisco>
      NETDISCO_SCHEMA_REGISTRY_HOST=<host-netdisco>
      Edit the KAFKA_SERVER host and port in the etc/nasm-netdisco-observer.yml file. Change the following line:
      KAFKA_SERVER: <host-asmcore:19093>
    2. Install the components for the application and network discovery services:
      For application discovery
      On your application discovery host (<host-appdisco>):
      yum install nasm-db2 nasm-app-disco-discovery nasm-app-disco-primarystorage nasm-nginx nasm-common
      Remove the redundant references to the application discovery services (now installed on a separate server) from the $ASM_HOME/docker-compose.yml file. Run the following bash command from the command line:
      for service in [ "cassandra" "layout" "merge" "inventory" "topology" "ui-api" "kafka" "kafkarest" "zookeeper" "postgres" ]; do
          /opt/ibm/netcool/asm/bin/update-docker-compose.py /opt/ibm/netcool/asm/docker-compose.yml delete $service
      done
      
      For network discovery
      On your network discovery host (<host-netdisco>):
      yum install nasm-nginx nasm-net-disco-collector nasm-net-disco-config nasm-net-disco-consul nasm-net-disco-control nasm-net-disco-sidecar nasm-net-disco-worker nasm-net-disco-schema-registry nasm-net-disco-security nasm-net-disco-sftp nasm-net-disco-status nasm-net-disco-swagger-service nasm-net-disco-swagger-ui nasm-net-disco-topogram nasm-common nasm-elasticsearch nasm-kafka nasm-zookeeper
      Remove the redundant references to the network discovery services (now installed on a separate server) from the $ASM_HOME/docker-compose.yml file. Run the following bash command from the command line:
      for service in [ "cassandra" "layout" "merge" "search" "topology" "ui-api" "kafka" "kafkarest" "zookeeper" ]; do
          /opt/ibm/netcool/asm/bin/update-docker-compose.py /opt/ibm/netcool/asm/docker-compose.yml delete $service
      done
      Remove the following references to Kafka and Zookeeper from the $ASM_HOME/docker-compose.yml file:
      # egrep '(kafka|zookeeper)' $ASM_HOME/docker-compose.yml 
          - kafka
          - kafka
          - zookeeper
          - kafka
      Add the following environment variables to the $ASM_HOME/.env file to set up a connection to the Agile Service Manager core version of Kafka:
      SHERPA_ROUTABLE_HOSTNAME=<host-netdisco>
      KAFKA_EXTERNAL_SERVER=<host-asmcore:19093>
    Note: During the installation of the packages, the related images are loaded. However, the database required by the discovery services is only created and configured when Agile Service Manager is started. If in addition you are deploying the probe and gateway services, you must configure them before starting the services. No data can be retrieved or discovered until you have also defined observer jobs for the discovery services (and any other observers you are deploying).
  2. Required: Again review and accept the license after installation has completed using the following command:
    /opt/ibm/netcool/asm/bin/license-review.sh
    Note: If you do not complete this step and accept the license, Agile Service Manager will not start. You only start Agile Service Manager once you have completed all installation tasks for your optional components (that is, one or more of the probe and gateway, application discovery, or network discovery services).

Optional (required for SE Linux only)

  1. If required, allow SSH for sysadmins:
    sudo setsebool -P ssh_sysadm_login on
    
  2. Enable SELinux.
    SELinux not disabled
    If you installed Agile Service Manager with SELinux not disabled (and completed the double installation of the nasm-common package etc), perform the following steps:
    1. Edit the /etc/selinux/config file with a suitable editor and set SELINUX=enforcing.
    2. Use one of the following to apply your changes:
      fixfiles onboot
      Or
      touch /.autorelabel
    3. Reboot your system.
    SELinux disabled
    If you installed Agile Service Manager with SELinux disabled, perform the following steps:
    1. Edit the /etc/selinux/config file with a suitable editor and set SELINUX=permissive.
    2. Use one of the following to apply your changes:
      fixfiles onboot
      Or
      touch /.autorelabel
    3. Reboot your system.
    4. Check /var/log/messages and /var/log/audit/audit.log for new SELinux messages.
    5. Edit the /etc/selinux/config file with a suitable editor and set SELINUX=enforcing.
    6. Use one of the following to apply your changes:
      fixfiles onboot
      Or
      touch /.autorelabel
    7. Reboot your system.
  3. Provide write access to log and data directories.
    To allow Podman write access to the logs, data and security directories, you must apply the semanage and restorecon SELinux commands after the Agile Service Manager packages have been installed.
    semanage fcontext -a -t container_file_t '/opt/ibm/netcool/asm/logs(/.*)?'
    semanage fcontext -a -t container_file_t '/opt/ibm/netcool/asm/data(/.*)?'
    semanage fcontext -a -t container_file_t '/opt/ibm/netcool/asm/security(/.*)?'
    restorecon -vrF /opt/ibm/netcool/asm/logs/
    restorecon -vrF /opt/ibm/netcool/asm/data/
    restorecon -vrF /opt/ibm/netcool/asm/security/
    
  4. To allow couchDB to start with SELinux, apply the following settings:
    sudo semanage fcontext -a -t container_file_t '/opt/ibm/netcool/asm/etc/couchdb/entrypoint/couchdbEntrypoint.sh'
    sudo restorecon -vrF /opt/ibm/netcool/asm/etc/couchdb/entrypoint/couchdbEntrypoint.sh
    

Required for Docker Observer only

  1. Enable podman access for Docker Observer.
    To grant access to the podman socket file, run the following commands:
    semanage fcontext -a -t container_file_t /var/run/user/984/podman/podman.sock
    restorecon -vrF /var/run/user/984/podman/podman.sock 
    
    Docker Observer SELinux errors: When Docker Observer is run for the first time, errors may occur.
    1. Run the following command:
      grep "SELinux is preventing" /var/log/messages | tail -2
    2. From the log message, apply the suggested SE Linux policy, which will be similar to the following example:
      ausearch -c 'epollEventLoopG' --raw | audit2allow -M my-epollEventLoopG
      
      semodule -i my-epollEventLoopG.pp
      

Configure integration and verify deployment

  1. To configure communication between DASH and the Agile Service Manager UI, run the following script:
    bin/set-dash-config.sh
    You are prompted to enter the following information:
    DASH URL endpoint
    DASH server root URL
    Enter a complete path so that it can be reached from the Agile Service Manager server (including the protocol, host name, and port number).
    Example: https://dash-host.ibm.com:16311
    DASH administrator username
    The DASH administrator user name.
    Example: smadmin
    DASH administrator password
    The password for the DASH administrator user.
    ASM hostname
    Enter the fully-qualified host name of the server on which Agile Service Manager is installed.
    Example: asm-host.ibm.com
    This hostname must match the one entered when the Netcool Hybrid Deployment Option Integration Kit is installed, as documented here.
  2. Optional: You can verify that the images have been loaded running the podman images command as the nasm user.

What to do next

Note: You configure the deployed probe and gateway services after installing the core Agile Service Manager containers (including the probe and gateway containers), but before starting the Agile Service Manager services.
You provide configuration details for the deployed application discovery service after installing the core Agile Service Manager and application discovery containers, and after ensuring that the installed Application Discovery Observer, Application Discovery engines and the provided Db2 database container are running.
You configure the deployed network service after installing the core Agile Service Manager containers, and after starting them.
Inventory service: If you have upgraded from Agile Service Manager Version 1.1.16 (or earlier), you remove any residual Search indices in ElasticSearch after Agile Service Manager has been started and is up and running. This is described here.
Remember: You must reinstall the Netcool Hybrid Deployment Option Integration Kit each time you reinstall or upgrade Agile Service Manager.
Important: Agile Service Manager uses default passwords out of the box. For security reasons, change the default passwords after installation, as described in the Changing default passwords topic.
Network Discovery upgrade requirement: After an Agile Service Manager upgrade you must configure Network Discovery Services again before running a Network Discovery Observer discovery. See the following topic for more information on Network Discovery configuration: Configuring the network discovery services.