SevOne Data Platform Security Guide

SevOne NMS is targeted for use in internal and trusted network deployment. SevOne strongly recommends that you do not expose your cluster on an open internet. SevOne also recommends that you deploy the appliances close to the monitored devices. If you choose to purchase an HSA, it is recommended to be close to its primary appliance.

Exposing SevOne NMS in the open Internet may pose significant security risk to your organization!

SevOne cluster has sophisticated communication schema and relies on multiple required ports. Please refer to SevOne NMS Port Number Requirements Guide for a list of used ports. SevOne has clients that disable part of those ports, but this should be done with caution - only disable ports when you are certain that you do not use the functionality it serves and it does not compromise on security.

SevOne appliances have disabled firewall process by default. For maximum security, SevOne recommends that you enable this and configure for all standard and custom ports that your appliance uses. You can find detailed instructions on how to configure your firewall in the SevOne NMS Installation Guide. Additionally, you can contact SevOne Support to provide further guidance and assistance.

SevOne NMS appliance is delivered with a self-signed SSL certificate as Apache/Nginx will not start otherwise. Using Certificate Authority, you must immediately replace the certificate with the real one. For details, please refer to Generate a Self Signed Certificate or a Certificate Signing Request guide. You may contact SevOne Support for assistance with the installation of the certificate, if needed. Once SSL is enabled, default configuration of Apache/Nginx will accept TLS v1.2 only. However, you may configure it to use TLS v1.3 as well.

By default, Require HTTPS is disabled on the SevOne NMS appliance. You are strongly advised to enable the Require HTTPS option in Cluster Manager > Cluster Settings > Security.

This will force connections to the server via https protocol only and redirect all other attempts to https. It will also enable HTTP Strict Transport Security (HSTS), a mechanism that websites use to declare themselves accessible only via secure connections.

For improved security, SevOne recommends redirecting and rewriting the URLs that attempt to connect to the server via insecure protocol. Please contact SevOne Support to guide you through the process and/or assist in configuring the client appliance, if required.

You will now not be able to connect to the SevOne NMS appliance or the NMS REST API via simple HTTP call.

There is some functionality that requires REST APIs between the peers to communicate with each other. By default, REST API skips to validate the certificates of the other appliances when calling their REST API services. Enable the Rest API Validate by Certificates checkbox from Cluster Manager > Cluster Settings > Security to stop all untrusted calls. An inter-peer request will fail if it is not signed with a certificate from a valid Certificate Authority or if the certificate is not added to the java keystore of the appliances. Please make sure that you have added the SSL certificate in the trusted key for each peer. This can be done via:

1. Add the root certificate to the trust store of every peer (trust anchor /path/to/rootCA.crt)
2. Restart the REST API for each peer (svctl restart SevOne-restapi)

To add root certificate (and chain) to the operating system trust store, execute the following command.

Example

$ SevOne-act add-certificates --file /path/to/cert.crt

SevOne NMS relies on a built-in Role Based Access Control (RBAC) model. SevOne offers full capability to define and inherit User Roles. These roles enable granular control to grant/revoke permissions to access system functionality and/or data.

Users are assigned one or more user roles. There is also a mapping between LDAP groups and NMS roles.

The NMS roles can be granted or revoked permissions to:

• See specific NMS UI pages
• Access the SOAP and/or REST APIs
• Perform specific set of administrative actions in the system
• View and/or manage Device Groups
• View and/or manage concrete Devices
• View and/or manage other Users and User Roles

Generally, users of the system fall into two different categories:

• Need access to the application
• Need access to the Operating System of the SevOne NMS appliance

If you need access to the NMS application then, you can authenticate in one or more of the following ways:

• SevOne Local authentication (by default)
• Using external system (recommended to integrate external system/s). SevOne products currently support integration via LDAP/Active Directory user credentials

SevOne encourages you to configure the system to provide authentication via an external system (LDAP/Active Directory). You can manage servers from Administration > Access Configuration > Authentication Settings. SevOne recommends you to always use secure LDAP with SSL or TLS certificate.

It is a best practice to keep one or more administrative accounts authenticated via SevOne Local method. It will be available if the external authentication systems are unavailable or their connections need to be repaired. By default, the NMS appliance is shipped with one administrative account, admin, for the Web application. 'admin' is the only user that has the role of a System Administrator. Upon initial installation, you are required to change the password for this account.

Local login is recommended for simple deployments and POV projects. This method allows management of user accounts, enforcing complex passwords, keeping password history, forcing users to change passwords for their age, repetition, or any other reason.

For security reasons, along with user credentials, tables containing sensitive information for the devices being monitored along, are encrypted using AES256 encryption.

For security reasons, all credentials in the NMS database are encrypted when stored. This means that every plugin that needs to store authentication credentials, FTP server settings, SFTP passwords, SNMP community strings, etc in the database has to encrypt them on insertion and decrypt them on retrieval using reversible encryption.

Obtaining access to the sensitive database records would mean an attacker has already got access to the machine and the encryption key too. Improvements have been made to the database level. Usage of methods like SevOne::addDevice() or Device::getDevices() are transparent and there should be no change in the way you use these methods (as the encryption is handled by these methods). Everywhere inside the codebase of the NMS you should still be relying on the plain text values of all credentials.

However, if you need to query the database manually, for instance in the C/C++ backend, you should be aware of encrypted values. This affects the following:

Repeated failed login attempts results in account suspension.

The NMS administrator can define whether concurrent sessions by same user are allowed. If not allowed, upon new login by the same user credentials, the older session is terminated. By default, Cluster Manager > Cluster Settings > Login > Allow Concurrent User Sessions is enabled. It is recommended that this field is disabled.

The administrator can monitor and manage active user sessions from Administration > Access Configuration > Session Manager. When a suspicious activity is identified, a user session can be manually suspended by the administrator. . A notification is sent to the user, they are disconnected and redirected to the login screen.

By default, Allow Forcelogin is enabled.SevOne recommends that you should not select the Allow Forcelogin check box from Cluster Manager > Cluster Settings > Security. This will prevent you from accessing SevOne NMS via the forcelogin.php script.

There are several other utilities available (however, not explicitly documented here) from Cluster Manager > Cluster Settings > Security to ensure the security of your cluster and its users. Please refer to SevOne NMS System Administration Guide, section Cluster Manager for additional options.

SevOne NMS also supports generation of audit logs for important domain events, including security relevant administrative and system actions. By default all domain events are enabled and it is recommended that you should not disable them. However, if any domain event needs to be toggled, it can be done from Cluster Manager > Cluster Settings > Logging.

The list of actions to be logged is configurable by the system administrator via the application user interface. The system provides contextual details for the event - synchronized cluster timestamp, hostname / IP, system user that is triggering the event. Depending on the event type, the system logs further details (event specific) about the commands and result of the action.

By default, the logs are collected on the appliance. However, it is recommended that you forward the logs via syslog, optionally TLS enabled, to an external destination, using an external audit server to keep separate and allow robust operations on log files. It is possible for logs to grow too big in size and get harder to operate on. It is unlikely that the size of the log would impact the peer performance as there is a significant buffer. This can be done from Cluster Manager > Cluster Settings > Syslog.

The SevOne cluster has some features that require remote access between peers to be performed by the root user. Some customers have configured the SevOne cluster to prevent root from being a login user and while the majority of the functionality continues to work as intended, some features simply do not operate. SevOne does not recommend disabling it at this time and cannot guarantee proper cluster operations in this case.

The password to the root user is unique for each client - randomly generated when the box is shipped. The client administrators are given the generated password but they are advised to change the password as soon as they receive their appliance(s). To change the password, login as root and execute the command, passwd. Please respond to the prompt to enter the new password.

Login as 'root'

login as: root
Using keyboard-interactive authentication.
Password:

Command to change password

$ passwd
Changing password for user root.
New password:

It is highly recommended to not use the root user for normal login. There is a special admin user that is available for system administrators. The password for this user must be changed immediately after the client receives the appliance (it is forced on first login). Clients must keep this password secure and not share it with anyone outside of their organization - this includes SevOne Support, Expert Labs, or Upgrade teams the user may be working with. To add the admin user as a sudo user, the following steps must be performed.

1. Login as root.
   

   $ ssh root@<NMS appliance>

2. Execute the following command to modify the client's admin account.
   

   $ usermod -aG wheel admin

3. Run the visudo command to edit the sudoers file, which is used by the sudo command. Update secure_path in the Defaults section.
   

   $ visudo
   Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/bin:\
   /usr/local/sbin:/usr/sbin:/usr/local/scripts:/opt/dell/srvadmin/bin:\
   /home/admin/.local/bin:/home/admin/bin

SevOne employees do not have any known credentials to access appliances. If assistance is needed, SevOne can guide you through configuration, diagnostics, etc. If needed, you could provide them temporary access to the appliances by sharing the credentials. After the support case is completed, please change the passwords to keep unique and secret credentials. You may have a dedicated user to use for external access. SevOne recommends that you keep such accounts SSH disabled and only allow when you expect SevOne Support would need access to the machine. Account should be disabled after the case is resolved.

There is one more SSH allowed user in the NMS system - the support user. By default, the password is supportuser. You will be prompted and required to change the support user password. Enter the new password when prompted.

Further details about the users and initial set up, how to change the root, admin, or support user passwords, etc. post the initial set up, please refer to SevOne NMS Installation Guide.

There are standard requirements for password strength for all OS SSH users. Password must be longer than 15 characters, mixed case, special chars, numeric, etc. The passwords are hashed in 41 bytes.

SevOne offers a utility to allow changes in the credentials for the database users. SevOne strongly recommends clients to change their database credentials when they receive the appliance. SevOne Support can assist you with the process.

You have an option Require Strong Passwords for mysql users to enable complex password policy. By default, this option is disabled but it is recommended that you enable it. This is available from Cluster Manager > Cluster Settings > Security. Select the Require Strong Passwords for mysql users check box to enforce the complexity of MySQL user passwords. The minimum length of the MySQL password must be at least 14 symbols long, contain at least one special character +-_@[]:,.%, at least one number, at least one UPPERCASE letter, and at least one lowercase letter. The valid characters are a-z, A-Z, 0-9, +-_@[]:,.%. An example of a valid password: 8s0H43o@7]o%p3. If the current MySQL password does not meet this requirement, it will be change the password to a random, compliant password.

SevOne NMS uses the MySQL database for storing the NMS configuration, NMS user authentication, device configuration and device poll data information. By default, the MySQL root user is set with an empty password. It is recommended that you change the password of the MySQL root user in SevOne NMS.

SevOne-change-mysql-password command is used to update the NMS appliance's MySQL administrative user's username and/or password.

If the appliance is in an HSA pair, this must be run on the active appliance and the passive appliance in this pair will update automatically. However, this command will not automatically update other appliances in the cluster with the same username and password change. This procedure needs to be run on all Active appliances in the cluster individually. The NMS MySQL default user is root.

Example

$ SevOne-change-mysql-password --username "NEW_USERNAME" --password "NEW_PASSWORD"

--username (Required) Rename the current username to this
--password (Required) Change the password to this value

The following are valid characters:
a-z, A-Z, 0-9, +-_@[]:,.%

To run this procedure for all Active Appliances in the cluster as a single command, first set the password as a shell variable and then execute the command as shown below. This command must be executed from the Cluster Leader Active appliance.

$ MYPASSWORD='Welcome@123456789';

$ for IP in $(mysqlconfig -BNe 'select ip from peers;'); \
do echo "Executing on $IP"; \
ssh $IP "SevOne-change-mysql-password \
--username root --password $MYPASSWORD"; done

Identify your current NMS identity and IP.

$ SevOne-peer-whoami
=== My Peer info:
--- ID: 1
--- Name: pandora-01
--- IP: 10.129.14.168

Using the MySQL client, connect to the IP address returned above (for example, 10.129.14.168) from any appliance in the cluster. You will be prompted for the password. Enter the new password as set in the example (Welcome@123456789). If you used a username other than root, you can change to the correct username as shown in the command below. Successful completion of the command will bring you to the MySQL prompt. You should be logged in and be able to execute the MySQL queries.

$ mysql -h 10.129.14.168 -P 3307 net -u root -p

Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6151
Server version: 5.6.40-enterprise-commercial-advanced-log MySQL Enterprise Server - Advanced Edition
(Commercial)
Copyright (c) 2000, 2022, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

mysql> exit
Bye

If there is an error with the password or it has not updated correctly, then you may get an error as shown below - you may need to run the procedure all over.

$ mysql -h 10.129.14.168 -P 3307 net -u root -p

Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'pandora-01' (using password: YES)

SevOne does not store or operate on any personal or confidential data for the users. SevOne only stores authentication data. All data used for authentication of users or integration with other systems is considered sensitive and is kept secured. The Cluster configuration and Device information are considered confidential and are also encrypted at rest. There are two types of encryption that SevOne uses to persist data:

1. Passwords used to verify user identity - The application stores this data as a one-way hash and cannot decrypt it. All the passwords for local application users are stored in this manner.
   Important: The password hashing algorithm is strengthened by eliminating the external salt generation using the PASSWORD_BCRYPT algorithm, and increasing the cost factor to 14.
2. Information that the application uses to connect to outside systems. Example: Credentials used for device access, SFTP passwords, LDAP bind account/password to sync the groups, etc. The application needs to be able to decrypt this information to perform its function. SevOne stores this data encrypted by using the stronger encryption technique in MySQL (AES256). For additional details, please see section Credential Encryption in this document.

SevOne strongly recommends that the NMS system is not deployed on the open Internet and access is limited to authorized personnel only.

Data replication within the cluster is encrypted for both cluster configuration and high availability solution. During normal cluster operations, data is replicated from Cluster Leader Config to all peers Config. Additionally, each primary appliance replicates all its data to its secondary appliance (i.e., HSA). Also, there is authentication and encryption for all remote connections to the MySQL database for each peer. Only its local processes can access the database without encryption.

• As a best practice, SevOne recommends to allowlist peers by restricting access in the cluster so that only peers can contact other peers. Since the appliance can be deployed where members may have a NAT firewall separating members, this is not configured by default.
• For more security concerned clients, SevOne provides deployments where all traffic between the peers is encrypted using OpenVPN, an SSL/TLS-based VPN solution. However, this has performance impact on system operations.

SevOne is continuously decreasing the level of permissions needed to execute various commands and implementing the principles for least privileges.

FIPS

• key exchange algorithm: diffie-hellman-group14-sha1
• host key algorithm: ecdsa-sha2-nistp256
• server->client cipher: aes128-cbc MAC: hmac-sha1 compression: none
• client->server cipher: aes128-cbc MAC: hmac-sha1 compression: none

SevOne-requestd

ZeroMQ CURVE encryption - Curve25519 elliptic curve encryption

MySQL

TLS 1.2 with rsa encryption (2048-bit public key), sha256 signature algorithm

Kafka

TLS 1.2 with rsa encryption (2048-bit public key), sha256 signature algorithm, system-default TLS 1.2 cipher

REST API Live Maps

peers <-> Cluster Leader:443 (nginx will use its certificate configured at /secrets/nginx/nginx.crt), system-default TLS 1.2 cipher

SevOne strongly recommends that you configure secure polling via SNMPv3 for all devices, where applicable. SevOne supports authentication (MD5, SHA, SHA-224, SHA-256, SHA-384, and SHA-512) and encryption (AES, AES192, AES192C, AES256, AES256C, 3DES, and DES) over SNMPv3. This is done by providing username, password and encryption key. Please refer to SNMP Quick Start Guide and/or section SNMP Plugin in SevOne NMS User Guide for details.

SevOne NMS 6.8 release has migrated the operating system from CentOS 8 Stream to Red Hat Enterprise Linux (RHEL) 8.9.

SevOne's new deployments are not vulnerable to the dirty cow (CVE-2016-5195) attack. For maximum security, SevOne strongly advices our older clients to manually install and select the standalone package with the new kernel (4.9.X) that SevOne provides. For additional information and assistance, please contact SevOne Support.

Some of the processes on the NMS appliance are running under their own user for separation of concerns and minimal permissions. Example: Apache/Nginx, MySQL, REST API, Redis, Kafka, etc. There are other processes that are still running as root user with full permissions. Where there are security concerns, SevOne is making an effort to convert such services to their own restricted users and groups and restrict their permissions and eliminate the use of the root user completely.

Some SevOne customers manually configure SevOne-daemons to run as non-root user. This change however, makes the appliances difficult to upgrade and maintain. SevOne does not recommend this configuration. For more details, please contact SevOne Support.

Each peer in a SevOne cluster can be peered with a Hot Standby Appliance (HSA) to provide real time backup and fault tolerance. In this configuration, the primary appliance will replicate all data it contains (configuration and polling) to the HSA. The HSA is then able to take over all duties of the primary appliance if it should happen to fail (due to hardware failure, power outage, loss of network connectivity, etc.). This includes polling, baselining, reporting, and also administrative functions of the cluster leader. HSA will turn on as soon as there is no connection to the primary PAS for more than x minutes (x = 10 minutes / 600 seconds by default, but this field is configurable).

The return from HSA back to the PAS is a manual activity as it requires merging the data of the two appliances.

HSA is just a precaution. SevOne have clients with HSA that have never been used. They have had years of stable operation.

Due to backwards compatibility with legacy reports, by default, custom code in Simple reporting attachments is allowed. This means that any user who can edit a report, can potentially inject harmful code that would be executed when any other user views the report. In order to prevent this, Allow insecure code in Simple attachment check box from Cluster Manager > Cluster Settings > Security is available for SevOne NMS administrators to allow/disallow usage of custom code in the Simple attachments. By default, Allow insecure code in Simple attachment check box is enabled but SevOne recommends that this option should be disabled.

By default, Force Same Origin Policy check box is enabled and it is recommended that this option should not be disabled. With the Force Same Origin Policy check box enabled, it prevents SevOne NMS from being loaded outside of the current domain. This includes portals and the use of the force login script to load SevOne NMS into an iframe from where a malicious user could log your activity. Note: If you clear this check box, the application security is lowered in a way that can prevent SevOne NMS from passing specific security scans. By selecting this check box, it enforces the Same Origin Policy (SOP) or allows you to configure a small list of allowed origins. Also, it will reduce attack surface and prevent CSRF attacks.

After a user is deleted, there may be traces of their id or username for auditing purposes to keep track of specific actions which are critical for the correct operation of the system. For security reasons, this information is saved for a pre-defined period of time. The retention period is configurable using the syslog configuration and depends on the client's organizational policy. At the end of the retention period or when the log exceeds 100MB, the log is archived. There is a configurable age-out period which is set to one year by default. At the end of the age-out period, the logs are deleted.

The logs are used to ensure proper operation of SevOne NMS and to continue servicing other SevOne NMS users.

SevOne recommends that when a user decides to leave the system, the username and id are insufficient to uniquely identify a natural physical person. In cases where this is not possible, users must be notified that their identifier username would be kept a bit longer for auditing purposes. The exact policy and timeline for deleting this data must be shared.

In the scenario when the information is backed up to another location by the NMS administrator, there must be a policy in place to ensure that the user data is deleted immediately after the retention period has passed. Even when the logs are exported to an external system or backup server for further analysis, there must be a clear expiration policy in place.

A static IP address can serve as an online identifier and is considered personal data when additional information such as real name, phone number, email address, etc. are also stored along with it. At this time, there is no precedent for enforcing the GDPR policy however, there are generally accepted best practices.

• The dynamic IP address must not be considered personal data. Even if it is assumed that the ISP has a record of the user for this IP address, certain legal criteria must be satisfied prior to sharing the data. The client company must not have a way to uniquely identify the user.
• The static IP address is more likely, but not 100%, guaranteed to allow identification of a natural person and can point to a device (physical or virtual). In combination with other details such as the applications used, usernames, etc., it can point to a concrete person with a satisfactory confidence level.

The data that SevOne NMS collects is not enough on its own to identify a user and must not be considered personal data. SevOne recommends that the clients must make sure that there is no other data in their organization that in combination with SevOne data, may be enough to identify a person. If the client has accounting or other data for their network users, this information must be kept separate from SevOne NMS data. At no point must this data be combined. There must be no users that have access to both products and allowed to make combined reports.

• Log into SevOne NMS > from the navigation bar, go to SevOne NMS > Administration > Access Configuration > User Manager.
• Or, you may directly use SevOne NMS' REST API endpoints.

SevOne NMS 5.5.0.1 is officially Common Criteria certified. This serves to prove the trustworthiness of SevOne product and our commitment to keeping our client's appliances safe. The NMS certification report, protection profile, and security target are publicly available on the Common Criteria portal (https://www.commoncriteriaportal.org/products/#NS).

The appliances can be configured to operate to the specifications of Security Technical Implementation Guide (STIG) but, it reduces functionality. This is not recommended configuration. For additional details, please contact SevOne Support.

SevOne provides optional configuration files for web servers (Apache or Ngnix depending on your SevOne NMS version) and OpenSSH. When selected, the application works in a FIPS 140-2 mode. This disables the usage of less secure ciphers that have proven, in practice, easier to overcome than more secure ciphers. The more secure ciphers need to be supported by the SSH clients you use.

1. Using ssh, login to NMS appliance as root.

   $ ssh root@<NMS appliance>

2. Change directory to /config/ssh.

   $ cd /config/ssh

3. Enter into the NMS container.

   
   $ podman exec -it nms-nms-nms /bin/bash
   

4. Make a copy of the sshd_config file.

   $ cp sshd_config sshd_config.custom

5. Using the text editor of your choice, edit the new custom (sshd_config.custom) file and delete the offending SSH ciphers from the lines that start with the words Ciphers and MACs. Make sure you only delete the Ciphers and MACs that are necessary to be removed.
6. Save and close the file.
7. Verify SevOne-select shows the custom file.

   
   $ SevOne-select sshd config

8. Symlink the custom config to sshd_config.

   $ ln -sf sshd_config.custom sshd_config

9. Switch SevOne-select to use the new custom config.

   
   $ SevOne-select sshd config custom

10. Verify that it is selected correctly with SevOne-select sshd config.
11. Restart sshd.

    
    $ supervisorctl restart sshd

No. Industry best practices is NOT to publish scan results because it puts all customers at further risk by informing hackers where to focus their efforts.

IBM SevOne does use a series of commercially available tools to test our products in house to identify risks. IBM SevOne does not endorse the tools used but, uses them as a guide to perform due diligence.

IBM SevOne strongly encourages all customers to follow additional industry best practices and perform periodic scans against products deployed on their network. This to cover any misconfiguration when deployed. If upon scanning IBM SevOne there are any concerns, the customer should report them by opening a ticket with Support.

IBM does takes security and privacy very seriously.   For additional information on IBM's process, please refer to https://www.ibm.com/trust/security-spbd.

The MySQL database is not fully encrypted. The cluster and device information, passwords, etc. are encrypted at rest. However, time-series data, for example, is not.

It is AES256.

MySQL database versions on SevOne Data Platform are updated only when there is substantial security update or a performance defect fix is made available.

SevOne Data Platform does not support encryption for the entire filesystem at rest.

SevOne Data Platform supports TLS v1.2 and v1.3.

Yes, the non-default config can be changed to enable TLS v1.3.