Automated Build / Rebuild (Customer) Instructions

About

This document provides customer instructions for building / rebuilding a PAS or DNC, which can then be incorporated into a new or existing cluster or can be used to replace a lost node. It will NOT restore any lost data.

Warning: This process can help recover a lost cluster node, but it will not recover any lost data.

Important: This guide is for hardware deployments only.

Important: Starting SevOne NMS 6.7.0, MySQL has moved to MariaDB 10.6.12.

Prerequisites / Assumptions

Edit online

Customer has obtained or created on boot media (USB, DVD, etc.) with the required NMS recovery installation files. NOTE: For further details, see the appendix on boot media.
Customer must have physical access to the console of the target SevOne PAS hardware and a minimum USB stick size of 32GB.
Customer knows the required network settings to be installed:
- Hostname
- The network interface which should be used (e.g. eth0)
- IP Address, in dotted decimal notation (e.g. 10.10.10.30)
- Netmask, in dotted decimal notation (e.g. 255.255.255.0)
- Broadcast address, in dotted decimal notation (e.g. 10.10.10.255)
- Default Gateway, in dotted decimal notation (e.g. 10.10.10.1)
Customer knows the submodel capacity of the NMS installation. (e.g. 100K.)

Installation

Edit online

Important: SevOne NMS 7.0 ISO uses a standard Red Hat Enterprise Linux (RHEL) Anaconda Installer. By default, this setup only allocates 70 GB to the root (/) partition.
After the deployment, the filesystem can easily be expanded by using all available disk space.

In the simplest case where no /data drive is created and you want to allocate all available space to /, execute the following commands.


$ lvextend -l +100%FREE /dev/mapper/rhel-root

$ xfs_growfs /

Since /data is not its own drive or mount but a directory on the root filesystem, the space allocated this way will be available for /data. The only disadvantage is that the /data directory will not be a separate volume that can easily be isolated from the rest of the filesystem and moved/extended.

If you wish to create a separate /data lvm partition, or attach an additional disk, you may do so.

using ISO

Edit online

Insert boot media (or verify that SD card is present) with recovery installation files. Please refer to section appendix for boot media details.
Reset power on the server.
When the option appears, press F11 to enter Boot Manager.
In Boot Manager Main Menu, select One-shot BIOS Boot Menu.
In Boot Menu, find and select the boot media (USB, DVD or SD card).

using ISO with iDRAC

Edit online

Important: If the connection from the iDRAC to the hardware appliance is poor, loading files on the virtual media may suffer. For this case, a direct connection to the appliance is recommended.

The latest ISO file can be downloaded from IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact SevOne Support Team for the file.
Logon to iDRAC.
Install Red Hat Enterprise Linux 8.10.
Under SYSTEM, click Installation Destination.
Select the device(s) you want to install to. For example, DELL PERC H730P.
Click button.
On INSTALLATION OPTIONS screen, click button.
Click button to reclaim the disk space.
Important: Removing the file system will permanently delete all of the data it contains.
You are now ready to begin the Red Hat Enterprise Linux 8.10 installation.
Click button.
Important: The installation process takes about 45 - 50 minutes.
When done, click button in

to reboot your system.
On successful completion, use nmtui for advanced network configuration. Please refer to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_ip_networking_with_nmtui for details.
Time and Date can be configured using timedatectl command. Please refer to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-configuring_the_date_and_time for details.
To configure NTP/Chrony, please refer to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-configuring_ntp_using_the_chrony_suite/ for details.
SNMP Settings can be performed manually but updating /config/snmp/snmpd.d/90-custom.conf file.

Note: Once you have gained access to the appliance and the appliance is part of the cluster, use SevOne-change-ip scripts to change the configuration of the appliance.

OS-level User Names and Passwords

Edit online

You will need to change the default password for the support user account. This is important for security reasons.

Failure to change the default password presents a significant security risk. This publication includes the default password and is available to the public.

Change Support Password

Edit online

The support password should be changed on all peers. Perform the following steps to change the password for support.

SSH into your SevOne NMS appliance and log in as support.

At the Password prompt, enter supportuser.

login as: support
Using keyboard-interactive authentication.
Password:
Using keyboard-interactive authentication.
You are required to change your password immediately (root enforced)
Changing password for support.
(current) UNIX password:

You will be prompted and required to change the support user password. Enter a new password at the prompt.
Exit the session when you are done.

Enable Firewall

Edit online

On appliances that are running RHEL, the firewall will be disabled. This applies to both appliances with new SevOne NMS installs and appliances that were previously running CentOS, then moved to RHEL. Perform the following steps to enable the firewall.

SSH into the appliance as root.
Execute the following command to enable firewalld.
```
$ /usr/bin/systemctl enable firewalld
```
Execute the following command to start firewalld.
```
$ /usr/bin/systemctl start firewalld
```

Execute the following command to identify custom ports.

$ netstat -tuplet
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User
Inode PID/Program name
tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN root
515537 70512/sshd
tcp 0 0 localhost:mmcc 0.0.0.0:* LISTEN root
982646 206874/SevOne-maste
tcp 0 0 0.0.0.0:https 0.0.0.0:* LISTEN root
909604 194384/nginx: maste
tcp 0 0 localhost:pharos 0.0.0.0:* LISTEN root
903924 193160/messageswitc
tcp 0 0 localhost:ita-agent 0.0.0.0:* LISTEN root
890957 189990/SevOne-datad
tcp 0 0 localhost:60123 0.0.0.0:* LISTEN root
890899 189990/SevOne-datad
tcp 0 0 localhost:60124 0.0.0.0:* LISTEN root
896321 191424/SevOne-ips-c
tcp 0 0 localhost:60126 0.0.0.0:* LISTEN root
898164 191872/SevOne-ocd
tcp 0 0 0.0.0.0:snmp 0.0.0.0:* LISTEN root
913422 194796/snmpd
tcp 0 0 localhos:xmltec-xmlmail 0.0.0.0:* LISTEN prometheus
905473 193521/pushgateway
tcp 0 0 localhost:smux 0.0.0.0:* LISTEN root
913425 194796/snmpd
tcp 0 0 localhost:9095 0.0.0.0:* LISTEN root
897640 191872/SevOne-ocd
tcp 0 0 localhost:cslistener 0.0.0.0:* LISTEN root
927427 197405/php-fpm: mas
tcp 0 0 localhost:ircu-2 0.0.0.0:* LISTEN ingestion
895678 191375/SevOne-inges
tcp 0 0 localhost:redis 0.0.0.0:* LISTEN redis
904778 193510/redis-server
tcp 0 0 localhost:jetdirect 0.0.0.0:* LISTEN prometheus
905217 193398/node_exporte
tcp 0 0 0.0.0.0:sunrpc 0.0.0.0:* LISTEN root
15313 1/systemd
tcp 0 0 0.0.0.0:http 0.0.0.0:* LISTEN root
909602 194384/nginx: maste
tcp 0 0 localhost:sgi-esphttp 0.0.0.0:* LISTEN root
564775 75973/soa
tcp 0 0 localhos:personal-agent 0.0.0.0:* LISTEN root
564779 75973/soa
tcp6 0 0 [::]:33717 [::]:* LISTEN kafka
922030 196146/java
tcp6 0 0 [::]:ssh [::]:* LISTEN root
515539 70512/sshd
tcp6 0 0 [::]:https [::]:* LISTEN root
909605 194384/nginx: maste
tcp6 0 0 [::]:8123 [::]:* LISTEN root
31856 1592/(squid-1)
tcp6 0 0 [::]:snmp [::]:* LISTEN root
913423 194796/snmpd
tcp6 0 0 [::]:XmlIpcRegSvc [::]:* LISTEN kafka
928765 196146/java
tcp6 0 0 localhost:eforward [::]:* LISTEN kafka
921865 195873/java
tcp6 0 0 [::]:60007 [::]:* LISTEN root
899670 192260/SevOne-reque
tcp6 0 0 [::]:mysql [::]:* LISTEN mysql
315413 34172/mysqld
tcp6 0 0 [::]:opsession-prxy [::]:* LISTEN mysql
314375 33715/mysqld
tcp6 0 0 [::]:sunrpc [::]:* LISTEN root
15315 1/systemd
tcp6 0 0 [::]:webcache [::]:* LISTEN restapi
951715 194898/java
tcp6 0 0 [::]:http [::]:* LISTEN root
909603 194384/nginx: maste
tcp6 0 0 [::]:11347 [::]:* LISTEN kafka
921342 195873/java
udp 0 0 0.0.0.0:52518 0.0.0.0:* squid
31855 1592/(squid-1)
udp 0 0 localhost:40676 0.0.0.0:* root
913414 194796/snmpd
udp 0 0 0.0.0.0:bootpc 0.0.0.0:* root
26792 1356/dhclient
udp 0 0 0.0.0.0:sunrpc 0.0.0.0:* root
15314 1/systemd
udp 0 0 0.0.0.0:ntp 0.0.0.0:* root
25007 1215/chronyd
udp 2176 0 0.0.0.0:snmp 0.0.0.0:* root
913415 194796/snmpd
udp 0 0 0.0.0.0:snmptrap 0.0.0.0:* root
902773 193126/SevOne-trapd
udp 0 0 localhost:323 0.0.0.0:* root
25005 1215/chronyd
udp 0 0 localhost:syslog 0.0.0.0:* root
912596 194713/syslog-ng
udp 0 0 0.0.0.0:17860 0.0.0.0:* root
902769 193126/SevOne-trapd
udp 0 0 0.0.0.0:palace-5 0.0.0.0:* root
898156 191805/SevOne-netfl
udp6 0 0 [::]:sflow [::]:* root
916463 195266/sflowtool
udp6 0 0 [::]:sunrpc [::]:* root
15316 1/systemd
udp6 0 0 [::]:ntp [::]:* root
25017 1215/chronyd
udp6 0 0 [::]:snmp [::]:* root
913421 194796/snmpd
udp6 0 0 [::]:snmptrap [::]:* root
902774 193126/SevOne-trapd
udp6 0 0 localhost:323 [::]:* root
25006 1215/chronyd
udp6 0 0 [::]:50222 [::]:* squid
31854 1592/(squid-1)
udp6 0 0 [::]:13640 [::]:* restapi
915760 194898/java
udp6 0 0 [::]:palace-5 [::]:* root
898157 191805/SevOne-netfl
udp6 0 0 [::]:38979 [::]:* restapi
915759 194898/java

Execute the following command to add a custom port to the firewall. Replace <portNumber> with the number of the port you want to add. Replace <connectionType> with tcp or udp.
```
$ firewall-cmd --zone=public --add-port=<portNumber>/<connectionType> --permanent
```
Execute the following command to reload the firewall and apply changes.
```
$ firewall-cmd --reload
```
Exit the session when you are done.

Shut Down and Reboot SevOne NMS

Edit online

SevOne NMS can run for extended periods of time. Occasionally it is necessary to shutdown or reboot an appliance. SevOne NMS stores data in cache and writes to the disk on a regular basis. The following shell commands back up the memory ring tables to the database on the disk to ensure that you do not lose data.

To shutdown SevOne NMS, you must run the command from the NMS container and not the host.


$ podman exec -it nms-nms-nms /bin/bash

$ SevOne-shutdown shutdown

To reboot SevOne NMS, you must run the command from the NMS container and not the host.


$ podman exec -it nms-nms-nms /bin/bash

$ SevOne-shutdown reboot

Appendix - Obtain/Create Boot Media

Edit online

In order to perform a build / rebuild installation, you must first have bootable media with the correct NMS recovery installation files.

If your NMS hardware shipped during or since August 2016, it should have included an inserted SD card for this purpose. If this is the case, and if you have not upgraded to a more recent version of NMS, you can skip the rest of this appendix and start the Installation Steps using that SD card as your bootable media.

Otherwise, you must download a recovery installation ISO and use it to build boot media.

Contact SevOne Support and request a download link.
When you receive your download link, click it and download the ISO file. (NOTE: Download links expire and must be used within 24 hours of sending.)
Create bootable media from this ISO. (Please see the grid below for instructions by media type and OS)

Important: Example: ISO version

RHEL_NMS_v7.0.0.iso - if x = 0 and y = 0, version is for SevOne NMS 7.0.0 release. Please obtain the latest file from IBM Passport Advantage via Passport Advantage Online or SevOne Support Team.

Medium	OS		How to create bootable from ISO...
USB	Windows		Download and run the free utility "Etcher". https://etcher.io/ Click on Select image and choose the downloaded RHEL_NMS_v<7.x.y version>.iso Click on Flash! When done, you will get a message Flash Complete! Warning: Make sure you select the right device from the device drop down. If you select the wrong device, you can destroy data on other plugged in USB drives!
	Mac	Command Line	diskutil and dd diskutil `$ diskutil list /dev/disk0 (internal, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme 121.3 GB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_HFS Macintosh HD 120.5 GB disk0s2 3: Apple_Boot Recovery HD 650.0 MB disk0s3 /dev/disk1 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: NONAME 8.0 GB disk1 $ diskutil unmountDisk /dev/disk1 Unmount of all volumes on disk1 was successful` Warning: Do not destroy your OS! Make sure you get the right device from diskutil list. If youenter the wrong device you can destroy your Operating System(macOS)! dd `$ sudo dd if=RHEL_NMS_v<7.x.y version>.iso of=/dev/rdisk[DISKNUMBER]bs=1m &` Raw paths! If your disk is located at /dev/disk2, make sure you use of=/dev/rdisk2. /dev/disk# is buffered, requiring extra processing when writing (extra validations) /dev/rdisk# is raw, which allows for much faster writing (this is okay when doing a raw block-level copy using dd). The difference is very noticeable. Testing with a linux iso gave over 5x performance improvement: `sevoneuser@mac$ sudo dd if=RHEL_NMS_v<7.x.y version>.iso of=/dev/disk1 bs=2m 421+1 records in 421+1 records out 883949568 bytes transferred in 781.193450 secs (1131537 bytes/sec)` `sevoneuser@mac$ sudo dd if=RHEL_NMS_v<7.x.y version>.iso of=/dev/rdisk1 bs=2m 421+1 records in 421+1 records out 883949568 bytes transferred in 144.064811 secs (6135777 bytes/sec)`
	User Interface		Download and run the free utility "Etcher". https://etcher.io/ Click on Select image and choose the downloaded RHEL_NMS_v<7.x.y version>.iso. Click on Flash! When done, you will get a message Flash Complete! Warning: Make sure you select the right device from the device drop down. If you select the wrong device, you can destroy data on other plugged in USB drives!
	Linux		Use Startup Disk Creator, Disks, Disk Utility, or dd Warning: Make sure you get the right device from dmesg. If you enter the wrong device you can destroy your Operating System (OS)! dd `dd if=RHEL_NMS_v<7.x.y version>.iso of=/dev/YourUsbDevice & while $(kill -USR1 $!);do sleep 1;done`
DVD	Windows		Right click RHEL_NMS_v<7.x.y version>.iso -> Burn Disc Image OR Right click RHEL_NMS_v<7.x.y version>.iso -> Open With... -> Windows Disc Image Burner
	Mac		Select the ISO file in Finder, and choose “Burn Disk Image RHEL_NMS_v<7.x.y version>.iso to Disc...” from the File menu.
	Linux		Right-click the ISO file and use Brasero or any other DVD writing utility.