IBM Support

Network ports used for NIM communication

Question & Answer


Question

How can I configure my switches, routers, gateways, firewalls, etc., to allow for NIM operations requiring network communication between a NIM master and it's clients, to complete successfully?

Cause

This document is to provide the information necessary to configure any network switches, routers, gateways, firewalls, etc., which handle the network communication between a NIM master and it's client's (and vice-versa).

Answer


What this document does NOT cover:
- In-depth description of NIM service handler or 'nimsh'
- How-to steps for performing network installs
- Problem determination or debugging steps for failed network installs
 

Introduction to NIM service handler or 'nimsh':

AIX uses NIM Service Handler (NIMSH) to eliminate the need for rsh services during NIM client communication. 

NIMSH allows you to query network machines by hostname. NIMSH processes query requests and returns NIM client configuration parameters used for defining hosts within a NIM environment. Using NIMSH, you can define NIM clients without knowing any system or network-specific information. For environments where the standard rsh protocols are not secure enough, nimsh may be implemented. The client daemon has two ports registered with the Internet Assigned Numbers Authority (IANA) for use during network communication. With nimsh, the primary port used by the client is 3901, and it listens for service requests. The primary port on the client is used for stdin and stdout while stderr is redirected to secondary, which is port 3902.
When running the 'niminit' command on the client, the client will use one of the ports in the "rsh" range (ports 1023 - 513) to send packets to the a port in the "rsh" range on the NIM master. 
Although these ports (1023 - 513) must be opened for communication, 'rsh' does not need to be enabled on either the NIM master or client. After the client has been initialized (after running the 'niminit' command on the client), the client will start to use ports 3901 and 3902 to sends packets to the "rsh" port range on the NIM master, but because the the 'nimsh' daemon only runs on the client the NIM master will still use ports 1023 - 513 to communicate with the client's ports 3901 and 3902.

For more information on NIMSH, see IBM Documentation: Using NIMSH

NIM network boot process
The Install Overview gives a short summary of the install process. In this section, we will briefly describe the client/server communication during a NIM install. Both the 'push' and 'pull' installs will be briefly described.

Master Initiated Install - also known as a 'push' install
When a network install is initiated from the master's end, the NIM master prepares resources for install. When the master prepares its resources, it creates script files, NFS export resources (lpp_source, mksysb, etc), populates the /etc/bootptab file, populates the /tftpboot directory, and then executes an 'rsh' command on the client to set the bootlist for install over the network interface. The client resets and attempts to boot over the network using bootp and tftp services.

If the client is not up and running, a 'push' install will give a warning that it was not able to communicate with the client, but the resources will still be prepared and allocated to the client. You must boot the client from SMS manually at this point.
So if the client is not up and running, set the "Initiate reboot and installation now?" option to 'no' in SMIT, or use the '-a boot_client=no' flag if using the 'nim' command to initiate the bos_inst operation.

Client initiated Install - also known as a 'pull' install
When a network install is initiated from the client's end, the NIM master still prepares its resources for the install as it would for a 'push' install. The difference is the fact that during a 'pull' install, the NIM master does not execute an 'rsh' command to set the bootlist of the client and then reset it to boot from the network. The /etc/bootptab file and /tftpboot directory are still populated, along with the script files created, and NFS resources exported.

Clean-up process after a NIM network boot operation
Upon completion of install, the client sends state information to the master via 'nimclient' calls to the master's nimesis daemon. The NIM master then deallocates all install resources from the client object which has completed installing. The deallocation process consists of:

1. Removing files from designated tftp directory
- Remove client.info file
- Remove link to boot image
- Remove file entry in /etc/bootptab

2. Unexporting nfs resources from client
- Remove entries from /etc/exports
- Export remaining entries from /etc/exports

3. Updating client object information in the NIM database
- Machine state -- running -or- not running
- Command result -- success -or- failure
- Client state -- ready for an operation -or- not running

 

Protocols used during a NIM network boot operation
When a NIM network boot operation is performed, there are several protocols involved. The following outline should provide adequate steps for determining network administration tasks:

Install Overview

NIM performs network installs by using a client/server model based on the bootp/tftp protocols for receiving a network boot image. Once the boot image is obtained, the client requests (using tftp) a configuration file (client.info) to determine which server contains the install image and other necessary install resources. The install image and resources are nfs mounted using nfsd/mountd services. Once all mounts succeed, the install begins and subsequent information is sent to the NIM master via nimclient calls to the nimesis daemon (nim).


RSH
stdin/stdout:
'rsh' requires clients to connect using source ports. There is a reserved port range for 'rsh' communication, and the port range for 'rsh' communication is 1023-513. Since NIM clients do not have a client service, the clients communicate by calling 'rcmd()'. This command then calls the 'rreservport()', which will create a TCP socket. When the TCP socket is created, it binds a port from the privileged port range of 1023-513. The port is determined by initializing the starting port (1023), and attempting to bind to it. If the attempt to bind to port 1023 fails, an attempt to bind to the next port, in a descending order (1022, then 1021, etc) will be made until a successful connection to an unused port copy is secured. If no attempts are successful after finally reaching port 513, the process will result in an overall failure.
stderr:
The 'rcmd()' command allows the option of binding a secondary, or auxiliary port for any stderr. When set, rreservport() is called again, but this time the starting port is based on the source port we obtained in the previous step. The same method is used to find the next open port. Source - 1, in a descending order until a successful connection to an unused port copy is secured. The secondary port will listen for any stderr received from the destination service.

NIMSH
The client daemon has two ports registered with the Internet Assigned Numbers Authority (IANA) for use during network communication. These ports are referred to as the primary and secondary ports. The nimsh client daemon listens on these ports for requests initiated by the master using the TCP protocol. The primary port listens for service requests on reserved port 3901. When a request is accepted, the primary port is used for stdin and stdout requests. The stderr requests are redirected to secondary port 3902. This behavior is similar to auxiliary connections in rcmd(). This implementation allows the NIM master connection to stay consistent with current support of client connections through the rsh command. Using a reserved secondary port in NIMSH allows network administrators to write rules for accepting incoming connections on privileged ports from the secondary port. These rules can have the requirement that the originating socket address (hostname : secondary port) comes from a trusted source.
*REMEMBER:
When running the 'niminit' command on the client, the client will use one of the ports in the "rsh" range (ports 1023 - 513) to send packets to the a port in the "rsh" range on the NIM master. 
Although these ports (1023 - 513) must be opened for communication, 'rsh' does not need to be enabled on either the NIM master or client. After the client has been initialized (after running the 'niminit' command on the client), the client will start to use ports 3901 and 3902 to sends packets to the "rsh" port range on the NIM master, but because the the 'nimsh' daemon only runs on the client the NIM master will still use ports 1023 - 513 to communicate with the client's ports 3901 and 3902.

BOOTP
The BOOTP protocol uses two reserved port numbers. 'BOOTP client' uses reserved port 68 (UDP/TCP), and 'BOOTP server' uses reserved port 67 (TCP/UDP). The client sends requests using 'BOOTP server' as the destination port. This request is usually a broadcast request. The bootp server (or NIM master in this case) sends replies using 'BOOTP client' as the destination port. Depending on the kernel or driver facilities in the server, this may or may not be a broadcast. The reason two reserved ports are used, is to avoid 'waking up' and scheduling the BOOTP server daemons when a boot reply must be broadcast to a client.

TFTP
The NIM client contacts the master on port 69 (UDP). Then TFTP protocol uses 'transfer identifiers' (TID's) as ports for communication. This means that the ports used for TFTP must be between 32,768 and 65,535. In order to create a connection, the master and the client both choose a TID for theirselves to be used for the duration of that connection. The TID's chosen for a connection should be randomly chosen, so that the probability that the same number is chosen twice in immediate succession is very low. Every packet has associated with it the two TID's of the ends of the connection, the source TID and the destination TID.
These TID's are handed to the supporting UDP (User Datagram Protocol) as the source and destination ports. A requesting host (NIM client) chooses its source TID randomly as described above, and sends its initial request to the known TID on the serving host (NIM Master, port 69). The master's response to the request uses a TID chosen by the NIM master as its source TID, and the TID chosen previously by the client as its destination TID. The two chosen TID's are then used for the remainder of the transfer.

NFS
The NFS version 2 and version 3 protocol currently uses the UDP port number 2049. This is not an officially assigned port, so later versions of the protocol use the "Portmapping" facility of RPC.

The port mapper program maps RPC (Remote Procedure Call) programs and version numbers to transport-specific port numbers. This program makes dynamic binding of remote programs possible. This is desirable because the range of reserved port numbers is very small and the number of potential remote programs is very large. By running only the port mapper on a reserved port (111), the port numbers of other remote programs can be ascertained by querying the port mapper.

The port mapper also aids in broadcast RPC. A given RPC program will usually have different port number bindings on different machines, so there is no way to directly broadcast to all of these programs. The port mapper, however, does have a fixed port number. So, to broadcast to a given program, the client actually sends its message to the port mapper located at the broadcast address. Each port mapper that picks up the broadcast then calls the local service specified by the client. When the port mapper gets the reply from the local service, it sends the reply back to the client.

In the case of NFS mounts, the client will connect to the port mapper service for the master (port 111), which will provide the port numbers that the client should use to communicate with the mountd, nfsd. lockd, and statd daemons on the master. The nfsd daemon will use port 2049, but mountd, lockd, and statd use an arbitrary port number between 32,768 and 65,535 by default. This behavior can be changed for all of these services, except for statd, by adding the desired port number you wish the service to use in the /etc/services file. For example, to use port 9999 for mountd, you would add the lines:

mountd 9999/tcp
mountd 9999/udp

Then stop and restart the mountd daemon:

# stopsrc -s rpc.mountd
# startsrc -s rpc.mountd

You can check that the port you specified is being used by running:

# rpcinfo -p | grep mountd
100005 1 tcp 9999 mountd
100005 2 tcp 9999 mountd
100005 3 tcp 9999 mountd
100005 1 udp 9999 mountd
100005 2 udp 9999 mountd
100005 3 udp 9999 mountd

*Note that statd cannot be bound to a specfic port, and will always choose an arbitrary port in the 32768 - 65535 range.
 
NFSv4 is slightly more complicated to set up, but it is more port friendly.
If you are using NFSv4, you only need to open your ports for portmap (111 udp/tcp) and nfsd (2049 tcp).

NIM (nim/nimreg)
Clients send the client's status information during a NIM install to the NIM master's well-known service ports 1058 (nim) and 1059 (nimreg). The status information contains details specific to the install progress. This information is updated in the NIM database and actions on the master's end are handled accordingly. This means that when necessary, resources are deallocated (unexported) and boot images are removed.

The clients do not have a registered port for client services, so they use rcmd() to obtain sockets based on the rules mentioned above in the RSH section of this document. The API (Application Programming Interface) is passed to the NIM master's service port 1058 for establishing a connection to the NIM master. The nimesis daemon runs on the NIM master and listens on the NIM service port (1058). When a request for service is received, the nimesis daemon accepts the connection, verifies the originator, and sends an ACK signal as expected by rcmd. Upon a successful connection, state information is passed and commands are placed on the client using the secondary port which has a file descriptor associated with the socket. NIM clients do not have access to the NIM database, so all NIM commands are interpreted on the NIM master's end. Subsequent operations are then placed on the client for shell execution. This detail is important to understand because clients are allowed the option of requesting NIM operations. Since clients have no knowledge of which commands must execute per NIM operation, the requests are always sent to the NIM master's port 1058, and the master responds by pushing the necessary commands on the client machines.

The registration port (1059) is used when clients attempt to add themselves to a current NIM environment. The clients use rcmd() to obtain sockets and pass the service port 1059 for establishing a connection to the NIM master. When connected, clients pass machine configuration information to the NIM master. The NIM database is updated with the newly defined client object and rsh permissions are given to the NIM master.
 

List of ports and directions ports are used for NIM communication

>NOTE: Protocols marked with an "*" are required for "shell" communication
Protocol:    Port(s)
nimsh:                3901 - 3902
rsh*:                   513 - 1023
rlogin*:              513
shell*:                514
bootp:                67 - 68
tftp:                    69 and 32,768 - 65,535
nfs:                     2049
mountd:            32,768 - 65,535 or user's choice
portmapper:    111
NIM (nim/nimreg):     1058 - 1059
NIM operations using "nimsh" as the communication protocol:
Remember that when running the 'niminit' command on the client, the client will use one of the ports in the "rsh" range (ports 1023 - 513) to send packets to the a port in the "rsh" range on the NIM master.
Although these ports must be opened for communication, 'rsh' does not need to be enabled on either the NIM master or client.
After the client has been initialized (after running the 'niminit' command on the client), the client will start to use ports 3901 and 3902 to sends packets to the "rsh" port range on the NIM master, but because the the 'nimsh' daemon only runs on the client the NIM master will still use ports 1023 - 513 to communicate with the client's ports 3901 and 3902.
NOTE: When creating a client definition on the NIM master, the NIM master does not actually communicate with the client. This is why the /etc/niminfo file is not created on the client, and why you must still run the 'niminit' command on the client, even when the client definition has already been created on the master.
Therefore, there are no network rules needed for creating a client definition on the NIM master

Directions to open ports, if you do not want to open the above referenced ports bi-directionally

BOOTP:  (See "BOOTP" section of Protocols used during a NIM network boot operation for details)
Client UDP (Port 68) --> Master UDP (Port 67)
 
Master UDP (Port 67) --> Client UDP (Port 68)

 

TFTP:  (See "TFTP" section of Protocols used during a NIM network boot operation for details)
Exchange of the "/tftpboot/CLIENT_NAME" file - 
Client UDP (random port between 32768 and 65535) --->  Master UDP 69
Master UDP 69 ---> Client UDP (port client sent request from)
Master UDP (random port between 32768 and 65535)  --->  Client UDP (random port between 32768 and 65535)
Client UDP (random port between 32768 and 65535) ---> Master UDP (random port between 32768 and 65535)
Exchange of the "/tftpboot/CLIENT_NAME.info" file - 
Client UDP (random port between 32768 and 65535) --->  Master UDP 69
Master UDP 69 ---> Client UDP (port client sent request from)
Master UDP (random port between 32768 and 65535)  --->  Client UDP (random port between 32768 and 65535)
Client UDP (random port between 32768 and 65535) ---> Master UDP (random port between 32768 and 65535)

*NOTE: The port the client uses to send the requests for the "CLIENT_NAME" and "CLIENT_NAME.info" files cannot be predicted, but those requests will always be sent to port 69 of the NIM master.
The master responds with an acknowledgement from port 69 back to the same port on the client which the request came from. 
Then random UDP ports are chosen on the master to transfer the files ("CLIENT_NAME" and "CLIENT_NAME.info") to the client, and the client's the acknowledgements of the packets containing the requested files.
NIMSH:  (See "NIMSH" section of Protocols used during a NIM network boot operation for details)
When running 'niminit' on the client -
Client TCP (Ports 1023 - 513)  --->  Master TCP (Ports 1023 - 513)
Master TCP (Ports 1023 - 513)  --->  Client TCP (Ports 1023 - 513)
Communication other than  the 'niminit' from the client -  
Client TCP (Ports 3901 and 3902)  --->  Master TCP (Ports 1023 - 513)
Master TCP (Ports 1023 - 513)  --->  Client TCP (Ports 3901 and 3902)
NFS:  (See "NFS" section of Protocols used during a NIM network boot operation for details)
Master UDP (Port 111)  --->  Client UDP (Port 111)
Client UDP (Port 111)  --->  Master UDP (Port 111)
Master TCP (Ports 32768 - 65535, or 'mountd' & 'lockd' port in /etc/services)  ---> Client TCP (Ports 32768 - 65535, or 'mountd' & 'lockd' port in /etc/services)
Client TCP (Ports 32768 - 65535, or 'mountd' & 'lockd' port in /etc/services)  ---> Master TCP (Ports 32768 - 65535, or 'mountd' & 'lockd' port in /etc/services)
Master TCP (Port 2049)  ---> Client TCP (Port 2049)
Client TCP (Port 2049)  ---> Master TCP (Port 2049)

NIM (nim/nimreg):  (See "NIM" section of Protocols used during a NIM network boot operation for details)
When clients attempt to add themselves ('niminit') to a current NIM environment - 
Client TCP  1023-513 --->  Master TCP 1059
Master TCP 1059  --->  Client TCP 1023-513 
    >Once connection is made, while 'niminit' operation is still running -
    Client TCP  1023-513 --->  Master TCP 1023-513 
    Master TCP  1023-513 --->  Client TCP 1023-513
Establishing connection to NIM master (operations other than 'niminit') - 
Client TCP  1023-513 --->  Master TCP 1058 
Master TCP 1058 --->  Client TCP 1023-513 
    >Once connection is made -
    Client TCP  1023-513 --->  Master TCP 1023-513 
    Master TCP  1023-513 --->  Client TCP 1023-513 

Additional Considerations
Additional considerations:

NIM makes use of several protocols which are generally considered "risky" services. It is recommended that users who desire network protection within their NIM environment follow a few rules:

1.) The NFS program usually runs at port 2049 which is outside of the privileged port space. Normally, access to portmapper (port 111) is needed to find which port this service runs on, but since most installations run NFS on this port, hackers can bypass NFS and try this port directly. NFS was designed as a LAN service and contains numerous security vulnerabilities when used over the Internet. NFS services should not be run on firewall machines. If a NIM master resides on a firewall machine, then resources should reside on another client - clients may also be used as resource servers in a NIM environment.

2.) TFTP servers should not be placed on firewall machines since no authentication is needed when requesting service. The TFTP protocol does allow for denying access based on entries contained in /etc/tftpaccess.ctl. NIM manages access to files in /tftpboot only; so all other directory locations should be off limits. When managed properly, TFTP access can be viewed as acceptable in the NIM environment.

3.) In order to reduce the amount of open ports for the NIM "shell" communication to be successful in the NIM environment, the following rules may be applied:

3a.)  For every NIM communication, leave five (5) ports open starting at 1023 and descending from there. So when using 'rsh' as the communication protocol and a client is communicating with the NIM master ports (1023-1019) should be left open for both the master and client. This is an estimate and may not work in all environments since other services may call rreservport() prior to, or during, NIM operations.
3b.)  Users may also add secondary interfaces for each client participating in the NIM environment. The additional interfaces should be packet filtered.
3c.)  When NIM clients no longer participate in the NIM environment, or are temporarily removed from the NIM environment, users should disable rsh services on client machines by removing /.rhosts and/or removing rshd service.

Related Information

NIM Setup Guide

Using nimsh

[{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z000000cvoiAAA","label":"Install"},{"code":"a8m0z000000cvz3AAA","label":"Install-\u003EMigration"},{"code":"a8m0z000000cvz2AAA","label":"Install-\u003ENIM"},{"code":"a8m0z000000cvyjAAA","label":"Install-\u003Emksysb\/backups"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions"}]

Document Information

Modified date:
25 November 2024

UID

isg3T1011808

Page Feedback