How To
Summary
When a system administrator tries to mount an NFS export to the client filesystem, they receive server not responding error messages as follows:
# mount <SERVER_NAME>:/path/to/export /mnt
mount: 1831-010 server <SERVER_NAME> not responding: RPC: 1832-015 Remote system error - A remote host did not respond within the timeout period.
mount: retrying
Tried to view the NFS exports by running showmount command on the client:
# showmount -e <SERVER_NAME>
showmount: 1831-263 RPC: 1832-008 Timed out
Objective
Determine the root cause for the issue and get it resolved.
Steps
NFS server:
1. Check if rpc.mountd daemon is active:
# lssrc -g nfs
biod nfs 7799284 active
nfsrgyd nfs 8061190 active
nfsd nfs 8847634 active
rpc.mountd nfs 8913180 active
rpc.statd nfs 8257798 active
rpc.lockd nfs 8651082 active
If mountd is not active, we need to start in order to be able to mount successfully. If it's already active move to the next step.
2. Check whether portmap daemon is active:
# lssrc -s portmap
portmap portmap 3670448 active
If active then move to step 1.3.
3. Determine whether the NFS daemons are registered with TCP/IP port numbers:
# rpcinfo -p
program vers proto port service
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
400003 1 udp 32775 mcm
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
200006 1 udp 2049
200006 4 udp 2049
200006 1 tcp 2049
200006 4 tcp 2049
100005 1 tcp 32769 mountd
100005 2 tcp 32769 mountd
100005 3 tcp 32769 mountd
100005 1 udp 32803 mountd
100005 2 udp 32803 mountd
100005 3 udp 32803 mountd
400005 1 udp 32804 mcm
100021 1 udp 32807 nlockmgr
100021 2 udp 32807 nlockmgr
100021 3 udp 32807 nlockmgr
100021 4 udp 32807 nlockmgr
100021 1 tcp 32771 nlockmgr
100021 2 tcp 32771 nlockmgr
100021 3 tcp 32771 nlockmgr
100021 4 tcp 32771 nlockmgr
100024 1 tcp 32772 status
100024 1 udp 32805 status
100133 1 tcp 32772 nsm_addr
100133 1 udp 32805 nsm_addr
200001 1 tcp 32772 pyramid_reserved
200001 1 udp 32805 pyramid_reserved
200001 2 tcp 32772 pyramid_reserved
200001 2 udp 32805 pyramid_reserved
4. Since nfsrgyd daemon is active, that means we are using NFSv4. Check the NFS domain name as follows:
# chnfsdom
Current local domain: nfs-domain.com
The same domain name needs to exist on the NFS client for the mount command to succeed.
NFS Client:
1. Make sure you can ping the NFS server by name or IP. If you can ping the IP address but not the name, then you have a name resolution issue. In case the server is not reachable by name or IP, then it's a network issue. If the server is reachable, then move to the next step.
2. Check the NFS daemons are active:
# lssrc -g nfs
biod nfs 8192000 active
nfsrgyd nfs 7405804 active
Move to the next step if the daemons are active.
3. Check whether the portmap daemon is running:
# lssrc -s portmap
portmap portmap 7667758 active
If active more to the next step.
4. Check the NFS domain name. It needs to be similar to the one on the server:
# chnfsdom
Current local domain: nfs-domain.com
5. Capture an IP trace for the failing mount attempt:
# startsrc -s iptrace -a "-a -b /tmp/iptrace.bin"
# mount <SERVER_NAME>:/path/to/export /mnt
mount: 1831-010 server <SERVER_NAME> not responding: RPC: 1832-015 Remote system error - A remote host did not respond within the timeout period.
mount: retrying
# stopsrc -s iptrace
When the server is reachable but mountd is not for some reason, the IP trace looks like the following:
The client sends a RPC GETPORT call to the server to determine the TCP port for mountd and the server responds it's port number 32769/TCP:
Frame 272: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
Ethernet II, Src: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19), Dst: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43)
Internet Protocol Version 4, Src: 10.154.192.41, Dst: 197.1.91.28
User Datagram Protocol, Src Port: 840, Dst Port: 111
Remote Procedure Call, Type:Call XID:0x5e0484a3
XID: 0x5e0484a3 (1577354403)
Message Type: Call (0)
RPC Version: 2
Program: Portmap (100000)
Program Version: 2
Procedure: GETPORT (3)
[The reply to this request is in frame 273]
Credentials
Verifier
Portmap GETPORT Call MOUNT(100005) Version:3 TCP
[Program Version: 2]
[V2 Procedure: GETPORT (3)]
Program: MOUNT (100005)
Version: 3
Proto: TCP (6)
Port: 0
Frame 273: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43), Dst: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19)
Internet Protocol Version 4, Src: 197.1.91.28, Dst: 10.154.192.41
User Datagram Protocol, Src Port: 111, Dst Port: 840
Remote Procedure Call, Type:Reply XID:0x5e0484a3
XID: 0x5e0484a3 (1577354403)
Message Type: Reply (1)
[Program: Portmap (100000)]
[Program Version: 2]
[Procedure: GETPORT (3)]
Reply State: accepted (0)
[This is a reply to a request in frame 272]
[Time from request: 0.000543900 seconds]
Verifier
Accept State: RPC executed successfully (0)
Portmap GETPORT Reply Port:32769 Port:32769
[Program Version: 2]
[V2 Procedure: GETPORT (3)]
Port: 32769
Then, it sends another RPC GETPORT call to the server to determine the UDP port for mountd and the server responds it's port number
32803/UDP:
Frame 363: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
Ethernet II, Src: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19), Dst: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43)
Internet Protocol Version 4, Src: 10.154.192.41, Dst: 197.1.91.28
User Datagram Protocol, Src Port: 792, Dst Port: 111
Remote Procedure Call, Type:Call XID:0x5fe49549
XID: 0x5fe49549 (1608815945)
Message Type: Call (0)
RPC Version: 2
Program: Portmap (100000)
Program Version: 2
Procedure: GETPORT (3)
[The reply to this request is in frame 364]
Credentials
Verifier
Portmap GETPORT Call MOUNT(100005) Version:1 UDP
[Program Version: 2]
[V2 Procedure: GETPORT (3)]
Program: MOUNT (100005)
Version: 1
Proto: UDP (17)
Port: 0
Frame 364: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43), Dst: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19)
Internet Protocol Version 4, Src: 197.1.91.28, Dst: 10.154.192.41
User Datagram Protocol, Src Port: 111, Dst Port: 792
Remote Procedure Call, Type:Reply XID:0x5fe49549
XID: 0x5fe49549 (1608815945)
Message Type: Reply (1)
[Program: Portmap (100000)]
[Program Version: 2]
[Procedure: GETPORT (3)]
Reply State: accepted (0)
[This is a reply to a request in frame 363]
[Time from request: 0.000581154 seconds]
Verifier
Accept State: RPC executed successfully (0)
Portmap GETPORT Reply Port:32803 Port:32803
[Program Version: 2]
[V2 Procedure: GETPORT (3)]
Port: 32803
Then, the client attempts the mount on the UDP port 32803 but never got a response. The frame got retransmitted two times before the whole process started over. Notice the same call number (XID) in the three frames below:
Frame 365: 82 bytes on wire (656 bits), 82 bytes captured (656 bits)
Ethernet II, Src: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19), Dst: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43)
Internet Protocol Version 4, Src: 10.154.192.41, Dst: 197.1.91.28
User Datagram Protocol, Src Port: 792, Dst Port: 32803
Remote Procedure Call, Type:Call XID:0x5fe49549
XID: 0x5fe49549 (1608815945)
Message Type: Call (0)
RPC Version: 2
Program: MOUNT (100005)
Program Version: 1
Procedure: NULL (0)
Credentials
Verifier
Mount Service
[Program Version: 1]
[V1 Procedure: NULL (0)]
Frame 368: 82 bytes on wire (656 bits), 82 bytes captured (656 bits)
Ethernet II, Src: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19), Dst: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43)
Internet Protocol Version 4, Src: 10.154.192.41, Dst: 197.1.91.28
User Datagram Protocol, Src Port: 792, Dst Port: 32803
Remote Procedure Call, Type:Call XID:0x5fe49549
XID: 0x5fe49549 (1608815945)
Message Type: Call (0)
RPC Version: 2
Program: MOUNT (100005)
Program Version: 1
Procedure: NULL (0)
Duplicate Call/Reply
Duplicate to the call in: 365
Credentials
Verifier
Mount Service
[Program Version: 1]
[V1 Procedure: NULL (0)]
Frame 432: 82 bytes on wire (656 bits), 82 bytes captured (656 bits)
Ethernet II, Src: 16:1c:5d:62:f8:19 (16:1c:5d:62:f8:19), Dst: CheckPoi_6b:07:43 (00:1c:7f:6b:07:43)
Internet Protocol Version 4, Src: 10.154.192.41, Dst: 197.1.91.28
User Datagram Protocol, Src Port: 792, Dst Port: 32803
Remote Procedure Call, Type:Call XID:0x5fe49549
XID: 0x5fe49549 (1608815945)
Message Type: Call (0)
RPC Version: 2
Program: MOUNT (100005)
Program Version: 1
Procedure: NULL (0)
Duplicate Call/Reply
Duplicate to the call in: 365
Credentials
Verifier
Mount Service
[Program Version: 1]
[V1 Procedure: NULL (0)]
This indicates that mountd is unreachable by the NFS client, and hence the errors listed in the beginning of this document. In case of a Network with a Firewall like this situation, we need to contact the network admins to make sure the following ports are allowed on the firewall:
- rcp.mountd TCP and UDP ports as determined by rpcinfo command. It's not a well-known TCP/IP port so it's different on each environment.
- nfsd: ports number 2049/TCP and 2049/UDP.
- portmap: port 111/TCP and 111/UDP.
In case of no firewall, we need to make sure that:
- NFS server is reachable.
- rpc.mountd is active.
- portmap is active.
- NFS domains on both NFS server and client are matching one another.
Document Location
Worldwide
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z0000001fMuAAI","label":"AIX General Support"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]
Was this topic helpful?
Document Information
Modified date:
15 September 2021
UID
ibm16245480