Topic
2 replies Latest Post - ‏2013-04-24T09:27:11Z by DmitryS.
SystemAdmin
SystemAdmin
2092 Posts
ACCEPTED ANSWER

Pinned topic Processes of SMB hangs in deadlock

‏2013-04-04T12:49:53Z |
Hi,

i have strange situation with processes of smb which wait something.

version gpfs 3.4.0.19
samba 3.5.21

with smb.conf

  1. Global parameters
global
workgroup = PixelBears
netbios name = NAS
server string = Samba Server %v
map to guest = Bad User
log file = /var/log/samba/log.%m
max log size = 50
preferred master = No
local master = No
dns proxy = No
security = user
###GPFS
fileid:mapping = fsname
vfs objects = gpfs fileid
gpfs:sharemodes = No
force unknown acl user = yes
nfs4: mode = special
nfs4: chown = yes
nfs4: acedup = merge
#gpfs:winattr=yes
#store dos attributes=yes
gpfs:getrealfilename = no

###ACL & LOCK
oplocks = Yes
veto oplock files = /*.lock/
level2 oplocks = No
mangled names = No
strict locking = No
acl map full control = yes
acl group control = yes
inherit acls = yes
inherit permissions = yes
unix extensions = no
  1. posix locking = no

###Performance options
strict allocate = yes
socket options=TCP_NODELAY IPTOS_LOWDELAY
Periodically multiple process (around 10-20) hang in deadlock. If I
look at /proc/*/wchan then few of them usually have lock_rename
state, and rest have do_lookup. Example ps output of recent hanging:

pbadmin 24189 0.5 0.0 153596 4248 ? D Apr01 16:08 /usr/local/samba/sbin/smbd -D
pb 13327 0.1 0.0 153164 3564 ? D 10:05 0:07 /usr/local/samba/sbin/smbd -D
pb 13626 0.0 0.0 153164 3024 ? D 10:51 0:00 /usr/local/samba/sbin/smbd -D
pb 13640 0.0 0.0 145452 2944 ? D 10:53 0:00 /usr/local/samba/sbin/smbd -D
pb 13658 0.0 0.0 153164 3488 ? D 10:56 0:00 /usr/local/samba/sbin/smbd -D
pb 13670 0.0 0.0 145452 2944 ? D 10:57 0:00 /usr/local/samba/sbin/smbd -D
pb 13817 0.0 0.0 145664 3624 ? D 11:07 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13832 0.0 0.0 145704 3672 ? D 11:09 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13909 0.0 0.0 145660 3272 ? D 11:19 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13920 0.0 0.0 145660 3376 ? D 11:20 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13929 0.0 0.0 145588 3000 ? D 11:22 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13940 0.0 0.0 145588 3008 ? D 11:23 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 13948 0.0 0.0 145660 3352 ? D 11:24 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14001 0.0 0.0 145660 3228 ? D 11:31 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14011 0.0 0.0 145660 3332 ? D 11:32 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14021 0.0 0.0 145660 3172 ? D 11:33 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14028 0.0 0.0 145660 3244 ? D 11:34 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14037 0.0 0.0 145660 3372 ? D 11:35 0:00 \_ /usr/local/samba/sbin/smbd -D
pb 14062 0.1 0.0 145688 3676 ? D 11:38 0:00 \_ /usr/local/samba/sbin/smbd -D

Usual /proc/*/stack trace of lock_rename process (24189 and 13327 in ps
outut:

<ffffffff81182c07> lock_rename+0xb7/0xe0
<ffffffff81186263> sys_renameat+0x113/0x260
<ffffffff811863cb> sys_rename+0x1b/0x20
<ffffffff8100b072> system_call_fastpath+0x16/0x1b
<ffffffffffffffff> 0xffffffffffffffff

Usual /proc/*/stack trace of do_lookup process (all rest processes):

<ffffffff8118497b> do_lookup+0x11b/0x230
<ffffffff811852c4> __link_path_walk+0x734/0x1030
<ffffffff81185e4a> path_walk+0x6a/0xe0
<ffffffff8118601b> do_path_lookup+0x5b/0xa0
<ffffffff81186c87> user_path_at+0x57/0xa0
<ffffffff8117b90c> vfs_fstatat+0x3c/0x80
<ffffffff8117ba7b> vfs_stat+0x1b/0x20
<ffffffff8117baa4> sys_newstat+0x24/0x50
<ffffffff8100b072> system_call_fastpath+0x16/0x1b
<ffffffffffffffff> 0xffffffffffffffff

Also have several of such messages in dmesg:

INFO: task smbd:13327 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000000 0 13327 24671 0x00000080
ffff88026730fcf8 0000000000000082 ffff88026730fd08 ffffffff81184ce5
ffff88026730fc78 ffffffff8118ef4a ffffffff8100bb8e ffff88026730fcf8
ffff880057817098 ffff88026730ffd8 000000000000fb88 ffff880057817098
Call Trace:
<ffffffff81184ce5> ? __link_path_walk+0x155/0x1030
<ffffffff8118ef4a> ? dput+0x9a/0x150
<ffffffff8100bb8e> ? apic_timer_interrupt+0xe/0x20
<ffffffff8104dfad> ? mutex_spin_on_owner+0x8d/0xc0
<ffffffff814eb66e> __mutex_lock_slowpath+0x13e/0x180
<ffffffff814eb50b> mutex_lock+0x2b/0x50
<ffffffff81182c07> lock_rename+0xb7/0xe0
<ffffffff81186263> sys_renameat+0x113/0x260
<ffffffff811c1f53> ? __posix_lock_file+0xa3/0x4e0
<ffffffff811c1463> ? locks_free_lock+0x43/0x60
<ffffffff811c2525> ? fcntl_setlk+0x75/0x320
<ffffffff810d3f47> ? audit_syscall_entry+0x1d7/0x200
<ffffffff811863cb> sys_rename+0x1b/0x20
<ffffffff8100b072> system_call_fastpath+0x16/0x1b
....
INFO: task smbd:13626 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D 0000000000000001 0 13626 24671 0x00000080
ffff8800511c9ba8 0000000000000082 0000000100000000 ffffffff81422af5
00000000000000e4 0000000000000286 ffff88001e7406c0 00000000000000e8
ffff88005fb1fab8 ffff8800511c9fd8 000000000000fb88 ffff88005fb1fab8
Call Trace:
<ffffffff81422af5> ? memcpy_toiovec+0x55/0x80
<ffffffff814eb66e> __mutex_lock_slowpath+0x13e/0x180
<ffffffffa05141d9> ? _Z21ReleaseDaemonSegAndSG12SegMapStatusiP11StripeGroupij16VfsOperationType+0x89/0xc0 mmfs26
<ffffffff814eb50b> mutex_lock+0x2b/0x50
<ffffffff8118497b> do_lookup+0x11b/0x230
<ffffffff811852c4> __link_path_walk+0x734/0x1030
<ffffffff8113af47> ? handle_pte_fault+0xf7/0xb50
<ffffffff81185e4a> path_walk+0x6a/0xe0
<ffffffff8118601b> do_path_lookup+0x5b/0xa0
<ffffffff81186c87> user_path_at+0x57/0xa0
<ffffffff8104338c> ? __do_page_fault+0x1ec/0x480
<ffffffff8109b669> ? ktime_get_ts+0xa9/0xe0
<ffffffff8117b90c> vfs_fstatat+0x3c/0x80
<ffffffff8117ba7b> vfs_stat+0x1b/0x20
<ffffffff8117baa4> sys_newstat+0x24/0x50
<ffffffff810d3f47> ? audit_syscall_entry+0x1d7/0x200
<ffffffff8100b072> system_call_fastpath+0x16/0x1b
Process hangs in deadlock while rename file or folder.
Setting on options

allowSambaCaseInsensitiveLookup
allowWriteWithDeleteChild
cifsBypassShareLocksOnRename
cifsBypassTraversalChecking
syncSambaMetadataOps

doesn't get to any results.
Updated on 2013-04-05T15:27:46Z at 2013-04-05T15:27:46Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2092 Posts
    ACCEPTED ANSWER

    Re: Processes of SMB hangs in deadlock

    ‏2013-04-05T15:27:46Z  in response to SystemAdmin
    This doesn't look a GPFS problem.
    • DmitryS.
      DmitryS.
      1 Post
      ACCEPTED ANSWER

      Re: Processes of SMB hangs in deadlock

      ‏2013-04-24T09:27:11Z  in response to SystemAdmin

      Which problem it looks to you, smbd or kernel? I thought 'mmfs26' mentioned in last stack trace is GPFS module.

      I change versions smb to 3.6 and gpfs to 3.4.0.20. waiting...