Topic
1 reply Latest Post - ‏2012-04-27T20:28:52Z by SystemAdmin
MichaelHabashy
MichaelHabashy
5 Posts
ACCEPTED ANSWER

Pinned topic after creating the mirrored raid array on a servraid 8k controller on x3650

‏2012-04-27T17:45:04Z |
after creating the mirrored raid array on a servraid 8k controller on x3650 7979.
I get the server acting very sluggish - it is running debian sarge.
the kernel issues are below - I have another machine (same config) not having issue.
Checked bios and firmare - they are the same.
Any help ?

http:// 2400.680070 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 2400.680072 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 2400.680078 ffffffff814811f0 0000000000200086 0000000000000000 ffff880037a17dd0
http:// 2400.680082 0000000000000000 0000000000000000 000000000000f9e0 ffff880037a17fd8
http:// 2400.680086 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 2400.680090 Call Trace:
http:// 2400.680101 <ffffffff810b34ea> ? find_get_pages_tag+0x46/0xdd
http:// 2400.680106 <ffffffff810b38f3> ? sync_page+0x0/0x46
http:// 2400.680111 <ffffffff812f7d72> ? io_schedule+0x73/0xb7
http:// 2400.680114 <ffffffff810b3934> ? sync_page+0x41/0x46
http:// 2400.680118 <ffffffff812f825a> ? __wait_on_bit+0x41/0x70
http:// 2400.680121 <ffffffff810b3ab8> ? wait_on_page_bit+0x6b/0x71
http:// 2400.680126 <ffffffff81064d9c> ? wake_bit_function+0x0/0x23
http:// 2400.680130 <ffffffff810bb78e> ? pagevec_lookup_tag+0x1a/0x21
http:// 2400.680134 <ffffffff810b4292> ? wait_on_page_writeback_range+0x69/0x11b
http:// 2400.680137 <ffffffff810b43b8> ? __filemap_fdatawrite_range+0x4b/0x54
http:// 2400.680141 <ffffffff810b4404> ? filemap_write_and_wait_range+0x43/0x52
http:// 2400.680156 <ffffffff8110b1ae> ? vfs_fsync_range+0x55/0x9e
http:// 2400.680159 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 2400.680162 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 2400.680168 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 2520.680065 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 2520.680069 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 2520.680072 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 2520.680077 ffffffff814811f0 0000000000200086 0000000000000000 ffff880037a17dd0
http:// 2520.680081 0000000000000000 0000000000000000 000000000000f9e0 ffff880037a17fd8
http:// 2520.680085 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 2520.680089 Call Trace:
http:// 2520.680100 <ffffffff810b34ea> ? find_get_pages_tag+0x46/0xdd
http:// 2520.680104 <ffffffff810b38f3> ? sync_page+0x0/0x46
http:// 2520.680110 <ffffffff812f7d72> ? io_schedule+0x73/0xb7
http:// 2520.680113 <ffffffff810b3934> ? sync_page+0x41/0x46
http:// 2520.680117 <ffffffff812f825a> ? __wait_on_bit+0x41/0x70
http:// 2520.680120 <ffffffff810b3ab8> ? wait_on_page_bit+0x6b/0x71
http:// 2520.680125 <ffffffff81064d9c> ? wake_bit_function+0x0/0x23
http:// 2520.680129 <ffffffff810bb78e> ? pagevec_lookup_tag+0x1a/0x21
http:// 2520.680133 <ffffffff810b4292> ? wait_on_page_writeback_range+0x69/0x11b
http:// 2520.680136 <ffffffff810b43b8> ? __filemap_fdatawrite_range+0x4b/0x54
http:// 2520.680140 <ffffffff810b4404> ? filemap_write_and_wait_range+0x43/0x52
http:// 2520.680155 <ffffffff8110b1ae> ? vfs_fsync_range+0x55/0x9e
http:// 2520.680158 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 2520.680161 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 2520.680167 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 2640.680066 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 2640.680071 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 2640.680073 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 2640.680079 ffffffff814811f0 0000000000200086 0000000000000000 ffff880037a17dd0
http:// 2640.680083 0000000000000000 0000000000000000 000000000000f9e0 ffff880037a17fd8
http:// 2640.680087 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 2640.680092 Call Trace:
http:// 2640.680103 <ffffffff810b34ea> ? find_get_pages_tag+0x46/0xdd
http:// 2640.680107 <ffffffff810b38f3> ? sync_page+0x0/0x46
http:// 2640.680112 <ffffffff812f7d72> ? io_schedule+0x73/0xb7
http:// 2640.680116 <ffffffff810b3934> ? sync_page+0x41/0x46
http:// 2640.680119 <ffffffff812f825a> ? __wait_on_bit+0x41/0x70
http:// 2640.680123 <ffffffff810b3ab8> ? wait_on_page_bit+0x6b/0x71
http:// 2640.680127 <ffffffff81064d9c> ? wake_bit_function+0x0/0x23
http:// 2640.680132 <ffffffff810bb78e> ? pagevec_lookup_tag+0x1a/0x21
http:// 2640.680135 <ffffffff810b4292> ? wait_on_page_writeback_range+0x69/0x11b
http:// 2640.680139 <ffffffff810b43b8> ? __filemap_fdatawrite_range+0x4b/0x54
http:// 2640.680142 <ffffffff810b4404> ? filemap_write_and_wait_range+0x43/0x52
http:// 2640.680158 <ffffffff8110b1ae> ? vfs_fsync_range+0x55/0x9e
http:// 2640.680161 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 2640.680164 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 2640.680169 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 2760.680067 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 2760.680072 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 2760.680075 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 2760.680080 ffffffff814811f0 0000000000200086 0000000000000000 ffff880037a17dd0
http:// 2760.680085 0000000000000000 0000000000000000 000000000000f9e0 ffff880037a17fd8
http:// 2760.680089 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 2760.680093 Call Trace:
http:// 2760.680104 <ffffffff810b34ea> ? find_get_pages_tag+0x46/0xdd
http:// 2760.680109 <ffffffff810b38f3> ? sync_page+0x0/0x46
http:// 2760.680114 <ffffffff812f7d72> ? io_schedule+0x73/0xb7
http:// 2760.680118 <ffffffff810b3934> ? sync_page+0x41/0x46
http:// 2760.680121 <ffffffff812f825a> ? __wait_on_bit+0x41/0x70
http:// 2760.680125 <ffffffff810b3ab8> ? wait_on_page_bit+0x6b/0x71
http:// 2760.680129 <ffffffff81064d9c> ? wake_bit_function+0x0/0x23
http:// 2760.680134 <ffffffff810bb78e> ? pagevec_lookup_tag+0x1a/0x21
http:// 2760.680137 <ffffffff810b4292> ? wait_on_page_writeback_range+0x69/0x11b
http:// 2760.680141 <ffffffff810b43b8> ? __filemap_fdatawrite_range+0x4b/0x54
http:// 2760.680144 <ffffffff810b4404> ? filemap_write_and_wait_range+0x43/0x52
http:// 2760.680160 <ffffffff8110b1ae> ? vfs_fsync_range+0x55/0x9e
http:// 2760.680163 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 2760.680166 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 2760.680172 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 2880.680077 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 2880.680081 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 2880.680084 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 2880.680089 ffffffff814811f0 0000000000200086 0000000000000000 ffff880037a17dd0
http:// 2880.680094 0000000000000000 0000000000000000 000000000000f9e0 ffff880037a17fd8
http:// 2880.680098 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 2880.680103 Call Trace:
http:// 2880.680114 <ffffffff810b34ea> ? find_get_pages_tag+0x46/0xdd
http:// 2880.680118 <ffffffff810b38f3> ? sync_page+0x0/0x46
http:// 2880.680124 <ffffffff812f7d72> ? io_schedule+0x73/0xb7
http:// 2880.680127 <ffffffff810b3934> ? sync_page+0x41/0x46
http:// 2880.680131 <ffffffff812f825a> ? __wait_on_bit+0x41/0x70
http:// 2880.680134 <ffffffff810b3ab8> ? wait_on_page_bit+0x6b/0x71
http:// 2880.680139 <ffffffff81064d9c> ? wake_bit_function+0x0/0x23
http:// 2880.680143 <ffffffff810bb78e> ? pagevec_lookup_tag+0x1a/0x21
http:// 2880.680147 <ffffffff810b4292> ? wait_on_page_writeback_range+0x69/0x11b
http:// 2880.680151 <ffffffff810b43b8> ? __filemap_fdatawrite_range+0x4b/0x54
http:// 2880.680154 <ffffffff810b4404> ? filemap_write_and_wait_range+0x43/0x52
http:// 2880.680171 <ffffffff8110b1ae> ? vfs_fsync_range+0x55/0x9e
http:// 2880.680174 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 2880.680177 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 2880.680183 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 3120.680070 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 3120.680075 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 3120.680077 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 3120.680082 ffff88037e463880 0000000000200086 0000000000000000 000000010000000e
http:// 3120.680087 ffff880082bbe7d8 ffff880037a17e18 000000000000f9e0 ffff880037a17fd8
http:// 3120.680091 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 3120.680095 Call Trace:
http:// 3120.680105 <ffffffff810b42f7> ? wait_on_page_writeback_range+0xce/0x11b
http:// 3120.680125 <ffffffffa0132d98> ? log_wait_commit+0xbf/0x112 jbd
http:// 3120.680130 <ffffffff81064d6e> ? autoremove_wake_function+0x0/0x2e
http:// 3120.680138 <ffffffffa01433f8> ? ext3_sync_file+0x94/0xc8 ext3
http:// 3120.680143 <ffffffff8110b1cc> ? vfs_fsync_range+0x73/0x9e
http:// 3120.680146 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 3120.680149 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 3120.680155 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 3240.680068 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 3240.680073 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 3240.680075 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 3240.680081 ffff88037e463880 0000000000200086 0000000000000000 000000010000000e
http:// 3240.680085 ffff880082bbe7d8 ffff880037a17e18 000000000000f9e0 ffff880037a17fd8
http:// 3240.680089 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 3240.680094 Call Trace:
http:// 3240.680104 <ffffffff810b42f7> ? wait_on_page_writeback_range+0xce/0x11b
http:// 3240.680124 <ffffffffa0132d98> ? log_wait_commit+0xbf/0x112 jbd
http:// 3240.680129 <ffffffff81064d6e> ? autoremove_wake_function+0x0/0x2e
http:// 3240.680137 <ffffffffa01433f8> ? ext3_sync_file+0x94/0xc8 ext3
http:// 3240.680142 <ffffffff8110b1cc> ? vfs_fsync_range+0x73/0x9e
http:// 3240.680145 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 3240.680148 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 3240.680154 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
http:// 3360.680070 INFO: task update-apt-xapi:3815 blocked for more than 120 seconds.
http:// 3360.680074 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
http:// 3360.680077 update-apt-xa D 0000000000000000 0 3815 3810 0x00020000
http:// 3360.680082 ffff88037e463880 0000000000200086 0000000000000000 000000010000000e
http:// 3360.680087 ffff880082bbe7d8 ffff880037a17e18 000000000000f9e0 ffff880037a17fd8
http:// 3360.680091 0000000000015780 0000000000015780 ffff88037c754db0 ffff88037c7550a8
http:// 3360.680095 Call Trace:
http:// 3360.680106 <ffffffff810b42f7> ? wait_on_page_writeback_range+0xce/0x11b
http:// 3360.680125 <ffffffffa0132d98> ? log_wait_commit+0xbf/0x112 jbd
http:// 3360.680131 <ffffffff81064d6e> ? autoremove_wake_function+0x0/0x2e
http:// 3360.680138 <ffffffffa01433f8> ? ext3_sync_file+0x94/0xc8 ext3
http:// 3360.680143 <ffffffff8110b1cc> ? vfs_fsync_range+0x73/0x9e
http:// 3360.680147 <ffffffff8110b276> ? do_fsync+0x28/0x39
http:// 3360.680150 <ffffffff8110b2a5> ? sys_fsync+0xb/0x10
http:// 3360.680155 <ffffffff810371c0> ? sysenter_dispatch+0x7/0x2e
Updated on 2012-04-27T20:28:52Z at 2012-04-27T20:28:52Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2364 Posts
    ACCEPTED ANSWER

    Re: after creating the mirrored raid array on a servraid 8k controller on x3650

    ‏2012-04-27T20:28:52Z  in response to MichaelHabashy
    I hate problems like this! Clearly something MUST be different between this machine and the one that is working well, but tracking that down is the hard part. Are these machines exact mirrors of each other? Is it possible that this is a sign of an impending hardware failure?

    A cursory look around didn't reveal much to me based purely on the error log. One discusion did mention the possibility of ext3 journaling issues. I'm not sure what questions to ask first. Anything unusual about running processes? Anything weird happen when you set it up?