Troubleshooting
Problem
If you are using a GPFS block size larger than 512 Kbytes, you may encounter 'blk_cloned_rq_check_limits: over max size limit' errors which lead to dm-multipath path failure. GPFS then will not be able to access the underlying block device.
Symptom
You may encounter the following error messages and dm-multipath path failures:
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 70:64.
Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 71:80.
Jun 14 09:33:38 testhost multipathd: 70:64: mark as failed
Jun 14 09:33:38 testhost multipathd: nsd_crs_04: remaining active paths: 1
Jun 14 09:33:38 testhost multipathd: 71:80: mark as failed
[...]
Cause
This is because the following upstream kernel commit is in the Linux distributions:
Commit bf4e6b4e757488dee1b6a581f49c7ac34cd217f8
Author: Hannes Reinecke <hare@suse.de>
Date: Thu Nov 26 08:46:57 2015 +0100
block: Always check queue limits for cloned requests
[...]
Environment
Red Hat Enterprise Linux 6.8, 7.2, and 7.3
Diagnosing The Problem
Check dmesg to see if there are 'blk_cloned_rq_check_limits: over max size limit' error messages.
Resolving The Problem
By default Linux devices are configured for a maximum 512Kbyte IO size. If you are using a GPFS blocksize larger than 512Kbytes, you should upgrade your level of code:
Note: IBM Spectrum Scale V4.2.3 is not affected by this issue.
- For IBM Spectrum Scale V4.2.0 thru V4.2.2.1, upgrade to V4.2.2.2 or later available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.2&platform=All&function=all
- For IBM Spectrum Scale V4.1.0.0 thru V4.1.1.12, upgrade to V4.1.1.13 or later available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=All&function=all
- For IBM GPFS V3.5.0.0 thru V3.5.0.33, upgrade to V3.5.0.34 available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%20software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=All&function=all
If you are unable to upgrade, contact IBM Service to obtain an efix referencing:
IBM GPFS V3.5.0.34, APAR IV93649, or
IBM Spectrum Scale V4.1.1.13, APAR IV93650, or
IBM Spectrum Scale V4.2.2.2, APAR IV92104
To contact IBM Service, see http://www.ibm.com/planetwide/
Workaround
Note: With the Fix/APARs above, the rules in /etc/udev/ are no longer needed.
If you are using a GPFS blocksize larger than 512Kbytes, and you do not apply the fix, you need to add an udev rule to increase the max_sectors_kb parameter.
For example:
If the GPFS file system block size is 4MB, and the storage is from IBM, you need to create a new rule in /etc/udev/rules.d/54-custom.rules with the following content:
ACTION=="add|change", SUBSYSTEM=="block", ATTRS{vendor}=="IBM*", RUN+="/bin/sh -c '/bin/echo 4096 > /sys/block/%k/queue/max_sectors_kb'"
You need to change the block size in the 'echo' command to match the file system block size. You also need to provide the correct storage vendor in the "ATTRS{vendor}" statement.
Was this topic helpful?
Document Information
Modified date:
01 August 2018
UID
ssg1S1009622