Topic
3 replies Latest Post - ‏2013-03-20T18:41:18Z by Novikov_Alexander
SystemAdmin
SystemAdmin
4779 Posts
ACCEPTED ANSWER

Pinned topic DS 4700 hight latency

‏2013-03-20T12:09:21Z |
Good all the time of day.

Briefly describe the hardware configuration:
3 host VMWare Esxi (Servers IBM)
1 store IBM DS 4700
2 FC Switch IBM System Storage SAN24B-4 Express (Brocade 300)
2 hosts are connected to the storage.
As the system is used for monitoring Veeam One.

Now let us briefly about the problem:
Occasionally get errors about "losing" storage, ie wheels off for a few seconds and connect again.
Also it is accompanied by increased delays to 250-750 milliseconds.
The problem occurs on two hosts at once, almost simultaneously.

Extract from log Esxi:
Device naa.600a0b800050beaa00000b0350af1498 performance has deteriorated. I / O latency increased from average value of 5900
microseconds to 275685 microseconds. warning

Just look at the number of lost words FC switches on the ports where the attached storage. The numbers are quite large about 1 million, but it's over a long period of time off including authorized hosts.

Also on our VMWare received advice that is sometimes required to expose certain fill word on ports Switch, I would like to clarify where this option is set, and what is required for my situation?

Also in the time of the error in the section esxtop disk devices were recorded as follows:
Parameter FMBRD / s periodically raised from 0 to 214
In the next cycle parameter DAVG / s rising to 6000

After 5 minutes, the problem has moved to the next stage
Parameter DAVG / s rose only to 500

Entire period QUED option for some devices equaled 34, ACTV - 30,% USD 100 respectively.

After another 5 minutes the problem went away.
Updated on 2013-03-20T18:41:18Z at 2013-03-20T18:41:18Z by Novikov_Alexander
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS 4700 hight latency

    ‏2013-03-20T12:23:17Z  in response to SystemAdmin
    Hi,

    performance drop can be caused by controller failover on DS4k.
    Are you getting "drive not on the preferred path" in recovery guru?

    Failover may be caused by link problems, improper setup of multipathing, or ds4k controller being rebooted for some reason.
    Check MEL of the storage for events with same timestamp as your performance problem.

    Fillword setting on a switch makes sense only on 8G link speed. Also a link will not come up if your setting is wrong, so most likely it's not a case here.

    /Sergey
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS 4700 hight latency

    ‏2013-03-20T12:27:38Z  in response to SystemAdmin
    Доброго всем времени суток.

    Кратко опишу конфигурацию оборудования:
    3 хоста VMWare Esxi (Сервера IBM)
    1 хранилище IBM DS 4700
    2 FC свича IBM System Storage SAN24B-4 Express (Brocade 300)
    2 хоста имеют подключение к хранилищу.
    В качестве системы мониторинга используется Veeam One.

    Теперь кратко о проблеме:
    Периодически получаю ошибки о «потери» хранилища, т.е. диски отключаются на несколько секунд и подключаются вновь.
    Так же это сопровождается повышением задержек до 250-750 милисекунд.
    Проблема возникает на двух хостах сразу, практически одновременно.

    Выписка из логов Esxi:
    Device naa.600a0b800050beaa00000b0350af1498 performance has deteriorated. I/O latency increased from average value of 5900
    microseconds to 275685 microseconds. warning

    Так же смотрел количество потерянных слов на FC свичах на тех портах где подключено хранилище. Цифры довольно большие порядка 1 млн, но это за длительный период времени включая санкционированное отключение хостов.

    Так же на форуме VMWare получил совет что иногда требуется выставлять определенные fill word на портах свича, хотелось бы уточнить где устанавливается этот параметр и какой требуется для моей ситуации?

    Так же в момент ошибки в esxtop в разделе disk devices было зафиксировано следующее:
    Параметр FMBRD/s переодически поднимался с 0 до 214
    В следующий такт параметр DAVG/s поднимался до 6000

    Через 5 минут проблема перешла в следующую стадию
    Параметр DAVG/s поднимался лишь до 500

    Весь период времени параметр QUED для некоторых устройств был равен 34, ACTV – 30, %USD соотвественно 100.

    Через еще 5 минут проблема ушла.
    • Novikov_Alexander
      Novikov_Alexander
      1398 Posts
      ACCEPTED ANSWER

      Re: DS 4700 hight latency

      ‏2013-03-20T18:41:18Z  in response to SystemAdmin
      Dear Владимир,

      looks like wrong configuration used. Read documentation and fix it, see section "Appendix B. Using a storage subsystem with a VMware ESX Server configuration":

      Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
      http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5090826

      BTW, if you prefer your native language you can use another IBM Forum:

      Системы хранения данных IBM
      https://www.ibm.com/developerworks/forums/forum.jspa?forumID=1443

      Regards,
      Alexander Novikov
      Russia, Moscow