Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
5 replies Latest Post - ‏2012-05-11T05:39:36Z by Daniel996
Daniel996
Daniel996
10 Posts
ACCEPTED ANSWER

Pinned topic V7000 - how to optimise for small reads/writes?

‏2012-05-07T03:07:38Z |
Hi All,

I have a newly installed V7000 for a customer, with a shelf of 24 x 600GB SAS and two shelves of 24 x 1TB SATA. Platform is XenServer, and VMs are spread across both SAS and SATA, in addition I have a volume on the SATA pool for a physical Windows backup server. All connections are 8Gb/s FC. Performance across the board is generally great as you'd expect except for their backup solution.

The customer uses a backup product called PHD that reads the VMs, compresses/dedupes and stores the backups on a shared folder on the backup server. Since we upgraded to the V7000 from their old HDS, the backup speeds (which are slow) haven't improved at all. It seems to be because the PHD software reads and writes in <1MB chunks.

What can I do to imrove performance of the backups? Currently it's taking 24 hours to run a full backup on 3.5TB of data. I've been through the Implementing V7000 redbook and there's not much abour performance tuning in there, nor in the V7000 info centre that I can find. The backup software vendor has been no help. Graphs of both MB and IOPS are attached - it doesn't look like the system is working hard. (Mbs graph in 2nd post, I can't attach more than one file to a post)

Regards
Dan (V7000 newbie)
Updated on 2012-05-11T05:39:36Z at 2012-05-11T05:39:36Z by Daniel996
  • Daniel996
    Daniel996
    10 Posts
    ACCEPTED ANSWER

    Re: V7000 - how to optimise for small reads/writes?

    ‏2012-05-07T03:08:17Z  in response to Daniel996
    MB/s graph
  • orbist
    orbist
    4 Posts
    ACCEPTED ANSWER

    Re: V7000 - how to optimise for small reads/writes?

    ‏2012-05-08T14:34:43Z  in response to Daniel996
    Hi, when you say small, what size are the actual reads / writes (you can work it out by dividing the MB/s by the IOPs I guess) - looking at the pictures, 30MB/s reads, and 700 iops - so something like 45KB I/O size.

    Anything 32kb and above is fine actually, and you should be able to get 500MB/s from a single stream read / write to a single array.

    If you are getting such a small load into the top end of the V7000, this would suggest its more of a concurrency issue via the backup software than anything.

    What are the default queue depths for the HBA and block devices on the backup server, and is there any way to increase the concurrency that the backup software issues - looks to me like its single threaded, single I/O at a time to get such a low rate.
    • Daniel996
      Daniel996
      10 Posts
      ACCEPTED ANSWER

      Re: V7000 - how to optimise for small reads/writes?

      ‏2012-05-09T04:01:11Z  in response to orbist
      Thanks for the response Orbist,

      Queue length is currently set to 32, which appears to be the default for XenServer. On the backup server it is set to 256, the default for Windows as far as I can tell.

      The PHD backup appliance is a virtual appliance that runs on one host on the XenServer pool. It has the option to use a locally attached vdisk instead of CIFS share, so I tried this out as well but performance is pretty much the same.

      When I queried them about the performance graph and the staggered writes, they said it's because of OS caching. Not really sure what that means when I am backing up to a vdisk attached to their appliance as Windows isn't involved.

      Cheers
      Dan
  • karl.h
    karl.h
    64 Posts
    ACCEPTED ANSWER

    Re: V7000 - how to optimise for small reads/writes?

    ‏2012-05-09T20:09:12Z  in response to Daniel996
    Hi,
    This does not sound like an issue of the V7k. As far as I understand this is a Linux based appliance which is doing the backup. I had some cases with XEN and Linux guests were changing the defualt IO scheduler parameters changed the throughput dramatically. I think ther are 3 different values for the IO scheduler (noop did the trick in most cases as far as I remember. But might be different in your setup)
    Just google for XEN and IO scheduler and you will find lot's of information about this issue.
    Hope this helps.
    • Daniel996
      Daniel996
      10 Posts
      ACCEPTED ANSWER

      Re: V7000 - how to optimise for small reads/writes?

      ‏2012-05-11T05:39:36Z  in response to karl.h
      Thanks Karl,

      Noop is the default in XenServer, but I'll see if I can find out from PHD what they use in their appliance (I don't have command line access to it).

      Cheers
      Dan