I have a newly installed V7000 for a customer, with a shelf of 24 x 600GB SAS and two shelves of 24 x 1TB SATA. Platform is XenServer, and VMs are spread across both SAS and SATA, in addition I have a volume on the SATA pool for a physical Windows backup server. All connections are 8Gb/s FC. Performance across the board is generally great as you'd expect except for their backup solution.
The customer uses a backup product called PHD that reads the VMs, compresses/dedupes and stores the backups on a shared folder on the backup server. Since we upgraded to the V7000 from their old HDS, the backup speeds (which are slow) haven't improved at all. It seems to be because the PHD software reads and writes in <1MB chunks.
What can I do to imrove performance of the backups? Currently it's taking 24 hours to run a full backup on 3.5TB of data. I've been through the Implementing V7000 redbook and there's not much abour performance tuning in there, nor in the V7000 info centre that I can find. The backup software vendor has been no help. Graphs of both MB and IOPS are attached - it doesn't look like the system is working hard. (Mbs graph in 2nd post, I can't attach more than one file to a post)
Dan (V7000 newbie)
Pinned topic V7000 - how to optimise for small reads/writes?
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2012-05-11T05:39:36Z at 2012-05-11T05:39:36Z by Daniel996
orbist 060000HPM54 Posts
Re: V7000 - how to optimise for small reads/writes?2012-05-08T14:34:43ZThis is the accepted answer. This is the accepted answer.Hi, when you say small, what size are the actual reads / writes (you can work it out by dividing the MB/s by the IOPs I guess) - looking at the pictures, 30MB/s reads, and 700 iops - so something like 45KB I/O size.
Anything 32kb and above is fine actually, and you should be able to get 500MB/s from a single stream read / write to a single array.
If you are getting such a small load into the top end of the V7000, this would suggest its more of a concurrency issue via the backup software than anything.
What are the default queue depths for the HBA and block devices on the backup server, and is there any way to increase the concurrency that the backup software issues - looks to me like its single threaded, single I/O at a time to get such a low rate.
Re: V7000 - how to optimise for small reads/writes?2012-05-09T04:01:11ZThis is the accepted answer. This is the accepted answer.
- orbist 060000HPM5
Queue length is currently set to 32, which appears to be the default for XenServer. On the backup server it is set to 256, the default for Windows as far as I can tell.
The PHD backup appliance is a virtual appliance that runs on one host on the XenServer pool. It has the option to use a locally attached vdisk instead of CIFS share, so I tried this out as well but performance is pretty much the same.
When I queried them about the performance graph and the staggered writes, they said it's because of OS caching. Not really sure what that means when I am backing up to a vdisk attached to their appliance as Windows isn't involved.
karl.h 1200007QQ764 Posts
Re: V7000 - how to optimise for small reads/writes?2012-05-09T20:09:12ZThis is the accepted answer. This is the accepted answer.Hi,
This does not sound like an issue of the V7k. As far as I understand this is a Linux based appliance which is doing the backup. I had some cases with XEN and Linux guests were changing the defualt IO scheduler parameters changed the throughput dramatically. I think ther are 3 different values for the IO scheduler (noop did the trick in most cases as far as I remember. But might be different in your setup)
Just google for XEN and IO scheduler and you will find lot's of information about this issue.
Hope this helps.
Re: V7000 - how to optimise for small reads/writes?2012-05-11T05:39:36ZThis is the accepted answer. This is the accepted answer.
- karl.h 1200007QQ7
Noop is the default in XenServer, but I'll see if I can find out from PHD what they use in their appliance (I don't have command line access to it).