Always use DRAID6 (The Data Reduction Pools edition)

Body

Last year in our Autumn UK user group, Barry brought along a bunch of T-shirts which say: "Always Use DRAID6", he has since had a second batch made which say: "Whatever the question, the answer is DRAID6". I'm one of the lucky recipients of the original batch. As an owner of one of the T-shirts I feel it's been too long since I evangelized the use of DRAID6, and with Data Reduction Pools now being out in production I thought I would use it as an example of why DRAID6 is so important.

Firstly, I'll explain a little bit about how Data Reduction Pools manage host writes. Data Reduction Pools use a Log Structured Array, every write ends up being written to the end (even if the write is for a previously written lba range). So this means a sequential write stream is constructed from all the writes to a volume. It also effectively merges the write streams together for different volumes in the same IO Group and the same pool into a single stream.

What this means is that by the time cache decides to write that data to an array we have built up lovely full stride writes which RAID6 loves. When doing a small write to a RAID6 array there are multiple IOs needed to read the old data, read the old parities, write the new data and write the new parities, with a full stride write, every part of a stride is being written, so the new parity can be calculated without reading the old data or old parities so the IO is a lot quicker. This workload can outperform RAID10 due to the fact that less data is being written.

However, in Spectrum Virtualize we use an extent system, so the Data Reduction Pool will be writing that stream into an extent, when that extent is full, it will use another one. Each extent belongs to a single mdisk, so belongs to a single array. With the default extent size of 1GB, cache will be typically writing 1GB of data to one extent, then 1GB of data to the next, with a small overlap period where it is writing to both at the same time.

Imagine a system with 72 drives in the pool, those are arranged into 6 traditional RAID arrays so you have 12 drives per array. For 1GB of writes one of those arrays will be active and so your performance is limited to what you can get from those 12 drives. There will be some metadata writes that will go to the other arrays, so the other drives aren't completely idle but compared to the one that the write stream is aimed at it the IO rate should be insignificant. If the performance of that single array is bad enough, cache will try to write to other extents but there is only so much it can mitigate, especially when you start moving to extent sizes of 8gb.

Even worse, imagine that you get a drive failure, when writing to the array that is rebuilding the performance will drop. You may think that only having 1/6th of the extents having bad performance is fine and cache will try to compensate and issue writes to other extents on other arrays, but eventually the cache will end up skewed so that nearly all the data belongs to the rebuilding array.

So what's the difference with DRAID6? Not every write will write to every drive, but with only a very small amount of writes every drive should be written to. So you now get the performance of using all the drives. You also have the benefit of having DRAID6 using a lower rebuild rate on a single failure, so it has less impact on the system's performance. And during the rebuild, you get much more consistent performance at the mdisk level which keeps the cache much happier. Also with the rebuild being much faster (even in it's reduced single failure mode) the problem resolves itself a lot quicker to get back to full health.

Data Reduction Pools : another reason to always use DRAID6