With the development of the information technology there are more and more data generated everyday in both Enterprise’s and individual scenarios. It requires both on-demand capability and efficiency to access data with low cost. Faster and bigger capability hardware are being used. It's common that various types of storage mediums coexist in a one storage environment , for example ,SSD, SAS, SATA. And usually high performance storage mediums are with high cost and vice-versa. we need to balance the cost and the efficiency. We call such a solution is Multi-Tiering .
- How does it work
- In this solution, LVM (Logical Volume Manager), can provide unified storage capacity for users by combining different types of physical storage devices, and it also provides the capability to migrate data at block-level , while not affecting accesses to the data in logical storage.
- In this solution, blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space.
- In this solution, it can generate a heat map to represent the heat distribution among the user space based on the IO traces from blktrace.
- In this solution, it can intelligently recognize the hot data (Intensive IO access ) based on the heat map and migrate it to the fast tier (high performance, high cost) to increase performance.
- In this solution, it can intelligently recognize the warm data (medium IO access) based on the heat map and migrate it to the middle tier (medium performance, medium cost) to balance IOPS and cost.
- In this solution, it can intelligently recognize the cold data (No IO access) based on the heat map and migrate it to the slow tier (low performance, low cost) to save the capacity of high-performance storage capacity, and additionally to improve the efficiency of storage utilization.
LVM can manage disk drives and similar mass-storage devices. It is an advanced function on Unix platform, then it is transplanted to Linux. It is on top of the hard disks and partitions, and it creates an illusion of continuity and ease-of-use for managing hard-drive replacement, repartitioning, and etc. It's of the following capabilities:
The capability to copy and share contents from one disk to another without disrupting service;
The capability to online migrate data between logical volumes (LV) at a block-level, extent level;
The capability to move an online logical volume between physical volumes (PVs) on the same Volume Group;
Supporting RAID by using RAID arrays as LVM physical volumes;
Reliable and commonly used among Unix/Linux platforms.
- blktrace & Intelligent Data Migration Advisor
blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space. It can provide block level IO statistics, including reads/writes, IO sizes, and IO response times for each extent in the user space.
Then, Intelligent Data Migration Advisor can generate a heat map of all the data based on the IO statistics provided by blktrace. Additionally, it will generate data migration advice abiding by the following basic guides to provide an cost effective way to achieve high performance. The migration advice will then be sent to LVM, and LVM will online move data to the location given by Intelligent Data Migration Advisor.
- Designed Functions
Data with medium IO access is considered as warm data, and warm data is supposed to locate/relocate on medium-performance storage devices;
Data with low/no IO access is considered as cold data, and cold data is supposed to locate/relocate on low-performance storage devices;
- How to deployment the solution on Linux