IC5Notice: We have upgraded developerWorks Community to the latest version of IBM Connections. For more information, read our upgrade FAQ.
Topic
  • 7 replies
  • Latest Post - ‏2013-05-14T20:47:08Z by Bill_Buros
harshGN
harshGN
3 Posts

Pinned topic Java workload -- Configuration and scaling of storage devices as disks ?

‏2013-05-08T18:57:26Z |

Hi,

           As per our team requirement i have setup drives on 4 SSDs we had on P7+ system (PowerLinux RHEL 6.4). our java workload suite stresses disks or drives. Below are the results i have obtained for stressing /root vg , (/dev/sda) , 1-VG  2-LVs, 2 mount points (2 drives -- /dev/sdc) and  1-VG 4-LVs 4 mount points / file systems_ext4  (4 drives -- /dev/sdd) .   Our Java workload Performance score doesn't seem to scale with increase in no of file systems created for a given SSD or goup of SSDs.  Please suggest for choice of  SSDs (we have multipath devices also and other  RAID disks -- 8 SSDs ) to have 1,2,4,8 disks for workload scaling.

NAME                              ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE   RA
sr0                                                       0    512      0     512     512    1 cfq       128  128
sdb                                                      0    512      0     512     512    1 cfq       128  128
sdb1                                                    0    512      0     512     512    1 cfq       128  128
sdb2                                                    0    512      0     512     512    1 cfq       128  128
sdb3                                                    0    512      0     512     512    1 cfq       128  128
vg_perf144145-lv_root (dm-8)              0    512      0     512     512    1           128  128
vg_perf144145-lv_swap (dm-9)            0    512      0     512     512    1           128  128
vg_perf144145-lv_home (dm-10)         0    512      0     512     512    1           128  128
sda                                                       0    512      0     512     512    1 cfq       128  128
sda1                                                     0    512      0     512     512    1 cfq       128  128
sda2                                                     0    512      0     512     512    1 cfq       128  128
sda3                                                     0    512      0     512     512    1 cfq       128  128
sdc                                                        0    512      0     512     512    1 cfq       128  128
vg_n1disk-N1lv1 (dm-14)                 0    512      0     512     512    1           128  128
vg_n1disk-N1lv2 (dm-15)                 0    512      0     512     512    1           128  128
sdd                                                        0    512      0     512     512    1 cfq       128  128
vg_n2disk-N2lv1 (dm-16)                 0    512      0     512     512    1           128  128
vg_n2disk-N2lv2 (dm-17)                 0    512      0     512     512    1           128  128
vg_n2disk-N2lv3 (dm-18)                 0    512      0     512     512    1           128  128
vg_n2disk-N2lv4 (dm-19)                 0    512      0     512     512    1           128  128
sde                                                        0    512   0     512     512    1 cfq       128  128
mpathe (dm-1)                                    0    512      0     512     512    1 cfq       128  128
vg_disk2-lv_disk2 (dm-12)               0    512      0     512     512    1           128  128
sdf                                                          0    512      0     512     512    1 cfq       128  128
mpathl (dm-4)                                      0    512      0     512     512    1 cfq       128  128
sdm                                                        0    512      0     512     512    1 cfq       128  128
mpathe (dm-1)                                     0    512      0     512     512    1 cfq       128  128
vg_disk2-lv_disk2 (dm-12)                0    512      0     512     512    1           128  128
sdn                                                         0    512      0     512     512    1 cfq       128  128
mpathl (dm-4)                                      0    512      0     512     512    1 cfq       128  128
sdk                                                          0    512      0     512     512    1 cfq       128  128
mpathg (dm-3)                                     0    512      0     512     512    1 cfq       128  128
sdg                                                          0    512      0     512     512    1 cfq       128  128
mpathj (dm-2)                                       0    512      0     512     512    1 cfq       128  128
sdi                                                           0    512      0     512     512    1 cfq       128  128
mpathh (dm-6)                                     0    512      0     512     512    1 cfq       128  128
vgdisk0-lvdisk0 (dm-13)                     0    512      0     512     512    1           128  128
sdl                                                           0    512      0     512     512    1 cfq       128  128
mpathf (dm-7)                                       0    512      0     512     512    1 cfq       128  128
sdj                                                           0    512      0     512     512    1 cfq       128  128
mpathi (dm-5)                                      0    512      0     512     512    1 cfq       128  128
sdh                                                         0    512      0     512     512    1 cfq       128  128
mpathk (dm-0)                                     0    512      0     512     512    1 cfq       128  128
vg_disk1-lv_disk1 (dm-11)                0    512      0     512     512    1           128  128
sdo                                                         0    512      0     512     512    1 cfq       128  128
mpathj (dm-2)                                      0    512      0     512     512    1 cfq       128  128
sds                                                         0    512      0     512     512    1 cfq       128  128
mpathg (dm-3)                                     0    512      0     512     512    1 cfq       128  128
sdp                                                          0   512      0     512     512    1 cfq       128  128
mpathk (dm-0)                                      0    512      0     512     512    1 cfq       128  128
vg_disk1-lv_disk1 (dm-11)                 0    512      0     512     512    1           128  128
sdq                                                          0    512      0     512     512    1 cfq       128  128
mpathh (dm-6)                                      0    512      0     512     512    1 cfq       128  128
vgdisk0-lvdisk0 (dm-13)                      0    512      0     512     512    1           128  128
sdr                                                            0    512      0     512     512    1 cfq       128  128
mpathi (dm-5)                                        0    512      0     512     512    1 cfq       128  128
sdt                                                            0    512      0     512     512    1 cfq       128  128
mpathf (dm-7)                                        0    512      0     512     512    1 cfq       128  128

DRIVES       Java Workload and Scores
                           Sequential   Random  ------------------> 2 workloads stressing Storage within the POWER 7+ system (Linux on POWER)
/root drive     13.292       38.801
2 drives           13.460      43.193  
4 drives           13.152      43.051

Here are the steps i followed to create the drives for SSDs.  Although not in the same order of executing commands.  (see attachement ConfigureVG_LV_PowerLinux.txt)

we have
4 SSDs
1) sda used for creating drive on /root
2) sdb used for dedicated drive to have workload performance score comparison to root drive
3) sdc used for 2 drives creation
4) sdd used for 4 drives creation

But before i could have above experiments, i had used multipath as you said to create VGs --> LVs --> mount file systems.  Since each  multipath had 2 devices (sdk sdg etc) i created
a) 1 VG to have 2 devices with 2 LVs and 1 mount point / filesystem (so i guess stressing 2 drives)
b) 2 VGs to have 4 devices with 4 LVs and 1 mount point / filesystem(FS)  (so i guess stressing 4 drives)
c) 4 VGs to have 8 devices with 8 LVs and 1 mount point  (so i guess stressing 4 drives) ?

Following is the performance scores for above
DRIVES                         Java Workload and Scores
                               Sequential           Random
2 drives 1 FS                27.381         53.020         /disk1
4 drives 1 FS                27.625         53.716        /disk2
8 drives 1 FS                27.363         53.114        /disk3 -- removed now for some reason.




[root@perf144145 SERT-1.0.0-PwrLnx]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_perf144145-lv_root
                       50G   42G  5.5G  89% /
tmpfs                 122G     0  122G   0% /dev/shm
/dev/sdb2             485M   52M  409M  12% /boot
/dev/mapper/vg_perf144145-lv_home
                      110G  188M  104G   1% /home
/dev/mapper/vg_disk2-lv_disk2
                       20G  2.2M   19G   1% /disk2    -----> using multipath    (4 drives )
/dev/mapper/vg_disk1-lv_disk1
                       33G  1.4M   31G   1% /disk1  -------> using multipath   (2 drives )
 

Please suggest for any other methods of configuring disks or drives -- SSDs to get scaling score results for the java workload.

Thanks

Harsha

Attachments

Updated on 2013-05-09T13:55:30Z at 2013-05-09T13:55:30Z by Bill_Buros
  • Bill_Buros
    Bill_Buros
    151 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-09T16:12:40Z  

    We are looking into this.     We will provide you with some suggestions on how to approach this.

  • Breno Leitao
    Breno Leitao
    14 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-09T20:53:39Z  

    Hi harshGN,

    Let me try to understand what you are trying to achieve. Are you looking to improve the disk performance (throughput) as you have more disks?  This seems to be a case of using RAID (PowerLinux machines usually have an embedded RAID controller). You an also use LVM to do so, using stripped LVM (uses lvmcreate -i).

    Also, what do you mean by driver as in "3) sdc used for 2 drives creation"? In this case, Did you create a VG for this disk (sdc), and then create 2 LV for this VG?

    Moreover, I didn't understand where did you apply multipath. Are you applying multipath over these LV from the same VG?

     

    Regards,

    Breno Leitão

  • Bill_Buros
    Bill_Buros
    151 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-14T13:51:39Z  

    Hello Harsha, can you provide us with some additional details on what you are really trying to do?    We can provide some ideas for best practices and considerations.

  • harshGN
    harshGN
    3 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-14T17:15:30Z  

    Hi harshGN,

    Let me try to understand what you are trying to achieve. Are you looking to improve the disk performance (throughput) as you have more disks?  This seems to be a case of using RAID (PowerLinux machines usually have an embedded RAID controller). You an also use LVM to do so, using stripped LVM (uses lvmcreate -i).

    Also, what do you mean by driver as in "3) sdc used for 2 drives creation"? In this case, Did you create a VG for this disk (sdc), and then create 2 LV for this VG?

    Moreover, I didn't understand where did you apply multipath. Are you applying multipath over these LV from the same VG?

     

    Regards,

    Breno Leitão

    Hi Breno,

                      Sorry for the delayed reply, i was unwell and logged in today to check mails.  I am doing certification for P7+ system and storage workload which stress disks is measured for it's scaling capabilities. i.e. with each disk configured to get stressed by java storage workload the performance of the workload should double. (2X) . So my expectation is for 1 disk if score is X, then for 2 disks it should be >= 2X , for 4 disks  score should be 4X and for 8 disks it should be 8X.

    Basically  Storage workload has two sub workloads sequential and random ---- writes and reads data files of size , about 1 GB -- reads blocks of data sequentially and randomly respectively.

    i created 1 VG and 1 LV for one disk (one SSD) and then different VG's for each SSD in the system.

    /dev/sda --> SSD has 1 PV, 1 VG, 1 LV  ---- 1 drive (for say) --- performance score of the workload is X

    /dev/sdb --> SSD has 1 PV, 1 VG, 2 LV  ----- 2 drives  --- performance score of the workload should be 2X

    /dev/sdc --> SSD has 1 PV, 1 VG, 4 LV's ------ 4 drives --- performance score of the workload should be 4X

    /dev/sdd --> SSD has 1 PV, 1 VG , 8 LV's ---- 8 drives --- performance score of the workload should be 8X..

     

    I am a newbie in case of multipath. In my P7+ system each multipath /devmapper/multipathh , i, j, k, l,m,n etc -- has 2 SSDs. So when i created 1 disk --- it means i created a VG on one multipath (i.e. having 2 SSDs) and 1 LV --> mounted it as a disk. Similarly created second disk with 2 multipath (in total 4 SSDs) but with 1 VG and 2 LV's --> 2 disks mounted. etc..

    /dev/sda , sdb, sdc, sdd --setups are different than configuration of multipath disks above.

    With both configurations i do not see the expected X, 2X, 4X, 8X workload score performance. Is there something i am doing wrong.. your suggestions are most welcome. Thanks in advance.

    By drive, i mean mount points of each  LV.

     

    Thanks

    Harsha

     

    Updated on 2013-05-14T18:48:07Z at 2013-05-14T18:48:07Z by jhopper
  • harshGN
    harshGN
    3 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-14T17:20:05Z  

    Hello Harsha, can you provide us with some additional details on what you are really trying to do?    We can provide some ideas for best practices and considerations.

    Hi Bill,

                  It's great to hear from the community. I have posted my explanation to Breno above regards my task to be achieved. 

     

    Thanks

    Harsha

  • Breno Leitao
    Breno Leitao
    14 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-14T18:47:34Z  
    • harshGN
    • ‏2013-05-14T17:15:30Z

    Hi Breno,

                      Sorry for the delayed reply, i was unwell and logged in today to check mails.  I am doing certification for P7+ system and storage workload which stress disks is measured for it's scaling capabilities. i.e. with each disk configured to get stressed by java storage workload the performance of the workload should double. (2X) . So my expectation is for 1 disk if score is X, then for 2 disks it should be >= 2X , for 4 disks  score should be 4X and for 8 disks it should be 8X.

    Basically  Storage workload has two sub workloads sequential and random ---- writes and reads data files of size , about 1 GB -- reads blocks of data sequentially and randomly respectively.

    i created 1 VG and 1 LV for one disk (one SSD) and then different VG's for each SSD in the system.

    /dev/sda --> SSD has 1 PV, 1 VG, 1 LV  ---- 1 drive (for say) --- performance score of the workload is X

    /dev/sdb --> SSD has 1 PV, 1 VG, 2 LV  ----- 2 drives  --- performance score of the workload should be 2X

    /dev/sdc --> SSD has 1 PV, 1 VG, 4 LV's ------ 4 drives --- performance score of the workload should be 4X

    /dev/sdd --> SSD has 1 PV, 1 VG , 8 LV's ---- 8 drives --- performance score of the workload should be 8X..

     

    I am a newbie in case of multipath. In my P7+ system each multipath /devmapper/multipathh , i, j, k, l,m,n etc -- has 2 SSDs. So when i created 1 disk --- it means i created a VG on one multipath (i.e. having 2 SSDs) and 1 LV --> mounted it as a disk. Similarly created second disk with 2 multipath (in total 4 SSDs) but with 1 VG and 2 LV's --> 2 disks mounted. etc..

    /dev/sda , sdb, sdc, sdd --setups are different than configuration of multipath disks above.

    With both configurations i do not see the expected X, 2X, 4X, 8X workload score performance. Is there something i am doing wrong.. your suggestions are most welcome. Thanks in advance.

    By drive, i mean mount points of each  LV.

     

    Thanks

    Harsha

     

    Hi harshGN,

    I'd not expect to have a bigger performance if you split a single disk in different LV. I'd say that the performance limit is on the disk, meaning that the accessing it through different paths (LV) would not improve the performance.

    So, if you have 4 LV for a single disk, you are going to have something around 1/4 of the disk performance per LV (if all the LV are being stressed at the same time).

    On the other side, if you are looking for data stripping[1], you need to use different physical disk (instead of Logical volumes over a single disk).

    [1] - http://en.wikipedia.org/wiki/Data_striping

  • Bill_Buros
    Bill_Buros
    151 Posts

    Re: Java workload -- Configuration and scaling of storage devices as disks ?

    ‏2013-05-14T20:47:08Z  
    • harshGN
    • ‏2013-05-14T17:20:05Z

    Hi Bill,

                  It's great to hear from the community. I have posted my explanation to Breno above regards my task to be achieved. 

     

    Thanks

    Harsha

    Thanks Harsha.     Could you send me an email directly (wmb at us.ibm.com)?    We probably can help you more directly answer some of your questions and have a call together in the coming days...