Although iSCSI technology is not a really news-breaking development in the storage and Linux world, it took some time until it found its way to Linux on POWER and more than that to AIX 5L.
iSCSI itself means to encapsulate SCSI commands and data into LAN frames and transfer them over a standard network. The important difference to file-based network storage like NFS is, that it is still a block oriented technology which may offer improved performance compared to other technologies in certain areas.
The need for shared storage, meaning that different systems or partitions can access the same physical devices, grow as more applications rely on this technology. Oracle's Real Application Cluster (RAC) or high availability products like Tivoli System Automation (TSA) are maybe two of the most well know representatives of this kind of applications.
On the other hand it is a fact that shared storage is still quite expensive - the disk drives, the cabling, the storage subsystems, the switches and their ports, the host bus adapters and so on. And sometimes you simply don't have either the whole infrastructure or not enough of it (e.g. no free switch ports on the FC switch) but you might have the need for shared storage for testing, devlopment or even productive environments.
In environments which requires a low cost but shared storage iSCSI might be the right solution providing block level I/O over a network.
Nowadays there're a lot of hardware vendors offering iSCSI solutions (like NettApp or IBM - you can add your prefered vendor) which offers different solutions for different szenarios. On one hand these disk subsystems are much cheaper than SAN devices and are highly specialised on the tasks they should fullfill. But on the other hand a standard Linux distribution (talking mainly about SLES and RHEL) comes with everything you need to implement iSCSI in your environment. Just try it.
The following sections will provide a brief overview of the iSCSI implementation which can be found in the supported Linux on POWER distributions, namely SLES9/10 and RHEL4. It's intention is not to give you a complete guide to the iSCSI implementation but to provide a kickstart like documentation for using iSCSI on Linux on POWER.
Some words about planning...
Carefull planning of an iSCSI infrastructure is vital like it is for all other storage infrastructures like SAN. Just let's think a minute or two about it - and I know that this list is not complete (feel free to add points if you'll like):
- Use a seperate LAN for iSCSI (prefered) whenever possible for the sake of performance and security.
- Know your application and the storage requirements in terms of I/O because iSCSI is slower than native technologies.
- Think about which system or LPAR or application should get access to which part of the iSCSI devices and implement correct access rules for them - otherwise you could kill your data or get effects not wanted.
- What type of network do you have or plan to use for iSCSI - although I've found no real recommendation on this I would say use Gb Ethernet. Fast Ethernet will work but much, much, much slower. By the way virtual Ethernet within a IBM System p box works, too.
- Think about the kind of devices for the clients. You can use whole disks (like sda), disk partitions (like /dev/sda1) or flat files in a file system. You can mix it as required.
- Think about the disk technology for the iSCSI server (Target) - i.e. local SCSI or FC disks.
Some words about definitions...
Ok, now you've thought about the general setup of your environment, you just should be sure not to get lost in terminology (like I did the first time).
- The iSCSI Target is the server which is connected to the physical storage.
- The iSCSI Initiator is the client which connect to the Target and issues commands and sends/receives data from it.
Setting up the iSCSI Target
Starting with Novell SuSE Enterprise Server 10 (SLES10) the iSCSI Target implementation is included in the distribution from Novell. Red Hat will provide the iSCSI Target somewhere in RHEL5 but don't ask for the definitve date.
As of today if you'll like to test / implement iSCSI Target you must use SLES10!
Check if you'll have the iscsitarget rpm installed.
bc1-js21-1:~ # rpm -qa | grep iscsitarget
iscsitarget-0.4.13-14.8
If not install it the way you prefer - using YaST/YaST2 or the command line.
 | Note...
If you don't already use SLES you can download a 30 day trial of SLES10 at the following side:
SLES10 Eval Download  |
As a first step you should decide what will be an iSCSI target. As mentioned you'll have the choice between whole disks, disk partitions or flat files or an mix of all those types.
In my szenario the system which will act as the iSCSI Target will export two SAN LUNs to two systems which are not connected to the SAN.
bc1-js21-1:~ # fdisk -l
...
...skipping some lines here...
...
Disk /dev/sda: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
Disk /dev/sdb: 12.8 GB, 12884901888 bytes
64 heads, 32 sectors/track, 12288 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdb doesn't contain a valid partition table
bc1-js21-1:~ #
So /dev/sda and /dev/sdb are my iSCSI Target candidates.
Now you know which devices you want to export you must tell the system to do so. You can do YaST/YaST2 for this task which is quite comfortable.
YaST @ bc1-js21-1 Press F1 for Help
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x YaST Control Center x
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
lqqqqqqqqqqqqqqqqqqqqk lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
xSoftware x xNTP Client x
xHardware x xNetwork Services (xinetd) x
xSystem x xProxy x
xNetwork Devices x xRemote Administration x
xNetwork Services x xRouting x
xSecurity and Users x xSamba Server x
xMiscellaneous x xSlpServer x
x x xTFTP Server w
x x xWOL x
x x xWindows Domain Membership x
x x xiSCSI Initiator x
x x xiSCSI Target v
mqqqqqqqqqqqqqqqqqqqqj mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
[Help] [Quit]
On the iSCSI Target Overview screen on the topic Service you can select if the service should be started when the system is started or if you prefer to start it manually. In addition if you're using a firewall on that system you can tell YaST to open the corresponding port for the iSCSI service (Port 3260 per default).
The topic Global offers the possibility to configure authentication (incoming - which means from the clients and/or outgoing - which means to the clients).
Finally in the section "Targets" you can configure the target devices. First, give it a name, then optionally an identifier, specify the LUN and finally point to the correct device.
| Please note...
The namin scheme of an iSCSI target has the requirement that it must be globally unique - well ok, make some sense. The name should be in the format: iqn.<yyyy-mm>.<tld.domain.some.host>[:<identifier>] where
...iqn stands for iSCSI Qualified Name...
...<yyyy-mm> is the date (year and month) at which the domain is valid - best use before  ?! ...
...<tld.domain.some.host> is the reverse order of the domain name...
...and optional you can specify an identifier (like data)... |
I've had some discussions with colleaques concerning the naming of iSCSI devices - like does it make sense or not. Beside this question you can assign each and every name to an iSCSI device (like blah:blah) but for consistency reasons and to be as unique as possbile I recommend to use the default naming scheme (e.g. iqn.com.ibm.de.stuttgart.bc1-js21-1:blahblah).
You're asked to configure authentication (if required/desired) for the device and after you've made all required changes press Finish to - hmm - finish the configuration.
You'll have to device whether the service should be restarted which means stopped and started! It might be important that a reload is not available yet!
And that's it!
| Please note...
To be honest there're more options available than you can configure using YaST. If you want to know which options are implemented either study the man page of the configuration file /etc/ietd.conf or have a look in that file because it comes well commented. |
After you've finished the configuration you can see that an additional kernel module was loaded and that the iSCSI Enterprise Target daemon (ietd) was started.
bc1-js21-1:~ # lsmod | grep iscsi
iscsi_trgt 114272 6
bc1-js21-1:~ # ps aux | grep [i]etd
root 4370 0.0 0.0 3372 812 ? Ss Sep29 0:00 /usr/sbin/ietd
To get a list of all active sessions, have a look at /proc/net/iet/session.
bc1-js21-1:~ # cat /proc/net/iet/session
tid:10 name:com.ibm.de.bc1-js21-1:test2
sid:630020162781219 initiator:iqn.1987-05.com.cisco:01.46997b83a0f6
cid:0 ip:9.154.2.113 state:active hd:none dd:none
tid:4 name:iqn.2001-04.com.example:storage.disk2.sys1.xyz
sid:630020162781220 initiator:iqn.1987-05.com.cisco:01.46997b83a0f6
cid:0 ip:9.154.2.113 state:active hd:none dd:none
sid:630020162781189 initiator:iqn.1987-05.com.cisco:01.e57ae2935f24
cid:0 ip:9.154.2.119 state:active hd:none dd:none
sid:630020162781187 initiator:iqn.1987-05.com.cisco:01.da95b2e8ba44
cid:0 ip:9.154.2.122 state:active hd:none dd:none
tid:3 name:iqn.2006-10.com.ibm.de.stuttgart.bc1.js21.1:flatfile
sid:630020162781221 initiator:iqn.1987-05.com.cisco:01.46997b83a0f6
cid:0 ip:9.154.2.113 state:active hd:none dd:none
sid:630020162781199 initiator:iqn.1987-05.com.cisco:01.da95b2e8ba44
cid:0 ip:9.154.2.122 state:active hd:none dd:none
sid:630020162781198 initiator:iqn.1987-05.com.cisco:01.e57ae2935f24
cid:0 ip:9.154.2.119 state:active hd:none dd:none
tid:2 name:iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:tiebreaker
sid:630020162781222 initiator:iqn.1987-05.com.cisco:01.46997b83a0f6
cid:0 ip:9.154.2.113 state:active hd:none dd:none
sid:630020162781190 initiator:iqn.1987-05.com.cisco:01.da95b2e8ba44
cid:0 ip:9.154.2.122 state:active hd:none dd:none
sid:630020162781185 initiator:iqn.1987-05.com.cisco:01.e57ae2935f24
cid:0 ip:9.154.2.119 state:active hd:none dd:none
tid:1 name:iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
sid:630020162781223 initiator:iqn.1987-05.com.cisco:01.46997b83a0f6
cid:0 ip:9.154.2.113 state:active hd:none dd:none
sid:630020162781191 initiator:iqn.1987-05.com.cisco:01.da95b2e8ba44
cid:0 ip:9.154.2.122 state:active hd:none dd:none
sid:630020162781186 initiator:iqn.1987-05.com.cisco:01.e57ae2935f24
cid:0 ip:9.154.2.119 state:active hd:none dd:none
That's it on the target side - for a really simple configuration (everyone can access the devices, no addl. performance etc.).
Setting up an iSCSI client
Setting up the client is as easy as the target. First of all check if all required rpm's are installed.
On SLES9/10 this is linux-iscsi.
bc1-js21-1-lpar3:~ # rpm -qa | grep iscsi
linux-iscsi-4.0.1-88.26
On RHEL4 it is called iscsi-initiator-utils.
[root@op710-1-lpar1 ~]# rpm -qa | grep iscsi
iscsi-initiator-utils-4.0.3.0-4
Next you must configure the service. To do this, open /etc/iscsi.conf with an editor of your choice. Once again it is well commented and comes with a quite good manpage.
The most simple setup for iSCSI client is to just set the DiscoveryAddress to the IP address or hostname of the iSCSI Target.
...
DiscoveryAddress=9.154.2.89
...
That's it - for this example. In a "real" world example you might also specify user credentials for authentication. Save the file and start the iscsi-daemon.
[root@op710-1-lpar1 ~]# service iscsi start
Checking iscsi config: [ OK ]
Loading iscsi driver: [ OK ]
Starting iscsid: [ OK ]
Now you can check what the iSCSI daemon has found with the iscsi-ls command.
[root@op710-1-lpar1 ~]# iscsi-ls
*******************************************************************************
SFNet iSCSI Driver Version ...4:0.1.11-3(02-May-2006)
*******************************************************************************
TARGET NAME : iqn.2006-10.com.ibm.de.stuttgart.bc1.js21.1:flatfile
TARGET ALIAS :
HOST ID : 43
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 9.154.2.89:3260,1
SESSION STATUS : ESTABLISHED AT Mon Oct 2 10:04:59 CEST 2006
SESSION ID : ISID 00023d000001 TSIH 25
*******************************************************************************
TARGET NAME : iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:tiebreaker
TARGET ALIAS :
HOST ID : 44
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 9.154.2.89:3260,1
SESSION STATUS : ESTABLISHED AT Mon Oct 2 10:04:59 CEST 2006
SESSION ID : ISID 00023d000001 TSIH 26
*******************************************************************************
TARGET NAME : iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
TARGET ALIAS :
HOST ID : 45
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 9.154.2.89:3260,1
SESSION STATUS : ESTABLISHED AT Mon Oct 2 10:04:59 CEST 2006
SESSION ID : ISID 00023d000001 TSIH 27
*******************************************************************************
Now you can use the devices like you would do with a "normal" block device locally connected - e.g. create partitions and so on.
| Please note...
SLES10 comes with the Open-iSCSI initiator from the Open-iSCSI project http://www.open-iscsi.org/. The configuration and setup of the initiator is completely different and the only usefull way to configure iSCSI initiator on a SLES10 system or LPAR is to use YaST / YaST2 -> Network Services -> iSCSI Initiator.
Furthermore you must/can use the iscsiadm command to get informations about the iSCSI setup. Here're three examples. The first one discovers the devices on the target ip 9.154.2.86 (the port number is optional). And the second example shows all targets which are available on this node. And example three shows the iSCSI configuration of a specific target.
bc1-js21-1-lpar3:~ # iscsiadm -mdiscovery -tst -p 9.154.2.89:3260
[303067] 9.154.2.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:tiebreaker
[5cbfc7] 9.154.2.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
...
bc1-js21-1-lpar3:~ # iscsiadm -m node
[297e2d] 192.168.128.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
[dbb383] 192.168.128.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:tiebreaker
[303067] 9.154.2.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:tiebreaker
[5cbfc7] 9.154.2.89:3260,1 iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
...
bc1-js21-1-lpar3:~ # iscsiadm -m node --record 5cbfc7
node.name = iqn.2006-09.com.ibm.de.stuttgart.bc1-js20-1:data
node.transport_name = tcp
node.tpgt = 1
node.active_conn = 1
node.startup = manual
node.session.initial_cmdsn = 0
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.reset_timeout = 30
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 0
node.session.iscsi.MaxConnections = 0
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 9.154.2.89
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.active_timeout = 5
node.conn[0].timeo.idle_timeout = 60
node.conn[0].timeo.ping_timeout = 5
node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
|
Setting up the iSCSI client on AIX 5L
Yes, you've seen right - AIX 5L can make use of the iSCSI Targets from a SLES10 client and this is true for a normal system or LPAR and in addition for the IBM Virtual I/O Server (VIOS).
You can find a quite good description in the IBM Infocenter - just follow the link below:
Here's a brief overview of the steps required on the AIX 5L client.
First check if you'll have the required filesets installed:
# lslpp -l | grep iscsi
devices.common.IBM.iscsi.rte
devices.iscsi.disk.rte 5.3.0.30 COMMITTED iSCSI Disk Software
devices.iscsi.tape.rte 5.3.0.30 COMMITTED iSCSI Tape Software
devices.iscsi_sw.rte 5.3.0.50 COMMITTED iSCSI Software Device Driver
devices.common.IBM.iscsi.rte
devices.iscsi_sw.rte 5.3.0.50 COMMITTED iSCSI Software Device Driver
See if the device is in the correct state.
# lsdev | grep "iSCSI Protocol Device"
iscsi0 Available iSCSI Protocol Device
Verify that the protocol device has the correct settings using {{smitty iscsi}. Namely the number of targets allowed and the iSCSI initiator name are important.
Change / Show Characteristics of an iSCSI Protocol Device
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
iSCSI Protocol Device iscsi0
Description iSCSI Protocol Device
Status Available
iSCSI Initiator Name [iqn.bc1-js21-2-vio.ho>
Maximum number of commands to queue to driver [200] +#
Discovery Policy file +
Maximum Targets Allowed [16] +#
Apply change to DATABASE only no +
Create an entry for each target in the iSCSI configuration file /etc/iscsi/targets. You must include one line per target.
...
9.154.2.89 3260 iqn.2006-10.com.ibm.de.stuttgart.bc1-js21-1:flatfile
...
And finally rescan the iSCSI protocoll device using cfgmgr.
# cfgmgr -l iscsi0
# lsdev -Cc disk
# lsdev -Cc disk
hdisk0 Available 01-08-00-1,0 SCSI Disk Drive
hdisk1 Available 01-08-01-1,0 SCSI Disk Drive
hdisk2 Available 01-11-01 1722-600 (600) Disk Array Device
hdisk3 Available 01-10-01 1722-600 (600) Disk Array Device
hdisk4 Available Other iSCSI Disk Drive
| Please note...
As mentioned this works for standard AIX 5L clients as well as for IBM Virtual I/O Server partitions with the correct filesets installed.
You can of course export the iSCSI devices on the VIOS to LPARs but I doubt that this makes much sense (but if you're interested you must set an PVID on the device in order to export it - use chdev -l hdisk4 -a pv=yes or chdev -dev hdisk4 -attr pv=yes) because you're adding an additional layer between the actual device and the client which will simply said kill performance! It's better to export the iSCSI devices via VLAN directly to the clients.
On the other hand it is quite handy to export a device to the VIOS to expand the storage capabilities there. |
Performance and usability
The next two sections will give you a feeling on the performance of iSCSI and discusses some usability szenarios.
Performance
Well, it's hard to say something general of the iSCSI performance beside the fact that it is slower than the physical backend (e.g. FC or SCSI). It depends on several things:
- LAN being used. Although you can use 100Mbps Ethernet it will drop performance dramatically. So use Gb Ethernet if possible.
- The storage backend used. If you'll have fast storage connected to the iSCSI Target you will benefit from that speed - if you don't have, don't expect that iSCSI will do some magic tricks - it simply depends on the backend system.
Find below some performance measurements of an iSCSI environment. For the I/O tests I've used xdd (http://www.ioperformance.com).
The setup of the environment is as follows:
- iSCSI Target: One js20 ppc970 BladeServer connected to a DS4300 Fibre Channel disk subsystem. It runs SLES10 and has one 2-Port Gb Ethernet controller.
- One Logical Partition connected to the iSCSI Target using virtual LAN and Shared Ethernet capabilities of the VIOS (i.e. it has no physical interface).
First let's have a look of the native performance on the iSCSI Target:
bc1-js21-1:~/xdd65.092206/bin # ./xdd.linuxsmp -op read -targets 1 /dev/sdb -rwratio 100 -queuedepth 1 -blocksize 512 -reqsize 256 -mbytes 2048 -passes 2 -verbose
...
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 1 2147483648 16384 19.178 111.974 854.29 0.0012 0.00 read 131072
TARGET PASS0002 0 1 2147483648 16384 19.370 110.869 845.86 0.0012 0.00 read 131072
TARGET Average 0 1 4294967296 32768 38.548 111.418 850.05 0.0012 0.00 read 131072
Combined 1 1 4294967296 32768 38.548 111.418 850.05 0.0012 0.00 read 131072
Ending time for this run, Fri Sep 29 13:21:56 2006
Now I've run the same test on the iSCSI Initiator:
bc1-js21-2-lpar2:~/xdd65.092206/bin # ./xdd.linuxsmp -op read -targets 1 /dev/sde -rwratio 100 -queuedepth 1 -blocksize 512 -reqsize 256 -mbytes 2048 -passes 2 -verbose
...
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 1 2147483648 16384 34.110 62.958 480.33 0.0021 0.00 read 131072
TARGET PASS0002 0 1 2147483648 16384 34.235 62.728 478.58 0.0021 0.00 read 131072
TARGET Average 0 1 4294967296 32768 68.344 62.843 479.45 0.0021 0.00 read 131072
Combined 1 1 4294967296 32768 68.344 62.843 479.45 0.0021 0.00 read 131072
Ending time for this run, Fri Sep 29 13:45:29 2006
As you can see the performance at the iSCSI Initiator is about 50% of the original performance achieved on the iSCSI Target.
And now - just to demonstrate - the performance of the same target device exported from the iSCSI Target to an IBM Virtual I/O Server which exports the device to a client (weird).
bc1-js21-1-lpar3:~ # ./xdd.linuxsmp -op read -targets 1 /dev/sdc -rwratio 100 -queuedepth 1 -blocksize 512 -reqsize 256 -mbytes 2048 -passes 2 -verbose
...
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 1 2147483648 16384 174.055 12.338 94.13 0.0106 0.00 read 131072
TARGET PASS0002 0 1 2147483648 16384 171.120 12.550 95.75 0.0104 0.00 read 131072
TARGET Average 0 1 4294967296 32768 345.174 12.443 94.93 0.0105 0.00 read 131072
Combined 1 1 4294967296 32768 345.174 12.443 94.93 0.0105 0.00 read 131072
Ending time for this run, Fri Sep 29 15:56:11 2006
...
As stated above, the performance drops dramatically! So don't ever think about doing this strange setup.
Usability
So the final question is: Can I use iSCSI? And the answer is yes!
I don't really discuss things like price etc. here. From my own experience you can use iSCSI to easily setup and use shared storage:
- For Oracle RAC (I don't believe it is officially supported by Oracle - but I may be wrong) building a cluster in a box (p5 510 running one VIOS, one LPAR as iSCSI Target and three LPARs as cluster nodes).
- For Tivoli System Automation running DB2 on a 2-node cluster
- You can think about other szenarios...
Reference and links
The main reference for the iSCSI Target and the iSCSI Initiator on Linux are the man pages and the configuration files:
- /etc/ietd.conf for the iSCSI Target
- /etc/iscsi.conf for the iSCSI Initiator
Another good source for informations is the iSCSI Target Wiki:
All things concerning benchmarking and stress testing are taken from the following web site:
