Note from the editor: There is limited USB support in AIX and only specific devices are supported. Other devices might work, but IBM does not support their usage. JFS2 file systems are not officially supported on USB devices, but you can try this at your own discretion.
Tasks that involve text file data extraction, where performance is of the essence and the text files are to be placed on temporary storage, requires some thought. Typically, one could consider using a RAM disk (file system), a USB memory stick, or perhaps on a more permanent basis using SSD (solid state disk). RAM disk file systems are extremely fast, but at some point you need to move the files from the file system since you could lose everything if the file system is unmounted or the system is rebooted. Plus, you need to have enough memory (pinned) to create the RAM disk in the first place. For SSD, this would be the ideal solution, but these are not as cheap as the USB memory devices (when the requirement is temporary moveable storage). Another consideration is that you cannot place an SSD or RAM disk in your pocket and re-plug it into another machine like you can with a USB (memory) stick. These sticks are also known as:
- pen drives;
- memory sticks;
- USB sticks;
- flash drives; and
- thumb drives.
In this article, I will demonstrate how to create a USB memory stick file system (hereafter called USB stick) and compare the different speeds on text extraction using a 2MB CSV file. I am using AIX 7.1 with a 4GB Sandisk memory stick. These USB sticks can come in sizes up to 256GB.
Memory sticks are useful in my opinion because:
- They are ideal for fast data access.
- They can be used in data transfer where network performance is poor.
- They are great for scratch or temporary file systems.
- They are cheaper than other alternatives.
The only downside is that they do have a limited life in that the write cycles on these drives typically last around 8-10 years, before the memory cells wear out. But as they are cheap to buy, this is really not a concern as you would no doubt replace them within that time period anyway.
The AIX 6.1 documentation states that it supports the following branded USB sticks:
However, I have used other branded and non-branded USB sticks with no issues found.
USB sticks are ideal for creating scratch file systems where you want to transfer a large amount of data, via courier, for quick extraction to a remote destination sie cheaply because there is no good network link. As well, you could use them for data manipulation of large text files. USB sticks are great for fast reads but does not offer good performance when writing to them; you need to keep in mind what type of processing task you use it for.
Mounting a USB stick
Be sure to have the correct fileset(s) installed before inserting your USB stick. The following file-sets are required to access a USB stick:
# lslpp -L devices.usbif.08025002.rte Fileset Level State Type Description (Uninstaller) devices.usbif.08025002.rte 22.214.171.124 A F USB Mass Storage Device Software # lslpp -L devices.common.IBM.usb.rte Fileset Level State Type Description (Uninstaller) devices.common.IBM.usb.rte 126.96.36.199 A F USB System Software
Insert the USB stick and run cfgmgr to discover it:
Once discovered, you have two devices (the block and raw device):
ls -l *usbms0 cr--r--r-- 1 root system 44, 0 Aug 22 18:59 rusbms0 br--r--r-- 1 root system 44, 0 Aug 22 18:59 usbms0
It is also shown in the devices output:
# lsdev |grep usb usb0 Available USB System Software usbhc0 Available 08-08 USB Host Controller (33103500) usbhc1 Available 08-09 USB Host Controller (33103500) usbhc2 Available 08-0a USB Enhanced Host Controller (3310e000) usbms0 Available 2.1 USB Mass Storage
For AIX 5.3, the first USB device is shown as
At this point, I could treat the USB stick as a raw volume and write data to it. However, in this demonstration, I will mount the USB stick as a file system. I feel there is no need to create a log device for this USB mount, since the purpose of a scratch file system is that it is temporary and thus speed is of the essence when transferring data. To create the USB stick file system use the following command:
# mkfs -V jfs2 -o ea=v2 /dev/usbms0 mkfs: destroy /dev/usbms0 (yes)? File system created successfully. 3927736 kilobytes total disk space. Device /dev/usbms0: Standard empty filesystem Size: 7855472 512-byte (DEVBLKSIZE) blocks
The previous command initialized the USB stick. Select "Yes" to destroy, or rather initialize, the device. The file system type is of jfs2 type. I do not believe that a type of jfs2 really matters for this task, as the file system is not going to use the jfs2 log device. But as this is the norm when creating file systems, I created it as a jfs2. To ensure the file system is scalable, I specify that with the 'ea' option. In this example, the device to use is the USB stick inserted, which is usbms0.
For AIX 5.3, there is no NOLOG option in the mount command.
Next, mount the file system specifying that is it to be mounted without a log device; be sure to create the directory it is to be mounted on first:
# mkdir /usb_mnt # mount -o log=NULL /dev/usbms0 /usb_mnt # df -g |grep usb /dev/usbms0 3.75 3.73 1% 4 1% /usb_mnt
If you wish to use a log device to mount the usb stick, use the inline log. This ensures that it is contained within the file system:
# mkfs -olog=INLINE,ea=v2 -Vjfs2 /dev/usbms0 mkfs: destroy /dev/usbms0 (yes)? logform: Format inline log for <y>?y File system created successfully. 3912376 kilobytes total disk space. Device /dev/usbms0: Standard empty filesystem Size: 7824752 512-byte (DEVBLKSIZE) blocks # mkdir /usb_mnt # mount -V jfs2 -o log=/dev/usbms0 /dev/usbms0 /usb_mnt
Unmounting a memory stick
Once you have finished using the a USB stick file system, unmount it, remove it, then delete the usbms0 device:
# umount /usb_mnt # rmdev -dl usbms0
Data on the stick
In this demonstration, I am using a CSV text file called plaks.txt. It is an extract from a database, partial contents of the file is shown below:
alpha,uk01w,12001,jan,-2 bravo,ge01w,98801,jan,-3 charlie,se01w,98111,jan,0 delta,my01w,18811,jan,4 echo,sg01w,34131,janq,2
Earlier I suggested that reading from a USB stick is quick but writing to the USB stick is slower compared to internal disks. This can be somewhat verified by doing an example copy. First, I copied a file from the USB stick to an internal disk. Then, I copied the same file from the internal disk back to the USB stick using the timex command to show the timings of the copy commands. Before both copy operations are carried out, the file systems have been unmounted and re-mounted, so that the operation does not use the file system cache and thus does not blur the timings.
First, the file is copied from the USB stick onto an internal disk to the
/holding directory, which resides on an internal disk:
# pwd /usb_mnt # timex cp plaks.txt /holding/ real 0.03
Next, I remove the file from the USB stick and unmount and remount the file systems.
Next, I copied the file back from
/holding to the USB stick.
# cd /holding # timex cp plaks.txt /usb_mnt/ real 0.06
The copy of the USB stick takes nearly half the time of the copy from an internal disk. I typically use the USB stick for text extraction. Using awk as an example on data extraction, I extract the pattern 'bravo' contained in the plaks.txt located on the USB stick:
# pwd /usb_mnt # timex awk '/bravo/' plaks.txt real 5.50
Doing the same operation on an internal disk:
# pwd /holding # timex awk '/bravo/' plaks.txt real 8.48
The text extraction is quicker on the stick; in fact, there is a difference of over 3 seconds. Here I am using a relatively small size text file. Though clearly the saving in time is greater when using larger sized exported file, which typically could be up to 8-10MB.
The above, in my opinion, justifies why I choose to use USB sticks for heavy test processing; however, as suggested earlier, if you need to write data you lose out big time. Here I create a 100MB file on the USB stick:
# pwd /usb_mnt # timex lmktemp myfile 100M myfile real 6.60
Next, I do it on an internal disk:
# pwd /holding # timex lmktemp myfile 100M myfile real 1.65
As you can see, there is a difference of over 5 seconds in the writing to a USB stick compared to an internal disk.
USB sticks are great for transferring data when there is no real alternative. I recall one incident, where one of our machines located in a remote region where the SAN disks were failing. These SAN disks contained work flow images. This machine did not have a sturdy network, and we wanted to get these images onto another remote machine for business users to be able to continue to work. The only method quickly available was to buy some 64GB USB sticks, mount them, and tar the images up on the USB sticks. The images were then extracted onto our other AIX box. This method worked with no data loss.
Mounting a previous initialized memory stick
If you are in possession of a memory stick that contains data what was previously mounted as a file system, to access it simply mount it. Be sure to create the mount point first. For example, assume the USB device is inserted in the second USB slot, then have the device discovered as usbms1:
# mkdir /usbstick # mount -o log=NULL /dev/usbms0 /usbstick
Using USB sticks is a great when you want to create a fast read access scratch file system for processing or maybe a raw device to dump some files into for transport to another AIX system. USB sticks offer one solution where you have a poor network, and you need to get a large amount of data transferred.
- The AIX 7.1 and AIX 5.3 Information Center is your source for technical information about the AIX operating system.
Get products and technologies
- Try out IBM software for free. Download a trial version, log into an online trial, work with a product in a sandbox environment, or access it through the cloud. Choose from over 100 IBM product trials.
- Follow developerWorks on Twitter.
- Participate in developerWorks blogs and get involved in the developerWorks community.
- Get involved in the My developerWorks community.
- Participate in the AIX and UNIX® forums:
Dig deeper into AIX and Unix on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Keep up with the best and latest technical info to help you tackle your development challenges.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.