IBM developerWorks: Wikis - AIX

nstress

These are a number of programs to stress your CPU, memory, disks and other features.

There are a number of uses:

Soak testing = check a new machine/disk to remove early life failures
Prove performance of machine upgrades or alternative disk configurations
Learn performance monitoring and tuning
For example I run a Performance Tuning Master Class and need to quick set up many different workloads and problems to be solved. With a 20 line shell script and these tools I don't have to spend a week of setup time.

The programs are:

Name	Purpose
ncpu	hammers the CPUs (can be slowed down to use a percentage)
ndisk	hammers the disks (can be slowed down to use a percentage)
ndiskaio	same as ndisk but does Asynchronous I/O (AIO handling changed in AIX 5L so not currently available)
ndiskmio	same as ndisk but uses Modular IO AIX Expansion pack library it is assumed this is installed (experimental not currently available)
nmem	hammers or touches memory
nipc	tests shared memory, semaphores and shared messages - take 1 CPU
nlog	generates output like error messages
nfile	creates, writes and deletes files to push the JFS log hard
ipctest	Manually test IPC
createfs.sh	Script to create the filesystems used by the below scripts - you will need to edit this for your system
dbstart.sh	Script to start a fake database RDBMS - you will need to edit this for your system
webstart.sh	Script to start a fake webserver - you will need to edit this for your system

Download the programs:

This tar file contains all the binaries and scripts.
It is compiled on AIX 5.1 but as these are very simple C programs and AIX is binary compatible these should run on and AIX 5L version.
Download: nstress_April2006.tar
Download: nstress_aix53ml4.tar exactly the same but compiled for AIX 5.3 ML4
Download: nstress_lop.tar exactly the same but compiled for Linux on POWER - I assume they work for any 2.6 based Linux but not checked.
Download: ndisk64_v62.tar new (March 2009) ndisk64 that handles large files (more then 2 GB) but only if your user has file size hard large enough (use: ulimit -f unlimited - or - edit /etc/security/limits fsize_hard for this user and mandatory relogin). Also includes rancheck, run ndisk64 with the -l option to create a NNNNNN.log file, rename the log to "checkfile" and run rancheck. It will show you the distribution of lseeks in the file to check for randomness (up to 60 GB). with new randomiser for multiple threads.

Manual

Below is the "Manual" for these tools.

Warrentee = none

Its strickly at your own risk.
Tested on AIX 5L.
If you run these as a regular user - no harm can be done but as the root user they can be danagous.

Warning

Note: ncpu running as root will try to boost its priority.
This will effectively lock out an entire CPU(s) (if the slowing down options are not used).
Which can be a good thing, if that is what you want - this effectively removes the CPU from your configuration.

Feedback

Put questions to http://www.ibm.com/eserver/pseries/community/ and take the Performance Tools Forum link.
If you develop a good script - why not put them up here.

Note: Most commands now have the following options

Options	Comments
-o	This allows the process to rename itself so that it looks like something else when it is running. For examplea database. * This HAS to be the LAST option on the line. This also fools programs like ps and nmon. I call the masquerading.
-z	This cause the program to not run flat out and so behave like a user driven command. This is performed at the millisecond level and is quick effective.
-s	This specifies the maximum time in second to run. ALWAYS use this to make sure the programs stop, otherwise you can slow down a machine forever (or reboot), also if you "over cook" the workload by starting too many programs and the machine stops responding - you can at least just wait for the timeout for recovery to happen (better than a reboot).

ncpu -h output

Usage: ncpu version 9.0 hammers the cpu(s)
Note: root users get a priority boost = effectively removes the CPU(s)

Hint hammer CPU mode: ncpu -p procs [-z percent] [-s secs] [-h secs] [-o "cmd"]
        -p procs   = number of copies of cpu to start (max=256)
        -z percent = Snooze percent - time spent sleeping (default 0)
        -s seconds = Seconds maximum run time (default no limit)
        -h seconds = Seconds to sleep after each second of run time
        -o "cmd" = Other command - pretend to be this other cmd when running
                Must be the last option on the line

Hint CPU counter mode: ncpu -c

ndisk -k output

Usage: ndisk version 5.9
Complex Disk tests - sequential or random read and write mixture
ndisk   -S          Seqential Disk I/O test (file or raw device)
        -R          Random    Disk I/O test (file or raw device)

        -t <secs>   Timed duration of the test in seconds (default 5)

        -f <file>   use "File" for disk I/O (can be a file or raw device)
        -f <list>   use separated list of filenames (max 16) [separators :,+]
                        example: -f f1,f2,f3  or -f /dev/rlv1:/dev/rlv2
        -F <file>   <file> contains list of filenames, one per line
        -M <num>    Mutliple processes used to generate I/O
        -s <size>   file Size, use with K, M or G (mandatory for raw device)
                        example -s 256M   or   -s 4G
        -r <read%> Read percent min=0,max=100 (default 80 =80%read+20%write)
                        example -r 50 (-r 0 = write only, -r 100 = read only)
        -b <size>   Block size, use with K, M or G (default 4KB)
        -O <size>   first byte offset use with K, M or G (times by proc#)
        -b <list>   or use a colon separated list of block sizes (536876292 max)
                        example -b 512:1k:2K:8k:1M:2m
        -q          flush file to disk after each write (fsync())
        -Q          flush file to disk via open() O_SYNC flag
        -i <MB>     Use shared memory for I/O MB is the size(max=536876100 MB)
        -v          Verbose mode = gives extra stats but slower
        -l          Loging disk I/O mode = see *.log but slower still
        -o "cmd"  Other command - pretend to be this other cmd when running
                        Must be the last option on the line
        -K num      Shared memory key (default 0xdeadbeef) allows multiple programs
        -p          Pure = each Sequential thread does read or write not both
        -P file     Pure with separate file for writers
        -z percent  Snooze percent - time spent sleeping (default 0)

Pre_POSIX AIO and needs fixing up 
Asynchronous I/O tests (AIO)
        -A         AIO  use -S/-R, -f/-F and -r, -M, -s, -b to determine I/O types
                (JFS file or raw device)
        -x <min>   minimum outstanding Async I/Os (min=0 and min<max)
        -X <max>   maximum outstanding Async I/Os (max=1024)
     see Above -f <file>  -s <size>   -R <read%>  -b <size>

For example:
        ndisk -f bigfile -S -r100 -b 4096:8k:64k:1m -t 600
        ndisk -f bigfile -R -r75 -b 4096:8k:64k:1m -q
        ndisk -F filelist -R -r75 -b 4096:8k:64k:1m -M 16
        ndisk -F filelist -R -r75 -b 4096:8k:64k:1m -M 16 -l -v

        ndiskaio for Asynch compiled in version
        ndiskaio -A -F filelist -R -r50 -b 4096:8k:64k:1m -M 16 -x 8 -X 64

nmem -h output

Useage: nmem version 2.0
Hint: scans memory
        nmem -m Mbytes [-s MaxSeconds] [-z percent][-o "cmd"]
                Mbytes     = Size of RAM to use
                MaxSeconds = maximum time of the test
                percent    = Snooze percent (process sleeps for the percentage o
f the time

        Memory speed test with increasing memory sizes
        - may highlight CPU cache sizes
        nmem -c [-s MaxSeconds]
                MaxSeconds = maximum time of the test (default 60)
Example:
        nmem -c -s 1000                  - cycle through tests (maximum of 1000 seconds)
        nmem -m 1 -s 60                  - 1 MB for 1 minutes (default)
        nmem -m 6 -o "sally -x"          - 6 MB pretend to be process sally with  parameter -x

nlog -h output

Usage: nlog version 2.0
Hint: generates a log file at a steady rate to standard error
        nlog -k Kbytes -s Seconds -m MaxSeconds -o "cmd"
        nlog -k 1 -s 60                 is the default
Example:
        nlog -k 2 -s 1 -m 600            - 2 K per second for 10 minutes
        nlog -k 2 -s 1                   - 2 K per second forever
        nlog -k 2 -s 1 -o "bert -x"      - 2 K per second pretend to be process bert with parameter -x
        nlog -k 1 -s 60 -m 300           - 1 K per minute for 5 minutes
        nlog -k 200 -s 1 -m 3600         - 200 K per second for an hour

        Total of 7442 error lines

nipc -h output

This runs processes that communicate with shared memory with control via a semaphore and then back again with a message queue.

Usage: nipc version 2.0
nipc: hammers inter-process communication (IPC) 
    that is shared memory, semaphores and message queues

hint: nipc -p procs -s seconds

nfile -h output

Usage: nfile version 2.0
Hint: creates and deltes files = generates JFS log file work
nfile -d directory [-k Kbytes] [-c Files] [-m MaxSeconds] [-z percent] [-o "cmd"]
        -d directory  - top level directory
        -k Kbytes     - size of the files to create in KB (default 4KB)
        -c Files      - number of files to maintain (+/- 10) (default 4096)
        -m MaxSeconds - stop of so many seconds
        -z percent    - percent of time to sleep/snooze
        -o "cmd"    - pretend to be a different cmd (must be the last option)
Example:
        nfile -d mydir -k 1 -c 10000 -m 600
        nfile -d /tmp/files -k 64 -o "bert -x"