Test workloads

Edit online

Different types of workload have been used in the performed tests.

The listed sections describe the used types of workloads in more detail:

Database BI workload

SwingBench was used for the database BI workload. SwingBench is a free load generator (and benchmark) designed to stress-test an Oracle Database. For more information, see: 
https://www.dominicgiles.com/swingbench.html

SwingBench comes with four customized benchmarks: OrderEntry, SalesHistory, CallingCircle, and StressTest. For the overcommitment project discussed in this publication, the SalesHistory benchmark was selected because it emulates a business intelligence (BI) workload.

SwingBench is driven by an external client that communicates with the Oracle Database. In our case, the client was installed on a x86-64 server under Linux™. The Oracle Database was installed within a virtual system running Linux on System z®. For details on the virtual system, see Virtual system configuration.

Communication between client and database was established by means of JDBC type IV connectors over an IP network.

The Oracle Database was configured using ASM as disk storage management tool for a data disk group and a log disk group:

  • two 256 GiB SCSI data disks within a data disk group
  • three 10 GiB SCSI log disks within a log disk group
Table 1 details the parameters that are set for the Oracle Database.
Table 1. Oracle Database parameters

Table with three columns describing the Oracle Database parameters
Parameter Name Value
SGA size sga_target 200 GiB
PGA size pga_aggregate_target 45 GiB

The SwingBench shwizard tool was used to create the database elements required for the SalesHistory workbench such that the total size of disk space used for the SalesHistory execution was about 300 GiB.

The SalesHistory workload was customized for producing specific loads on memory and I/O resources, as shown in Table 2.
Table 2. SwingBench SalesHistory workload parameters

Table with three columns describing the SwingBench SalesHistory workload parameters
Parameter Name Value
number of SwingBench users -uc 16

Database BI workload performance metric: The SalesHistory workload randomly selects the transactions it performs from a set of transaction types of varying complexity. For that reason, the Database BI workload performance metric was defined as a weighted transaction rate (transactions per second), using weight factors that were determined from the execution times that the various transaction types achieved during the reference execution. For each test execution, the weighted transaction rate then was determined, taking the transaction rates reported separately for each transaction type by the workload driver, factoring in the respective transaction type specific weight factor and summing up these weighted transaction rates.

Transactional WAS workload

For the transactional WAS workload, an adaptation of the DayTrader workload to the WebSphere® Application Server (WAS) was used.

DayTrader is an Open Source benchmark application emulating an online stock trading system. Users can log in and view their accounts or portfolios. In the Quotes/Trade section they can buy or sell stock shares out of their portfolios. The benchmark application is currently hosted by the Apache Geronimo project and was originally developed by IBM® as the WebSphere Trade performance benchmark. This benchmark was donated to the Apache Geronimo project in 2005.

DayTrader is an end-to-end Java™ Enterprise Edition (J2EE) web application composed of several Java classes, Java Servlets, Java Server Pages, Web Services, and Enterprise Java Beans (EJB). Thus, it is an ideal benchmark application for measuring the scalability and performance of a J2EE application server like IBM WebSphere Application Server (WAS). The presentation layer is based on Java Servlets and Java Server Pages, whereas the back end business logic uses Java database connectivity (JDBC), Java Message Service (JMS), EJB, and Message Bean techniques. For our tests, the Trade Database was hosted on a DB2® database server and was accessed via JDBC from the application server. For more details on the DayTrader benchmark, see DayTrader - a more complex application.

Transactional WAS workload variants: The transactional WAS workload was executed at two levels, medium and high, using two suites of z/VM® virtual systems. Each suite was composed of three virtual system running Linux on System z, where each virtual system had one of the following components installed:

  • IBM Http Server (IHS)
  • WebSphere Application Server (WAS)
  • DB2 database server (DB2)
For details on the virtual systems, see Virtual system configuration. For details on the software components, see Software.
The difference between the medium and the high level is shown in Table 3.
Table 3. Transactional WAS workload parameters

Table with three columns describing the Transactional WAS workload parameters
Level Component # virtual CPUs
medium WAS 2
DB2 1
high WAS 4
DB2 2
WebSphere Application Server configuration: WebSphere Application Server was installed using the Network Deployment (ND) variant. Table 4 details the parameters that were set for both levels of the WebSphere Application servers.
Table 4. WebSphere Application Server parameters

Table with four columns describing the WebSphere Application Server parameters with their level, parameter description, parameter name, and the value.
Parameter Name Value
Min. ORB pool size minOrbPool 10
Max. ORB pool size minOrbPool 50
Min. WebContainer pool size minWebPool 50
Max. WebContainer pool size minWebPool 50
Min. Default pool minDefaultPool 5
Max. Default pool minDefaultPool 150
Keep alives enable keepAliveEnabled TRUE
Min. JVM java heap size minHeap 2048
Max. JVM java heap size maxHeap 2048
Java garbage collection policy default WAS 8.5 with Java 7 gencon
Min. TradeDS pool minTradeDSPool 200
Max. TradeDS pool maxTradeDSPool 300
Min. Broker pool minBrokerPool 1
Max. Broker pool maxBrokerPool 150
Min. Streamer pool minStreamerPool 1
Max. Streamer pool maxStreamerPool 150
DayTrader workload configuration: Table 5 lists the parameters applied for both levels of the DayTrader workload.
Table 5. DayTrader workload parameters

Table with four columns describing the DayTrader workload parameters with parameter description, parameter name, and the value.
Parameter Name Value
run time mode runTimeMode 1 (direct JDBC)
order processing mode n/a Asynchronous_2-Phase
max. number of users minOrbPool 15000
max. number of quotes minWebPool 10000

Workload driver configuration: A workload driver was used to emulate user interactions that generate load against the DayTrader application. The workload driver was customized to run http transactions against the DayTrader environment. These transactions performed changes on the DayTrader database by means of Web Services provided through the WebSphere environment.

The workload driver was installed on a client system server. For details on the client system server, see Client system server. For details on the software components, see Software.

Table 6 lists the workload driver parameters that were applied:
Table 6. Workload driver parameters

Table with four columns describing the workload driver parameters with their level, parameter description, parameter name, and the value.
Parameter Name Value
think time [ms] thinktime 500
number of simultaneously running clients -c 500
element delay percent -e 0
dynamic cookies -D on

Transactional WAS workload performance metric: The transaction rate (transactions per second) determined over the complete workload execution time was used as metric for the performance of the transactional WAS workload.

File system I/O workload

For the file system I/O workload, the fio tool was used. fio is an I/O tool intended to be used both for benchmarking, and for stress/hardware verification. For more information, see the fio readme file.

fio has support for various types of I/O engines (such as sync or libaio), I/O priorities, throughput, forked or threaded jobs, and much more. It can work on block devices as well as files. fio displays all sorts of I/O performance information.

File system I/O workload variants: For our tests, four variants of the file system I/O workload were executed within separate Linux on System z instances running within z/VM virtual systems. For details on the virtual systems, see Virtual system configuration.
For each variant, the file system I/O workload was varied such that specific loads on memory and I/O resources were produced. The file system workload parameters are listed in Table 7.
Table 7. File system I/O workload parameters

Table with three columns describing the File system I/O workload parameters with their parameter description, parameter name, and the medium level and high level values.
Parameter Name Values
medium level high level
page-cached direct I/O page-cached direct I/O
enable direct I/O direct 0 1 0 1
type of I/O engine ioengine sync libaio sync libaio
I/O depth iodepth 1 4 1 4
Number of fio jobs numjobs 8 16
Number of files nrfiles 1
file size filesize 1536 MiB
block size bs 32 KiB
type of I/O pattern rw randrw
percentage of read rwmixread 80

Types of I/O operations: The page cache is memory that is maintained by the Linux kernel for the purpose of caching data related to user process I/O operations. If the use of the page cache is requested with a particular I/O operation (the default case), then the Linux system transparently caches I/O related data in the page cache. With write operations, the Linux system decides when to transfer data from the page cache to the respective file. For example, I/O operations may be queued to be performed when respective I/O and system resources are available. Of course, the use of the page cache implies a higher memory usage. On the other hand, if direct I/O is used, then the data transfer by the fio user process goes directly from the application buffers to the data files, avoiding the use of the page cache.

The direct parameter controls the use of the page cache:

  • direct=1 requests fio to perform direct I/O operations
  • direct=0 requests fio to perform page cached I/O operations

In addition, for direct I/O operations, the use of native Linux asynchronous I/O was requested by means of the ioengine parameter, and the number of parallel I/O operations per process was limited by means of the iodepth parameter.

Files used for I/O operations: The number of files in use by a particular fio workload was controlled by means of the numjob and the nrfiles parameters.

For the medium level file system I/O workloads, eight fio processes were used. For the high level file system I/O workloads, sixteen fio processes were used. In any case, each process was requested to only use one file. The distinction between medium and high levels was that the medium level workload operated on eight files, while the high level workload operated on sixteen files.

The characteristics of the files where set to a file size of 1536 MiB, and a block size of 32 KiB.

The last two lines in Table 7 show that for all fio executions random read and write operations were requested by means of the rw parameter, with the rwmixread parameter requesting a mixture of 80 % read and 20 % write operations.

File system I/O workload performance metric: The sum of the fio read rate and the fio write rate (both in byte per second as reported by the workload driver) for all fio processes, determined over the complete workload execution time, was used as metric for the performance of the file system I/O workload.

Java workload

For the Java workload, a Java program was used that executed a number of parallel threads, with each thread performing the same kind of transactions.

For our tests, two levels, medium and high, of the Java workload were executed, each within a separate Linux on System z operating system instance running within a z/VM virtual system. For details on the z/VM virtual systems, see Virtual system configuration.

The Java workload itself was identical for both levels. However, for the medium level workload, one virtual processor was used, whereas for the high level workload, two virtual processors were used.

The Java workload parameters are listed in Table 8.
Table 8. Java workload parameters

Table with three columns describing the Java workload parameters with their parameter description, parameter name, and the medium level and high level values.
Parameter Name Values
Java heap size (fixed) max_heap 3200 MiB
init_heap 3200 MiB
Java garbage collection algorithm -Xgcpolicy optthruput

Java workload performance metric: The sum of the transaction rate (transactions per second) for all processes, determined over the complete workload execution time, was used as metric for the performance of the Java workload.

Network workload

For the network workload, the uperf network performance tool was used. For details, refer to uperf - A network performance tool.

Client and server part were spread into multiple processes, with each pair of processes exchanging messages through separate TCP/IP network connections.

The network workload parameters are listed in Table 14.

Table 9. Network workload parameters

Table with three columns describing network workload parameters with their parameter description, parameter name, and the medium level and high level values.
Parameter Name Values
Number of processes n/a 250
Protocol n/a tcp

Network workload performance metric: The average data exchange rate [MiB/s] as reported by the workload driver aggregated for all processes and over the complete workload execution time was used as metric for the performance of the network workload.

Combined workload set

The combined workload set is the combination of all levels of the individual workloads:

  • Database BI workload
  • Transactional WAS workload (medium and high level)
  • File system I/O workload (page-cached and direct I/O, each at medium and at high level)
  • Java workload (medium and high level)
  • Network workload (medium and high level)