IBM WebSphere DataPower B2B XB60 Performance Testing

Resolving The Problem

The primary purpose of performance testing is to determine what the maximum capacity of the B2B appliance is with regard to processing real-world data utilizing all of the services on the device, and then using that information for capacity planning and for deciding how much to scale to meet your specific connection and/or throughput requirements.

Understanding the maximum capacity of a single device can help you determine how many devices will be needed to support your throughput requirements. Performance results are very subjective with many factors including the following:

• Network latency
• Firewalls, routers and switches in the path of the flow
• Average file/payload size
• Peek Volume and measurement period
• Usage of data encryption
• Usage of data signatures
• Usage of message disposition notifications (Sync or Async)
• Usage of transport protocols
• Usage connection security like SSL or TLS
• Usage of authentication
• Usage of authorization
• Usage of processing policy
• Usage of transformation and/or validation
• Method used to measure throughput
• The number of services actively running
• Concurrent connections

With so many factors to consider, the matrix of possible test case variation is rather large. For this reason, IBM does not publish performance results. Any results obtained from IBM lab testing will be of little value with regard to how the appliance functions with your data in your environment.


Tools

There are many open source benchmarking tools that can be used to simulate both the starting point and landing point for your transactions flows. A good list can be found at the following URL:

http://en.wikipedia.org/wiki/Web_server_benchmarking

Testing best practices

1. Review and configure the B2B appliances to handle appropriate capacity that is expected in production. Capacity Planning should be done prior to any performance testing. See Capacity Planning for WebSphere DataPower B2B Appliance XB60 for more information.
   
2. Test between two separate B2B appliances (source/sending partner to target/receiving partner) on the same network subnet (e.g. no router or firewall between the two devices) to be sure data can be consumed and produced as fast as possible. Be sure all testing tools are also on the same sub-net as the B2B appliances. This allows you to establish a benchmark baseline that can guide you when making decisions on how to scale to meet your throughput needs.
3. When testing AS protocols or ebMS, test a few transactions without security and MDNs first, correct any configuration issues and then test with unsigned MDN’s, followed by testing with full security. If connection security is used enable SSL/TLS first followed by data security. This makes it easier to quickly isolate configuration issues.
4. If using multi-step in your flow to transform or otherwise manipulate data, keep your policy disabled until after you have completed the data exchange tests between both systems. In general, it is a good idea to run some tests without the policies and capture the results and then turn on the processing policy and run additional tests. This will give you a good idea about how much overhead your processing policy is causing.
5. Do not use automatic retry or resend logic that exists in the destinations of the B2B messaging protocols.
6. Set the Connection Timeoutvalue in the destinations to a low number, typically 30 seconds.
7. When sending data into a DataPower service from a simulated back-end, be sure your test harness on the back-end is capable of providing connection concurrency and that it can produce test files as fast as the front-side handlers can consume them. This allows you to adjust volume such that you can effectively affect the CPU utilization of the device. If this is not possible, then you may want to consider using a poller front-side handler (set to disabled), loading a specific file count into the polling location and then enabling the handler to pickup all of the files as fast as it can.
8. The B2B appliance receiving the B2B messages from the sender should have a back-side that is capable of consuming the files as fast if not faster then they are being sent by the B2B appliance. If this is not possible, a Multi-protocol Gateway service on the device can be configured as a simulated back-end and be set to consume and throw away the files that it receives and to respond with a 200 OK back to the B2B Gateway service.
9. The best way to measure throughput is with controlled test runs of a fixed number of documents, then looking at the B2B viewer for the time stamp of the first message in and then finding the time stamp for the last message out and calculating using this formula: TimeOUT - TimeIN (convert to seconds)= Elapse time in seconds. Number Docs processed / Elapse Time = TPS. By doing the measuring using this manual method, you do not have the overhead of turning on throughput monitors in the XB60.
10. Run each test three times and use the average of the three tests as your final result for each testing variation. This will provide you with a more accurate value then any single test.
11. Run a Purge Now process on the B2B Appliance between each test run to keep the payload store and metadata store clean, preventing drive space issues.
12. If the intent is to use multiple services on the B2B appliance when in production your configuration must reflect this and your testing will need to send to front-side handlers of all services at the same time, this will give you a more accurate result when determining the saturation point of the device. Saturation happens when CPU utilization reaches in the upper 90’s to 100%.
13. The best way to monitor the B2B appliance is to watch the system usage and CPU status of both the sending and receiving devices, when either hit between 95% and 100% then this is the maximum threshold you can expect from the device.
14. Uncheck the "Use an Archive Monitor" during performance testing. Configure B2B Gateway / Archive Tab. When this is turned on, some transactions might be buffered until all the archive tasks are completed. This would preclude accurate testing of the maximum throughput of the appliance.

Information on monitoring methods for DataPower can be found at IBM’s developerWorks web site. For example, see Monitoring WebSphere DataPower SOA Appliances.

MRTG is a common tool monitoring SNMP information and can be obtained from the following URL: http://oss.oetiker.ch/mrtg/

Once a baseline result is established for the B2B appliances in isolation, there is a good starting point to for understanding and isolating bottlenecks that may exist within the end-to-end B2B flow.

Since the end-to-end performance of the B2B flow is going to be only as fast as the slowest link in the testing chain, the baseline value from the systems in isolation is really a sanity check that the devices are meeting your minimum requirements.

When firewalls, routers, the Internet, and network latency are added to the mix, Service Level Monitoring (SLM) can be added to compensate for the network deficiencies.

Given the complexity of performance testing, the best IBM resource for this work is our IBM ISSW consulting team. Our IBM consultants are fee-based and they can review your request in the context of your network environment and business solution. Contact your IBM marketing representative for further information.