This blog covers the solution to response time delays that may be seen with web service calls in IBM Business Process Manager Advanced V8.x. The problem and its solution that are discussed in this blog apply to IBM Business Process Manager Advanced topology that is configured with DB2.
For the sake of this scenario, we will assume that you are using a WebSphere Application Server Network Deployment V8.0.x.x and an IBM Business Process Manager Advanced V8.0.1 Fix Pack 1 two-node, four-cluster topology using DB2 ESE V9.7.0 Fix Pack 6.
As covered in Martin and Geza's article Parallel invocation of synchronous services in WebSphere Enterprise Service Bus, caution must be exercised when making the decision to use asynchronous with deferred response invocation style and the service invoke mediation primitive. The problem discussed in this blog can arise when this invocation style is used with a QoS of Assured Persistent (default reliability) on the SCA destination. The following response time graph shows the response time spikes that are exhibited by this set-up in a 10-minute performance test at about 45 transactions per second (tps).
With Assured Persistent QoS, comes a significant overhead of database transactions to the messaging database on each outbound web service request. This leads to increased database logging activity. Now, if the messaging database is set up for log archiving, with each log archiving event, the database log manager searches the history file (db2rhist.asc) for when the log was generated. It searches the file in order to update the entry with when the log is being archived as well as where it is being archived. The larger the size of this history file, the longer it takes for the log manager to search for an appropriate log entry in this file. As the log manager searches for the right entry, it holds a lock on the history file. While the log manager holds this lock, it precludes the log writer from updating the history file with an entry for a new log file. Commit and rollback activity, and possibly other database operations, are blocked until the log writer can add the history file entry and continue normal processing. The blocked log writer leads to response time spikes, which are seen in the previous response time graph.
To mitigate this issue, it is important that the size of the history file is managed appropriately. Typically, it is recommended that the history file size must not exceed 4MB. Following factors should be considered in order to limit the size of the history file:
- Maintain an appropriate sizing of the database log files to ensure that the log files are not being archived too frequently; thereby managing the frequency of updates to history file. It also helps manage the rate at which new log files get created.
- Prune the history file by setting appropriate values for NUM_DB_BACKUPS (Number of database backups to retain) and REC_HIS_RETENTN (Recovery history retention in days). The values for these parameters are a factor of throughput that is expected from the IBM Business Process Manager applications that are running on the platform. For these parameters to take effect, set AUTO_DEL_REC_OBJ=ON (Auto deletion of recovery objects).
We welcome your feedback on this topic! Share your thoughts below this blog entry!