Common reasons for BPM REST call performance problems
AndreMueller 2700061CVB Comment (1) Visits (17458)
When you experience long elapse times for Business Process Manager (BPM) REST calls, you could check the BPM server SystemOut log to find out which calls they are and how long they took. This dW Answers link shows what these messages in SystemOut look like.
How does the authentication process relate to the elapse time of the REST call?
When the group membership sync is done, this row is deleted.
How can I check in the BPM server's SystemOut log, for which user a group synchronization was done and how long it took?
[9/29/15 11:10:00:000 CEST] 00000191 wle_security I CWLLG0468I: Checking user Andre for new updates...
Common reasons why a REST call might take a long time to complete
1) Very large number of groups defined in LDAP
If there is a very large number of groups defined in your LDAP repository, your ldap server might have to scan through all of its groups to find out if the user is a member of that group. LDAP servers like IBM directory Server or MS Active Directory provide an attribute by which the group membership of a user can easily be retrieved, so that the overall scan can be avoided. With APAR JR47596, BPM allows the use of this kind of 'memberOf' attribute to increase the performance of the group membership replication during login. This is also described in the BPM product documentation in section Optimizing group membership retrieval during user login.
2) Large number of 'dynamic groups' defined in BPM
The meaning of 'dynamic groups' is explained in the BPM product documentation in section IBM Business Process Manager security groups. The BPM group definitions are saved in BPMDB table LSW_USR_GRP_XREF and 'dynamic groups' show a GROUP_TYPE of '4'. You could check the number of dynamic groups in your BPM system by running the following SQL query against the BPMDB:
select count(*) as dynamic_groups from LSW_USR_GRP_XREF where GROUP_TYPE = 4 with UR;
The 'with UR' clause is DB2 specific to run the SQL with isolation level 'uncommitted read'.
When updating the group membership during user authentication, BPM must also check all of the dynamic groups, which could take a long time depending of the number of dynamic groups. This can be deactivated by configuration as described in the documentation section Deactivating dynamic group updates. Please keep in mind that the dynamic group update should not be disabled completely, but it might not necessarily be done on every login. So only disable it for the login by setting this configuration parameter:
With APAR JR48172, administrative scripts for the group membership update were introduced, so you could run those on demand or at fixed times during the day. See section Synchronizing group membership by groups in the documentation for a description of these scripts.
3) Using a custom REST client that does not support http sessions
The trigger for BPM to (re)authenticate a user for an incoming REST call is the existence of a session in form of a JSESSIONID cookie containing a valid session id for that user. Under the following conditions, BPM must (re)authenticate the user which includes the group membership synchronization:
So if you use a custom REST client that does not support http sessions, each call will trigger a group membership replication! You would see the above mentioned messages CWLLG0468I and CWLLG1088I in SystemOut log for every request. Check this dW Answers thread for some more details.
4) Using the same technical user for REST calls
If your REST client is multi-threaded and sends out many REST calls in a short period of time, all with the same user, then each of them might (depending on http session) lead to a group membership update for the same user, which also includes inserting a record into the BPMDB table LSW_LOCK. As this record would be the same for all requests, this will lead to contention on the LSW_LOCK table!
To verify if that might be your problem, check the SystemOut log for the occurrence of many of the following message within a short period of time:
CWLLG0468I: Checking user <username> for new updates...
To alleviate that problem, use multiple different technical users for the REST calls.
5) Huge amount of data in the BPMDB
Depending on the kind of REST call, there will be a lot of DB interaction impacting the elapse time. To optimize the DB response time, it's important to purge old data from the BPMDB and keep the DB statistics current. This is shown in this article "Purging data in IBM Business Process Manager".
6) WAS VMM or LDAP server performance
If your WAS federated repository comprises a ldap repository with directory type 'custom' (for example when using an 'open ldap' server), then the following performance relevant configuration settings may be missing:
You can check this in the WAS Admin Console by navigating to 'Global Security' > 'Federated repositories' > 'Manage repositories' > name
When caching is enabled, it will take some time after a server restart until the cache reaches a good efficiency. So directly after a server restart, REST calls will take a bit longer.
It should be obvious, that the overall performance of the ldap server and the network also plays a very important role!