Performance Considerations for IBM InfoSphere Master Data Management Advanced Edition

Content

Before attempting to tune performance, we must have a clear idea as to what the exact problem is. A workflow being slow is not an actionable statement and so we need to know the exact use cases which are slow and those that aren't. Running test cases can usually narrow down a problem much more effectively than reviewing the logs. Questions such as does it only happen at peak load or is it a general issue? Is it limited to certain use cases, users, or components of the product? Depending on these exact details, we can chose an action plan best suited to the particular case .There are special considerations for all customer deployments like the architecture they use, customization, custom code, etc. IBM support or engineering would not be aware of these factors so in most cases, it works best if the administrator narrows down the slow use case to it's most specific level before attempting to tune. Other-wise the administrator might be asked to run use cases which do not impact overall performance. A well defined problem statement can save a lot of time in the long run.


MDM can only run as fast as the hardware allows it to and so better (or more) hardware to make your system faster. Adding another node to a cluster or creating a cluster from a standalone can double your throughput and response time. Having more buffer pools or faster disk or flash memory can make the SQLs run faster. Having more threads, contexts in your datasource along with CPU, I/O, or network bottlenecks can impact concurrency and throughput.
We can attempt to tune performance indefinitely in an attempt to make it faster but that has a cost associated with it (time, more servers, better hardware, faster disk, more memory etc). So there should be a realistic agreement on what constitutes acceptable performance.


MDM is an OLTP system (Online Transaction Processing), consisting of many ACID compliant transactions of varying complexity, mostly small. OLTP transactions include selects, inserts, updates, and deletes, which normally complete in seconds or sub-seconds and a series of these account for a workflow and those make a feature. Following are some of the high impact considerations to improve performance:

1. If you are seeing low throughput, ensure that the WebSphere JDBC datasources (MDM, DWLConfig, and DWLCustomer) have enough contexts to the database to service the connections required for desired concurrency.
2. SMART inquiry feature allows us to turn off certain queries which will not be used by the system. This saves CPU and memory in both application and database layer along with using system cache better. The gains depend on the number and nature of the queries which are turned off.
3. When you customize the data model through extensions, you can either update the corresponding out of box table with an additional column (in-line extension) or create a new table (side-table extension). If you have a high volume of update and insert, you should use in-line extension to avoid additional SQLs, joins, trips to be database, and optimize cache.
4. MDM has the ability to log extensive auditing information for organizations with strict data access and accounting policies. This is achieved through Transaction audit information logging (TAIL) and with it, any MDM services that add or modify data will have an additional logging. So you should enable it only when required and only for transactions and if possible, sub transactions that require it. TAIL also increases activity on some database tables.
5. The amount of data (type of business objects) which is retrieved from the database in a service is determined by the inquiry level specified. Having unnecessarily large payloads can increase the run time by increasing the processing required and introduce a network, memory, or CPU bottleneck in the database or application server. So you should only retrieve the data that you need.
6. Composite services are those which further invoke a series of more granular services to complete a work item. And example is MaintainParty composite service which gets a party, uses business rules to do computation, and finally updates the party which was got. So to improve performance you can model the service to only get the attributes you need using inquiry levels, update only the data in the database which has changed, and eliminate any redundant work. The design of your service greatly impacts it's execution time.
7. You also have the ability to define the SQL used for an inquiry level if you decide to use a custom inquiry level. If the SQL is not tuned, it can greatly increase the runtime, waste resources, or perform round trips to the database because of the design. Examples of bad SQL design include specifying columns that you do not need, performing joins, having multiple SELECT statements with OR clauses when an IN clause would be more efficient.
8. We have a history feature which stores all the historical creates, updates, deletes along with time stamp and allows you to construct how a record looked at a moment in time. In cases where the data is growing at a fast rate, maintaining the history can cause an overhead and more significantly skew the system cache causing unnecessary disk I/O.
9. Suspect duplicate processing is a very intense and resource consuming process. Whenever possible, run SDP asynchronously or in a batch as part of an evergreening process during period of low activity. This will ensure that the process does not impact end users and does not interfere or compete for resources with other processes.
10. Having good quality data with fewer duplicate, dummy or anonymous values will improve run time during searching, matching, suspect duplicate processing, and would reduce the overall system load. Use external validator such as disallowed values or disallowed patterns to prevent unwanted data that cannot be tolerated from entering InfoSphere MDM AE. If unwanted data must be tolerated, then account for it by using features such as search exclusion and match adjustment rules in suspect duplicate processing. External validation rules can still be configured using a warning level to allow the transaction to continue. The warning messages can be caught by additional processes that can trigger investigation of the records.
11. If you are using probabilistic matching, then for the probabilistic match/link/search part, you may use the following link to tune it: http://www-01.ibm.com/support/docview.wss?uid=swg27042420. Amongst the factors which will help the most are eliminating big buckets, big entities, and ensuring that most of the bucket (bktd) and comparison string (compd) work is done in the buffer pools with minimal disk I/O.
12. Profile your data to understand data outliers and address them and understand the characteristics of master data management transactions (inquiry and maintenance services). This is especially true of new sources. You may aggregate individual source system profile results into a combined data profile to understand what the profile of the data will be like inside of InfoSphere MDM AE. Perform data model mapping in the context of the consumers of the master data and the results from data profiling.
13. Leverage coarse grained business services where possible and create composite services to prevent multiple calls from a SOA or BPM layer to InfoSphere MDM AE. Consider performance when evaluating the possibility of reusing a composite service to satisfy new requirements. If possible, only provide new or changed data to coarse grained update services like UpdateParty to save the cost of redundant update checking. Consider using the Service Activity Monitor to satisfy application audit requirements such as auditing what users are querying data.
14. If using the Batch Processor for initial or delta loads, then set the number of submitters to two to three times the number of physical CPUs available to InfoSphere MDM AE.

• Having sufficient system queue depth such that it doesn't fill up during peak load because other-wise that may increase your latency considerably.
• Increase the number of available threads to increase concurrency and throughput.
• Limit the size of your XML notifications to what is needed by the consuming application to work. Unnecessarily large messages can cause slowness, waste resources like memory, and may be rejected.

• Object Request Broker (ORB) thread pool. This limits the the number of concurrent RMI transactions and should be higher than the expected number of users. This is set at Servers > Server Types > Application servers > server_name > Container services > ORB service > Thread pool.
• Web container thread pool. Each thread here is equivalent to a concurrent MDM service request. This is set in Servers > Application Servers > server_name > Thread Pool.
• Enterprise JavaBeans (EJB) cache size. This is the number of items that WebSphere will keep in memory and is set at Servers > Server Types > WebSphere application servers > server > EJB Container Settings > EJB cache settings.
• Prepared Statement cache size for Java Database Connectivity (JDBC) data source to cache the result of the prepare phase of prepared statement to improve SQL run time. This is set at Resources > JDBC > Data sources > <Data Source Name> > WebSphere Application Server data source properties > Statement cache size
• Ensure you do not have other enterprise applications running on the same WebSphere instance competing for resources.

• Buffer pools act as the "in memory" working area for the database. It keeps the recently fetched objects from the database in memory for data manipulation (except large objects and long field data). Higher the buffer pools, less disk I/O the system needs to perform and reducing disk I/O improves performance significantly. So you should ensure a high buffer pool cache hit ratio, say 99%, or have very fast disk, say flash drives, if your system is disk I/O bound.
• DB2 allows asynchronous I/O access of a read and written between the buffer pool and the disk for optimal performance. I/O servers asynchronously read data pages from the disk into the buffer pool anticipating their need by an application through a called "prefetching." This removed the overhead and is a significant improvement over Page cleaners which write changed pages from the buffer pool to disk before the space in the buffer pool is required by a database agent. As a result, database agents should not have to wait for changed pages to be written out so that they might use the space in the buffer pool.
• DB2 has two basic types of sorts: shared sorts and private sorts. The memory used for private sorts is allocated from an agent's private memory. So a private sort can only be accessed by a single agent. Shared sorts are used when it is desirable to have multiple sub agents feeding or fetching from a sort. The memory used for shared sorts is allocated from the database shared memory set. In both shared and private sorts, the SORTHEAP database configuration parameter is the maximum number of 4 K memory pages that will be used for a single sort. Private and shared sorts use memory from two different memory sources. The size of the shared sort memory area is statically predetermined at the time of the first connection to a database based on the value of sheapthres. The size must be at least two times the size of sortheap of any database hosted by the DB2 instance.
• The log buffer acts as a staging area in memory to hold log records instead of having the DB2 UDB engine write each log record directly to disk. Writing would be expensive and so higher your log buffer size, less writes, and faster the system.
• By default each COMMIT statement triggers one log buffer flush to disk. As a result, the logger process frequently writes small amounts of log data to disk, with additional delay caused by log I/O overhead. Commit grouping is a feature that allows the writing of the log buffer data to disk until a minimum number of commits have been requested. Increasing the MINCOMMIT parameter from its default value will result in more efficient logging file I/O as it will occur less frequently and write more log records each time it does occur. This feature can result in improved performance by reducing the log I/O overhead.