This blog promotes knowledge sharing through experience and collaboration. For more product information, visit our WebSphere Commerce CSE page. For easier navigation, utilize the Categories to find posts that match your interest.
Optimizing Search Indexing Performance for Commerce v9
Search indexing is required to get your catalog changes to display on the storefront, so the longer it takes to rebuild the search index, the longer it takes to get your changes out to your storefront. With Commerce v9, we've not only added improvements to search indexing logic, but have also simplified existing improvements to make it easier to utilize such improvements. To see improvements to search indexing process, we need to first establish a baseline to measure our changes against.
Search Indexing Baseline Performance
To set a baseline, I'm using both a Commerce v126.96.36.199 and Commerce v188.8.131.52 environment, which can help for comparisons between version levels. For both environments, I'm running these tests on 8 core VMs with 4GB heaps for each server using a ~150k catalog entry catalog. Note that these numbers are very environment specific, due to the many factors that come into play (OS, CPU, Memory, catalog size/structure, etc), so your own testing may vary from these results. Below are my results when performing indexing on my Commerce v184.108.40.206 and v220.127.116.11 environments:
Just switching my testing from Commerce v18.104.22.168 to v22.214.171.124 improved my indexing performance by just over 50%. This is a good start, but taking advantage of additional indexing optimizations should help us perform indexing even faster.
Parallel Processing and Distributed Indexing (Sharding)
The process to utilize sharding has been significantly simplified in Commerce v9, such that you only need a few simple changes to utilize sharding. First, you will need to make sure that the di-parallel-process.properties and password.properties files are consistent for your environment. Mainly, the database configurations will need to be updated to point to the database used for this environment, as well as any username/passwords that have been modified from default values. Note that in di-parallel-process.properties, it will automatically setup 3 shards, split catentries evenly across the shards. You can tune this accordingly depending on the size of your catalog as well as how much resources you have available for indexing.
I'd also suggest that you follow the steps on setting up horizontal and vertical sharding to utilize the vertical shard, as otherwise you'll duplicate vertical data (ex. catalog hierarchy data) across all of your other shards. For example, the catalog hierarchy table TI_APGROUP can be built inside of a single vertical shard with entire data set, rather than having to be built across all the horizontal shards. The steps to perform for setup can be automated through Dockerfile, here is an example of my Dockerfile for the ts-utils container:
To summarize, here are our indexing metrics so far:
Compared to v126.96.36.199 indexing, v188.8.131.52 sharding is just over 330% faster, which is quite the performance improvement. There is yet another indexing optimization we can utilize to decrease amount of data sent over network during preprocess to improve preprocessing performance.
Normally, preprocess will first perform a SELECT query to retrieve all the data for the preprocess table, and then perform INSERTs of this result set into the preprocess table. CopyColumnsDataPreProcessor is a new data preprocessor that will skip this network heavy operation by instead performing a INSERT INTO SELECT... statement, so that this data doesn't even need to leave database and be processed in the transaction or utility container. Currently, as of v184.108.40.206, this data preprocessor is available for the tables defined in the following preprocess XMLs:
The more data you're processing within these tables, as well as the higher the latency is between the database and your transaction/utility containers, the better performance you will see when using this data preprocessor. I've setup the following Dockerfiles to utilize these XMLs:
Once you build your customized ts-utils and ts-app containers, and start them up, you can then either call the indexing API or execute the sharding script from the ts-utils container to build your search index. Below are the current indexing metrics with CopyColumnsDataPreProcessor in use:
If you still have additional system resources you would like you to use to improve indexing time, then you should consider having sharding run in multithreaded mode for preprocess.
Multithreaded PreProcess for Sharding(NOTE: Blog post was updated Dec 6th 2018 to add this indexing scenario)
For the PreProcess part of parallel processing and distributed indexing (sharding), by default, it is processed in a single thread within ts-utils container. However, if you update your sharding properties file (di-parallel-process.properties) to include Global.preprocessing-multithread=true, then each preprocess XML will run in it's own parallel thread (except for a few XMLs containing dependency tables). Below are the final indexing metrics with multithreaded preprocess for sharding in use:
In addition to many of the other improvements included in Commerce v9, we can see here that there is also a massive improvement in search indexing performance, while also being much easier to configure such indexing performance optimizations than in Commerce v7/v8. If you have tested indexing performance on Commerce v9, feel free to post a comment below with the improvements you've seen with your indexing times.