This blog promotes knowledge sharing through experience and collaboration. For more product information, visit our WebSphere Commerce CSE page. For easier navigation, utilize the Categories to find posts that match your interest.
Passive Collection of Performance Data in Commerce v9
Outages can come at unexpected times, causing loss of revenue while also causing many working hours spent trying to perform root cause analysis on the outage. To mitigate the amount of time needed to troubleshoot such a scenario, the Commerce Support team has developed tooling for analyzing performance data, covered in our Resolve an Outage in 10 Minutes or Less blog post. In Commerce v9, we've simplified the process for passively collecting such performance data, so that it is even easier to setup collection of this valuable data for root cause analysis. Don't wait until after an outage occurs to start setting up passive collection of performance data for your environment!
Before we get started, I would suggest that you setup Docker volume for your /profile/logs directory in the Store, Search, and Transaction containers so that you synchronize these files outside of your containers. This will prevent data loss if the container was ever to crash or be shut down. For example, I format my docker volume configuration in my yml file to organize these directories under /opt/logs/ based on container name:
Now that you have your Docker volumes setup to organize your logs and performance data outside of your containers, let's start with setting up Java Health Center.
Java Health Center
Java Health Center is a great tool for collecting regular thread dumps as well as other performance metrics such as CPU/heap utilization during this period of data collection. Java Health Center has low overhead, so we have provided functionality OOTB to setup Java Health Center automatically. To enable Java Health Center for your environment, simply set ENABLE_HEALTH_CENTER=true in your container's environment variables, and after the container is started up, you will start generating HCD files containing the Java Health Center data inside of /profile/logs/healthcenter directory in your container. After waiting some time, you will see HCD files generated in this directory, which will also be synchronized to your Docker volumes specified directory:
root@ericscott:/opt/logs/search-app/healthcenter# ls -l
For reference, here are the default configurations used for Java Health Center:
If you want to modify these configurations, you can instead apply your own Java Health Center configurations to jvm.options through Dockerfile commands. With the HCD files, you can upload them to the WebSphere Commerce Health Center Report tool for a generated report that parses out this information into graphs and tables for easy review. Java Health Center is a good starting point for automated data collection, but we should now look into enabling verboseGC for garbage collection analysis.
verboseGC is another great tool for RCA by logging additional information about garbage collection. To enable verboseGC, you can configure your Dockerfile to include instructions to append verboseGC configurations to jvm.options:
After building your updated image through docker build and starting up a container using this updated image, you will start generating verboseGC files in /profile/logs/ with date and time included in the filename to quickly find the appropriate verboseGC logs. These files will also be synchronized to your Docker volumes specified directory:
root@ericscott:/opt/logs/search-app/# ls -l
Now that you have both Java Health Center and verboseGC setup, you will be better prepared to quickly perform RCA should an outage occur. Let us know in the comments below if you have any additional settings in place for passive collection of performance data in your Commerce v9 environment!