Elastic Stack performance

Follow this guidance on how to tune Elastic Stack configuration to suit your business needs.

Tuning the heap sizes for Elasticsearch and Logstash to accommodate heavy load
Scale up the default Elasticsearch and Logstash heap size for each service to match your deployment. For example, if you increase memory for your system, you can also increase the heap size to accommodate system performance and heavy workload.
Configuring Elasticsearch disk usage
You can configure Elasticsearch to control disk-based allocation for Elasticsearch to consider available disk space on a node before you determine whether to allocate new shards to that node or to actively relocate shards away from that node.
Configuring Elasticsearch to use a different resource group
By default, EGO is configured with three resource groups, one of which is the ManagementHosts resource group. The Elastic Stack with IBM Spectrum Conductor by default uses this ManagementHosts resource group. You can create a new Elastic Stack resource group and then switch the Elasticsearch to use that group, instead of using the predefined ManagementHosts group.
Scaling Elasticsearch and Logstash services to accommodate heavy indexing, queries, or both
When your system runs a heavy load, this can affect performance. For example, if you have Spark applications that complete but metrics are slow to display or appear in visualizations (that is, there is a delay in displaying the performance charts under Workload > Instance Groups > Applications > Performance in the cluster management console). To resolve this heavy load issue and to improve performance for the metrics and visualizations, scale up the Elastic Stack services instances to accommodate these heavy loads.
Scaling Elasticsearch replicas to provide redundant copies of data to protect against hard failure
By default, Elasticsearch indices are configured with one primary shard and one replica at index creation. This configuration can tolerate a failure of at most one host while Elasticsearch migrates shards to rebalance the cluster. You can scale up the default Elasticsearch replicas to increase redundant copies of your data.
Tuning the Spark metrics sink
The Spark metrics sink poll metrics to allow you to monitoring applications submitted to an instance group. You can also use charts to view application performance based on running and completed tasks and application resource usage based on cores and memory: the Running Tasks and Completed Tasks charts, show data with a minimum interval every minute; if an application completes within one minute, the charts do not show data. As such, by default, metrics for these charts are retrieved from Spark gauges that are written every minute. You can change the frequency the metrics are written.

Parent topic: Monitoring and the Elastic Stack