Configuration best practice

As of watsonx.data™ version 1.1.1, some settings are enabled by default to improve the performance. You can still customize these settings based on your needs.

JVM Xmx is calculated based on the selected t-shirt size. If the memory allocated to the Presto (Java) pod is more than 8GB, 80% of memory is assigned to JVM Xmx else 70% is assigned. The users still have an option to customize the value.

Following properties are set to maximum by default based on JVM Xmx allocated to Presto (Java) coordinator or worker:

  • Maximum total memory per node (query_max_total_memory_per_node): JVM Xmx * 0.795
  • Maximum memory per node (query_max_memory_per_node): JVM Xmx * 0.795
  • Memory heap headroom per node (memory_heap_headroom_per_node): JVM Xmx * 0.2
  • Maximum memory for query (query_max_memory): 1TB
  • Maximum total memory for query query_max_total_memory): 2TB

You can customize these properties.

If you have a customized JVM, the properties are calculated based on that customized JVM. You can customize the JVM to resolve the OOM issues and maximize the memory usage.

Following cache types are enabled by default:

  • Metastore versioned cache
  • Header and footer cache (metadata cache) for Parquet and ORC and DWRF

Metadata caching is calculated considering JVM Xmx.