IBM Support

Yarn queue memory configurations.

Question & Answer


Question

Can I limit the amount of memory that a yarn queue can use on a specific host?

Answer

Assume a scenario where you have 3 queues and you want one of the queues, (in this case Queue3) to not exceed a specific amount (in this example 6GB) of memory usage per data node.
 
1. Queue1
2. Queue2
3. Queue3
 
The limiting of the resources is done at the whole cluster level, and not at per data node. So the 6GB limit that you may want to enforce is not something that can be enforced per data node, but can be enforced on the whole cluster. This means Queue3 will use just 6GB in total from all nodes combined.  In order to set this limit for the cluster, you have to take the below settings into account.
 
So assume you have say 10GB per node in a 4 node cluster:
 1. Set yarn.nodemanager.resource.memory-mb to 10GB
 2. Set Queue3 queue capacity to be 15%.
yarn.scheduler.capacity.<queue-path>.capacity=15

    So capacity would be calculated as such: 10GB*4*15% = 6GB

 3. The rest of the 85% can be split between Queue1 and Queue2 queue as you want by setting the value for yarn.scheduler.capacity.<queue-path>.capacity for each queue.

 
That way, above capacity setting allows in total to use 6GB of memory in the cluster for Queue3 to use.

But note, it means any of the data nodes can provide that 6GB as the containers could be allocated on any of the nodes so the memory limit is not enforced per data node.
 
The important point here is that the queue capacity is used to limit queue resource usage based on the whole cluster resources and not per data node.

[{"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Open Source Tools","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.1.0;4.2.0;4.2.5","Edition":"Enterprise Edition","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
18 July 2020

UID

swg22005294