Universal Agent using high CPU

Body

It seems I need to change my mind and admit Universal Agent is far from being fully replaced by Agent Builder.
For the second time in few days I found useful material that worth to be shared about this solid and reliable tool.
After having learned how to troubleshoot scenarios where subnodes are not showed in TEP
(see blog article /support/pages/node/1085121)

we will now focus on an issue that I faced many years ago and that re-surfaced recently in a different ITM environment.

Universal Agent instance was running on this machine since long time and never highlighted problems.
More or less one month ago the UA process, called KUMA620, started using high CPU, up to 100%, impacting the other applications running on the same machine.
When the agent is restarted, it works fine for some time and then problem occurs again.

The fact that the problem occurs on other UA nodes that are on the same subnet, led me to think about possible relationship with network activity, even because the
UA RAS1 log also showed error messages like these:

(Wed Feb 17 22:20:44 2018.0008-2:kumdsock.cpp,1812,"ipcSock::IPCServer") Ignoring buffer with invalid length 1330664521 received on DCH port 1919
(Wed Feb 17 22:20:49 2018.0000-14:kumpstcp.c,211,"KUMP_ProcessTCPconnection") ***** TCP dynamic inbound connection rejected from <10.11.12.13>, no explicit metafile specification
(Wed Feb 17 22:20:53 2018.0000-28:kdhsiqm.c,848,"KDHS_InboundQueueManager") Unsupported request method " "
(Wed Feb 17 22:20:53 2018.0001-28:kdhsiqm.c,850,"KDHS_InboundQueueManager") error in HTTP request from ip.tcp:#10.11.12.13:57326, status=7C4C803A, "unknown method in request"
(Wed Feb 17 22:20:53 2018.0002-28:kdhsiqm.c,848,"KDHS_InboundQueueManager") Unsupported request method " "
(Wed Feb 17 22:20:53 2018.0003-28:kdhsiqm.c,850,"KDHS_InboundQueueManager") error in HTTP request from ip.tcp:#10.11.12.13:57342, status=7C4C803A, "unknown method in request"
(Wed Feb 17 22:20:54 2018.0000-28:kdhsiqm.c,848,"KDHS_InboundQueueManager") Unsupported request method " "
(Wed Feb 17 22:20:54 2018.0001-28:kdhsiqm.c,850,"KDHS_InboundQueueManager") error in HTTP request from ip.tcp:#10.11.12.13:57354, status=7C4C803A, "unknown method in request"
(Wed Feb 17 22:20:54 2018.0002-28:kdhsiqm.c,848,"KDHS_InboundQueueManager") Unsupported request method " "
(Wed Feb 17 22:20:54 2018.0003-28:kdhsiqm.c,850,"KDHS_InboundQueueManager") error in HTTP request from ip.tcp:#10.11.12.13:57368, status=7C4C803A, "unknown method in request"
(Wed Feb 17 22:20:54 2018.0004-14:kumpstcp.c,211,"KUMP_ProcessTCPconnection") ***** TCP dynamic inbound connection rejected from <10.11.12.13>, no explicit metafile specification
(Wed Feb 17 22:21:13 2018.0000-15:kumpsosr.c,1281,"KUMP_CommonSocketServer") API connection lost while receiving additional data for TCP connection from 10.11.12.13
(Wed Feb 17 22:21:13 2018.0001-15:kumpspst.c,786,"KUMP_ProcessSockRequestByState") *** API request rejected from 10.11.12.13, incorrect version < >

I suspected the thread starts looping in this very moment and so this is when kuma620 starts consuming high CPU.
Port 1919 is the port used by the DCH component of UA to talk with the Data Providers.
Above messages are issued because the DCH received unexpected packets and it is not able to recognize them and associate them to running applications (metafiles).
Additionally, some time the length of the received buffer exceeds the expected limits, so it seems UA is receiving packets that are not actually expected from it.
Per my previous experiences, this kind of scenario occurs due to a Port Scanner, even in in such cases I often observed a crash/core of one of the kuma620 threads,
while in this case there are no crashes. The process just starts looping and using high CPU.
What to do if you face similar scenarios and you also find the same messages in your Univeral Agent log files ?
First of all, check with your network administration to know whether he is running a port scanner on the machine having IP addess showed in message "error in HTTP request from ip.tcp:#".
In my case it was 10.11.12.13.

If this is confirmed to be a Port Scanner, we need to "protect" Universal Agent against it.

We can do it in two possible ways:

1) Setting the variable

KUMA_DCH_HOST=127.0.0.1

into um.ini file and restarting the agent.

This will cause UA's DCH port to listen on the local loopback address instead of using the external IP address, which will help against a
port scanner or denial-of-service attack targeting port 1919.
The idea is that even when not running a standalone DP but instead running UA as a single unified process,
the KUMA_DCH_HOST=127.0.0.1 setting would allow UA's internal DP-to-DCH socket connection to utilize the local loopback address.
That should protect the socket connection from a port scanner, which normally would not be scanning the local loopback interface on a system.

2) Cooperate with network admin to exclude port 1919  (or the whole UA machine) from Port Scanning.

Of course the first option is the best one, it has been tested successfully in many similar cases where UA agent started using high CPU or crashed apparently for no reason.
It helped me also in this case, so I wanted to share this experience with the community, hoping it can be useful for you as well.

Best Regards

Subscribe and follow us for all the latest information directly on your social feeds: