Java utilities such as jps or jstat unable to monitor DSE processes

Problem

Summary

This article discusses possible causes for why Java utilities are unable to monitor DataStax Enterprise and Apache Cassandra processes.

Symptoms

When attempting to access Java VMs on a node, tools such as jps do not see running DSE processes. For example:

$ jps -l
21228 sun.tools.jps.Jps

In contrast, another system detects the DSE processes as per the following example:

$ jps -l
2307 sun.tools.jps.Jps
9154 com.datastax.bdp.DseModule
9252 org.apache.spark.deploy.worker.DseSparkWorker

Cause

The 2 most common reasons for Java tools not able to detect JVM processes are:

• permission issues
• performance data collection issues

PERMISSIONS

Java processes collect and save data in /tmp/hsperfdata*. If the user running the command such as jps or jstat does not have access to the directory (or /tmp filesystem), then it will fail to detect the JVMs.

COLLECTION ISSUES

If the DSE process does not have permission to write to the /tmp filesystem, performance data collection will fail.

Similarly, data collection could have been disabled which is why the JVM is not detected.

Solution

Follow these steps in order to enable monitoring for Java tools.

Step 1 - Confirm that performance stats collection is not disabled, i.e. confirm that the following JVM option is not included in the startup:

-XX:+PerfDisableSharedMem

If so, you will need to remove this from the startup script (e.g. cassandra-env.sh) and restart DSE.

JVM performance stats collection was disabled from Apache Cassandra 2.2 (CASSANDRA-9242). See FAQ - Why Java performance utilities like jps no longer work in DSE 5.0+ for details.

Step 2 - Confirm that the user running DSE has permissions to the /tmp filesystem. If not, the stats collection will not be able to save performance statistics.

For example, DSE is being run with the user automaton:

$ sudo lsof -i -n | grep LISTEN | grep java
java       9154 automaton  418u  IPv4  31503      0t0  TCP 127.0.0.1:7199 (LISTEN)
java       9154 automaton  419u  IPv4  31504      0t0  TCP 127.0.0.1:52784 (LISTEN)
java       9154 automaton  427u  IPv4  31508      0t0  TCP 10.240.0.65:afs3-fileserver (LISTEN)
java       9154 automaton  439u  IPv4  31513      0t0  TCP 10.240.0.65:9042 (LISTEN)
java       9154 automaton  440u  IPv4  31514      0t0  TCP 10.240.0.65:9160 (LISTEN)
java       9154 automaton  469u  IPv4  41589      0t0  TCP 10.240.0.65:7077 (LISTEN)
java       9154 automaton  472u  IPv4  41590      0t0  TCP 10.240.0.65:6066 (LISTEN)
java       9154 automaton  477u  IPv4  41596      0t0  TCP *:7080 (LISTEN)
java       9252 automaton  324u  IPv4  29510      0t0  TCP *:7081 (LISTEN)
java       9252 automaton  344u  IPv4  32049      0t0  TCP 10.240.0.65:56397 (LISTEN)

and user automaton has permissions to write to the /tmp filesystem:

$ ls -lhd /tmp
drwxrwxrwt 7 root root 4.0K Apr  2 05:45 /tmp

Step 3 - Confirm that the user running the Java tools such as jps has permission to read /tmp/hsperfdata*.

For example:

$ ls -lhd /tmp/hsperfdata_*
drwxr-xr-x 2 automaton automaton 4.0K Apr  2 05:41 /tmp/hsperfdata_automaton
drwxr-xr-x 2 root      root      4.0K Apr  2 05:41 /tmp/hsperfdata_root