IBM Support

IBM Runtime Diagnostic Code Injection for the Java Platform (Java Surgery)

How To


Summary

The IBM Runtime Diagnostic Code Injection for the Java™ Platform (RDCI) tool injects and runs Java code (such as requesting a thread dump on Windows) into a running Java Virtual Machine (JVM) which has late attach enabled.

Steps

  1. Download surgery.jar: https://public.dhe.ibm.com/software/websphere/appserv/support/tools/surgery/surgery.jar
  2. If the target Java process is on a remote host:
    1. Copy surgery.jar to the remote host.
    2. Log into the remote host as root/Administrator or as the same user that is running the target Java process (this is required due to security restrictions of late attach).
  3. Open a terminal and change directory to where surgery.jar is located.
  4. Use the same Java executable that is running the target Java process to also execute surgery.jar, replace ${PID} with the process ID of the target Java process, and specify the command to execute and any optional parameters. For example, to request a thread dump on J9 JVMs:
    1. ${DIRECTORY_OF_TARGET_JAVA}/java -jar surgery.jar -pid ${PID} -command JavaDump
    2. If using certain versions of HotSpot Java, you may need to add the JDK's tools.jar to the classpath. For example (on Windows, use the semicolon delimiter):
      1. ${DIRECTORY_OF_TARGET_JDK}/java -classpath surgery.jar:${DIRECTORY_OF_TARGET_JDK}/lib/tools.jar -jar surgery.jar -pid ${PID} -command JavaDump
Late attach notes:
  1. Late attach is enabled by default on J9 JVMs, except on z/OS. It may be enabled or disabled by restarting the JVM with -Dcom.ibm.tools.attach.enable=[yes|no]
Available commands:
  1. CollectGarbage - Call java/lang/System.gc()
  2. ExecuteMethod - Uses reflection to execute a method. ExecuteMethod Arguments: -class CLASS -method METHOD
  3. GetPID - Sends the PID of the current process to available transports.
  4. HeapDump - IBM JVMs only. Call the com.ibm.jvm.Dump.HeapDump API to request a PHD heapdump.
  5. InstallOSGiBundle - Install an OSGi bundle fragment
  6. JavaDump - IBM JVMs only. Call the com.ibm.jvm.Dump.JavaDump API to request a javacore.
  7. KillJVM - Call java/lang/System.exit(1)
  8. PrintHelloWorld - Simply prints the String 'Hello World' to System.out.
  9. PrintInfo - Print WAS Information
  10. PrintOSGiInfo - Print OSGi Information
  11. PrintProcessInfo - Prints Date and TimeZone information, ulimits, envars, and system properties.
  12. SystemDump - IBM JVMs only. Call the com.ibm.jvm.Dump.SystemDump API to request a system core dump.
  13. ThrowException - Throws an instance of com.ibm.rdci.surgery.builtin.commands.CustomException1
  14. UninstallOSGiBundle - Install an OSGi bundle fragment

Additional Information

The IBM Runtime Diagnostic Code Injection for the Java™ Platform (RDCI) tool introduces the concept of performing surgery on a running Java Virtual Machine (JVM) that supports the Late Attach API. It has been tested on recent versions of both IBM/OpenJ9 and HotSpot JVMs. It is a command line JAR file that you run, passing the process ID and the surgical command to execute. The tool injects and runs this code in the JVM (which is done within a custom classloader to minimize the "scar" left on the process). In principle, the code can do anything that Java can do, from printing diagnostic information ("exploratory surgery") to actually modifying accessible fields and properties ("invasive surgery"). The tool is extensible and comes with a few useful commands.
For example, the original use case came from a colleague who wanted a quick way to get a thread dump on the IBM JVM on Windows (there is no equivalent to "kill -3" when there is no console attached and using the JVM MBean is non-trivial). The thought came up to just inject code that calls the com.ibm.jvm.Dump.JavaDump() method. Here is RDCI in action:
java -jar surgery.jar -pid ${PID} -command JavaDump
HotSpot JVMs require explicitly putting tools.jar on the classpath to use late attach (on Windows, use the semicolon classpath delimiter):
java -classpath surgery.jar:$JDK_HOME/lib/tools.jar Surgery -pid 16715 -command CollectGarbage
The basic RDCI agent that runs the command only uses about 20 KB of Java heap itself in the target JVM.
Extending Java Surgery
Example creating a custom surgery command using Eclipse (any IDE or just javac can be used):
First, create a new Java Project. On the next screen, you'll see various Java settings, starting with the "Source" tab. Select the "Libraries" tab and click "Add External JARs" and find surgery.jar. I also recommend you expand this added entry, double click on Javadoc location, select "Javadoc in archive," select the same surgery.jar file and set "Path within archive" to doc. Click Finish.
Create a new plain old class (with or without a package) under the "src" folder. Then implement com.ibm.rdci.surgery.ISurgeryCommand and its two methods. For example:
import com.ibm.rdci.surgery.*;
public class TestSurgeryCommand implements ISurgeryCommand {
    @Override
    public void run(ISurgeryConnection connection, String[] args) throws SurgeryException {
        System.out.println("Hello World");
    }
    @Override
    public String getDescription() {
        return "My test command";
    }
}
Next, right click on the project and select Export > Java > JAR file. Specify a .jar file name and click Finish.
Finally, add the JAR file to your class path and use the new command (on Windows, use the semicolon classpath delimiter):
java -classpath surgery.jar:mycommand.jar Surgery -pid ${PID} -command TestSurgeryCommand
The javadoc is available inside surgery.jar. Simply expand the JAR file using a ZIP expander or "jar xvf surgery.jar" and then go into the doc folder and open index.html in a browser.
When your command gets executed inside the JVM, you are limited in what you can do primarily by class visibility. Your class is loaded by com.ibm.rdci.surgery.SurgeryCustomJARClassLoader which delegates to a general Application ClassLoader (sun.misc.Launcher$AppClassLoader). This means that if your application has a complex ClassLoader hierarchy (such as OSGi), you can't just do a Class.forName to find the class.
Luckily, the JVM provides an instance of java/lang/instrument/Instrumentation to the agent, which we pass to your command. The first argument to your run method is an instance of com.ibm.rdci.surgery.ISurgeryConnection which has a method called getInstrumentation. The primary method of interest in the Instrumentation class is getAllLoadedClasses. This will return a list of all loaded classes in the JVM regardless of ClassLoader. Once you've found the one you need, you can use reflection to execute its methods. The com.ibm.rdci.surgery.util.Util class has various helper methods to simplify much of this.
The next limitation is that having a Class is usually not enough. Often what you want is a list of all instances of a class. Unfortunately, I'm not aware of any way to get this information, so you're currently limited to static method calls.
Due to this limitation, if you want to burrow deeply into the patient, you'll need to follow any exposed hooks for extension. For example, in OSGi, you can install a bundle and from there you can ask for various services. The Util class also has helper methods to do this. You can also consider attaching a bundle fragment onto an existing bundle to get access to non-externalized classes. Unfortunately, none of this is trivial, but who ever said surgery was easy :) The analogy to surgery is that doctors are limited in what they can cut through on their way to the final destination. There are certain components which the doctors have to snake around. In this same sense, we can't just access or modify anything willy-nilly. Instead, we need to work with the structures and pipes that the JVM and application provide to get to what we need. Since we are in the early days of surgery, this may be nearly impossible in some cases, but that's why we ne need to add improvements to our applications (and JVM) to make this easier. We can also package some common techniques (such as complex OSGi maneuvering) into surgery.jar for all surgeons to use.
Frequently Asked Questions (FAQ)
Question #1: Doesn't this tool expose a security hole?
Answer #1: First, this tool does not open any new holes. The late attach API is the mechanism that allows code injection, and this tool simply makes that a bit easier to use. Second, late attach uses operating system security to limit who can inject code and also requires that a user is logged into the target system (i.e. it is not done through a socket). Even if late attach is disabled, it is still quite easy to inject arbitrary code into a running JVM if a user has shell access to the running process, so user access is where you should focus your security efforts. If your organization requires security-in-depth, then you can disable late attach, but you must go much farther to actually defend against this class of attacks.
The Java Surgery tool is provided as is without any warranty or support. The tool is maintained as time permits by Kevin Grigorenko (kevin.grigorenko@us.ibm.com).
Known Issues
  1. com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException100

    This has been known to be caused by running surgery.jar as root. Instead, try running surgery.jar as the same user as the one that is running the target process.

    Example stack:
    Could not attach to 107187
    Surgery: Error: java.lang.reflect.InvocationTargetException
    com.ibm.rdci.surgery.client.AttachException: java.lang.reflect.InvocationTargetException
            at com.ibm.rdci.surgery.client.BaseAttacher.loadAgent(BaseAttacher.java:64)
            at com.ibm.rdci.surgery.client.Client.attach(Client.java:130)
            at com.ibm.rdci.surgery.client.Client.main(Client.java:98)
    Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
            at java.lang.reflect.Method.invoke(Method.java:508)
            at com.ibm.rdci.surgery.client.BaseAttacher.loadAgent(BaseAttacher.java:62)
            ... 2 more
    Caused by: com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException100
            at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.parseResponse(OpenJ9VirtualMachine.java:372)
            at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.loadAgent(OpenJ9VirtualMachine.java:265)
            ... 7 more
    Surgery: Finished

Document Location

Worldwide

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}},{"Business Unit":{"code":null,"label":null},"Product":{"code":"SG9NGS","label":"IBM Java"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
22 June 2020

UID

ibm11109937

Page Feedback