July 31, 2014 | Written by: Michael Dawson
Categorized: Community | How-tos
Share this post:
For Java developers, IBM’s Bluemix provides the environment needed to be able to develop and deploy applications without having to worry about infrastructure. While deployment and management of applications is well understood, there have been some questions about debugging. You may have looked into debugging in a Bluemix or Cloud Foundry environment and come across the discussion here.
The current architecture of Bluemix/Cloud Foundry doesn’t permit an incoming debug connection and at the same time, most debug clients are behind some kind of firewall, so we can’t configure our deployed application to make the connection from Bluemix/Cloud Foundry to the client.
What’s needed is either something that allows the debug connection to be established through the single port provided to the application by Bluemix/Cloud Foundry or alternatively something that can help the two ends “meet in the middle”.
In the discussion referenced above you’ll see that Patrick Mueller has a work around for node cf-node-debug which shares the single port. More, recently we’ve tried out a work around using the “meet in the middle” approach for Java, which has sometimes been referred to as a “debug server”.
We started with a simple forwarder that can accept two connections, one from the deployed Java application and the other from the debugger. It has a number of drawbacks but is a simple way to prove out the concept. The concept is simply:
You may ask if this approach will work with existing Java Virtual Machines and debuggers. It turns out that the answer is yes. Typically you configure the JVM to expect a connection from the debugger or the debugger to expect a connection from the JVM. However, it is possible to configure both the debugger AND the JVM to connect to the debug server intermediary instead of each other directly. The underlying JDI protocol is such that this just works.
The configuration on the JVM side would be as follows:
-agentlib:jdwp=transport=dt_socket,suspend=y,address=<address of debug server>:<port debug server is listing on>
The configuration can be added to a Bluemix application by setting it in JAVA_TOOL_OPTIONS as a USER-DEFINED environment variable either through the manifest file used when the application is pushed or via the Bluemix GUI:
On the debugger side we simply point at the debug server as well:
Configuring to allow for longer time outs is a good idea since we are debugging remotely with an intermediary. For example, in Eclipse using Window->Preferences->Java->Debug:
The debug server itself can be quite simple. The following is an example that we used to debug with successfully: example
Of course, tuning the TCP options like NoDelay could further optimize this approach. Given that we have a debug server, the question is where to host the intermediary. Unfortunately the simple debug server (cfForwarder) cannot be deployed to Bluemix because it depends on a non-HTTP TCP connection which is currently not possible given the current architecture of Bluemix/Cloud Foundry. This approach also has the drawback of being insecure. There is no authentication mechanism and the traffic between the external debugger and the deployed application is in plain text. In our case, we deployed it in our own virtual private server (VPS), which was publicly accessible.
In order to address these limitations we explored an extension to the basic approach where the debug server is deployed in Bluemix itself and any data exchanged is encrypted.
The main drawback to the second approach is that you have to start an additional “local proxy” on the machine where the debugger is running. That said, the benefits are worth this extra work.
The proxy fits into the picture as follows:
Instead of the debugger connecting directly to the debug server it connects to the local proxy. This proxy is responsible for making the connection (using SSL) to the debug server running in Bluemix and upgrading the session to an SSL protected websockets TCP connection that can transfer JDWP (Java Debug Wire Protocol) packets. The result is that traffic from the local proxy to Bluemix is protected by SSL.
Note that we don’t need a forwarder in the container for the deployed application because that application can access the debug server directly since it is internal to Bluemix. In our script, we obtain the internal address of the debug server by querying the applications status using the cf app command and enabling trace with CF_TRACE=true.
We’ve provided a remote debug script to simplify the debug process. The script will deploy the debug server to Bluemix, start the local proxy for the debugger to attach to, do the authentication with your provided token over SSL and show you the value you’ll need to use for JAVA_TOOL_OPTIONS when deploying the application to be debugged. Note that the script only works on Linux.
Ok, now let’s walk through the steps of debugging an application:
1) Make sure you’ve installed JRE7 and the Cloud Foundry CLI V6.
2) Download the remote debug tool from the site remotedebugtool.tar.gz into your workspace, and extract it using: tar –zxf remotedebugtool.tar.gz. Note that the code provided is not production level code, has not been optimized, hardened or brought up any specific coding standards. It is provided solely to demonstrate the work around.
Change into the remotedebugtool directory and make sure there is a script file named cf-debug.sh and one jar file called simpleproxy.jar.
3) Log into your Bluemix account using cf login and then execute the cf-debug.sh script.
./cf-debug.sh 7777 debugforhelloworld mycredentialtoken
7777 is the port on which the local debug proxy will listen. Note: you will make your Eclipse debugger connect to this port later on.
debugforhelloworld is the name for debug server. You can name the debug server whatever you like. It is recommended that the debug server’s name have a relationship with the target application, since there is a one-to-one relationship between the debug server and the application to be debugged. For example, debugforhelloword is a reasonable name for the server used to debug the helloworld application in Bluemix.
mycredentialtoken is the authentication token, you can set whatever you like, eg. 12345, but don’t specify an empty string. The script will set the DEBUG_AUTH_TOKEN= mycredentialtoken as system environment variable to the debug server and pass the token to the local proxy. When the local proxy does the handshake with the debug server, it will send the token value as HTTP header via SSL to the debug server. If the passed token matches the value the debug server retrieves from DEBUG_AUTH_TOKEN, a successful message is returned and a trusted channel is established.
The script will output the instructions about how to attach your Eclipse IDE to the local proxy as well as the JAVA_TOOL_OPTIONS you will need to set as an environment variable for the Bluemix application being debugged:
4.1 Attach the Eclipse debugger to the local proxy.
4.2 Specifying the JAVA_TOOL_OPTIONS to the target application in Bluemix.
4.3 Restart the target application in Bluemix.
If you’ve set a break point in the application and then hit the url of your application, you’ll see a similar picture to this Eclipse session (if that’s what you are using):
At this point you can debug from Eclipse normally. It maybe a little slower than normal due to the increased latency because you are remote, but you are debugging your application hosted in Bluemix!
The code for the proxy and debug server is a bit more complicated than what we showed earlier, but for our basic implementation its still pretty small. The source code is available here: remotedebugSrc.tar.gz
NOTE: If you use the local proxy and debug server without the helper script provided, one thing to make sure of is that you connect the local proxy to the debug sever over SSL otherwise your debug data will not be protected.
So the answer to “Can I debug in Bluemix” seems to be “yes with some work”.
The work around is not perfect as you have to execute an additional script to prepare the debug environment, and debugging may be slower because of the proxy and intermediary hops. There are still some security concerns which you may or may not be concerned about (the connection within Bluemix from the application to the debug server is not encrypted as an example).
I’m sure we’ll see a more integrated approach from Bluemix. Until then what we have described provides a way to debug which has got to be better than not being able to do it at all.
This blog and the associated code samples were a collaborative effort between Michael Dawson (firstname.lastname@example.org), Yi Xiao (email@example.com) and Stephen J. Kinder(firstname.lastname@example.org).
* Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates