Dr. Debug's Advice
Dr.Debug 3100008NWU 7,098 Views
Some of you may already know me, some may not. And some might know me, but might not be aware of it.
But don't worry, we'll sure get to know each other soon.
As I said, my name is Dr. Debug and I work in IBM Support. I have a combined age of about 1400 years and more than 600 years of experience in IBM Support.
My area of specialisation is the WebSphere Portfolio, including the Application Server, DataPower, Business Process Manager (BPM) as well as the IBM PureApplication System.
You may think: Wow, this guy is a bit of a know it all - and you're right. I have seen many many things in my career. And I have always found a solution to a problem.
Put a bandaid on the wound first, then find the cause, kill the pain, and get you back on track all healthy and happy.
That's my job. And I love it.
I thought I'd share my experience with you, so you gain some insights of how I help patients, so you may soon be able to assist me.
And I hope the hints, procedures, tipps and tricks I will show you will be beneficial for your future work with our products. At least they became quite handy over the years for me.
So, welcome to my new blog, enjoy your stay, come again, bring some friends and if nothing helps:
Take two of these and call me in the morning....
G'day my friends,
Today I'd like to share with you an excellent piece of work that will make your debugging life so much easier.
This IBM Support TV video hosted on our youtube channel will show you various ways to record HTTP traffic using your browser:
This becomes quite handy in cases where IBM Support asks your for HTTP traffic captures or Fiddler traces, but your company policy does not allow you to install fiddler.
I would suggest to bookmark this video. We will need this as a basis for future debugging.
And if neither this video, nor fiddler helps...
Take two of these and call me in the morning,
Dr.Debug 3100008NWU Tags:  har performance slow bpm debug loading coaches long coach 10,866 Views
Good morning, Fellas!
Today, I will tell you why my last posting with the video of how to capture http traffic is so important.
You all probably know the symptoms - you run your Coach and it's loading. You check the news in a separate tab, you jump back to the Coach, still loading,
loading aaaaaaand loading. Okay, I am exaggerating a bit, but sometimes the loading times feel like they are taking forever.
Now, before contacting IBM Support to help you solving your performance issue, I wanted to share some steps you can use to further diagnose your problem.
And for that, we will need this *.HAR (HTTP Archive) file that is being generated.
Now, to look at this HAR file, you will require an HAR Viewer (http://www.softwareishard.com/blog/har-viewer/).
Now simply drag&drop your HAR file onto the page and click on the Preview tab.
Here you will see all the requests with request/response, as well as header information, the response codes to the requests and most importantly - the timings.
Since we have long loading times for our Coach, we are particularly interested in those loading times.
A good indicator is the bar on the right hand side. If it is purple and the number given next to it is quite small (it's the actual time this request took in milliseconds), that's good.
Just some general remarks:
The purple bar indicates the wait time, i.e. the time the client waited for a response from the server. The smaller, the better. Of course it depends on the size of the request, network latency, server load etc).
You may see a brownish bar that indicates Blocking time. This is the time the request spent waiting for a network connection.
Blue bars indicate the time the DNS lookup took, green is the time it took to create the actual TCP connection.
Red bars show the time spent on sending the HTTP request to the server, and as said earlier, purple is the waiting time for the response.
They grey bar stands for the time it took to receive and read the complete response from the server.
In some cases you might even see SSL timings, which show how long the handshake etc took.
A good sample can be found here.
Just open the twisties at the bottom named "Cuzillion" and you can review these timings.
Or you can reload this page, grap the HAR from your browser console and it might look something like this:
I had a patient the other day, whose HAR file showed something very interesting. The actual waiting times were pretty small, all only a couple of milliseconds. Yet he still told me his Coach was ailing. There were pages and pages of requests. I stopped counting after 200. The farther i scrolled down in the list, I noticed that the brown bar continuously increased.
Brown bar? Yes! Blocking time! What was that again?
Exactly! The time waiting for a connection. I am not a plumber, but this smelled like a clogged drain. To use the medical term - the patient was probably suffering of constipation.
I asked my patient to take the longest taking call (it was a REST call - I could see that from the request details) - and run it against the server from the REST-API tester while capturing the HTTP traffic again. In isolation, no other requests, just this one.
The result: It finished in no time. So we knew it was not the DB or Server that had a problem with that request. This supported the drain theory.
What can cause such a constipation are the connection- and thread-pool settings for your server. In other words: If you eat more than you can egest, at some point you will feel constipated.
I gave him the advice to increase the pool sizes in an iterative process, always monitoring the performance of his Coach while doing so.
And voilá - the clogged drain was free again.
And if none of this helps, take two of these and call me in the morning.
Your Dr. D
Tutorial: How to investigate startup issues of Eclipse based products where the Java Process terminates unexpectedly during the start-up
Dr.Debug 3100008NWU Tags:  im crash iid howto log tutorial java trace dr.debug eclipse debug 10,412 Views
Good morning friends of the hands-on debugging,
Some of you may have already encountered such a situation. You get your morning coffee, boot up your workstation, you start your IBM Integration Designer (IID) and bang - there it is. A crash notification. What a good start in your day.
Java was started but returned exit code=1
If you have not encountered this yet - Congratulations! I would still recommend reading this post, maybe some days you will need this .
So, what's next?
Step 1: Prepare yourself for some debug fun. If your coffee is gone, I recommend getting another one, then continuing with the below steps. If you still have coffee left, move right on to the next step.
Step 2: Did this installation ever work or did you just install it? If this error shows up after a new installation of the Eclipse based product (here IID), check out the installation logs to make sure it was installed without any exceptions. To do so, have a look at the IBM Installation Manager log files under <Installation_Manager_Install_Dir>/logs
In my case:
Open the index.xml file and review all logs linked in there for any exceptions or failures during the installation.
Failed or partially failed installations may cause such statup issues.
Step 3: Did anybody mess with your Eclipse startup parameters? If this is not a new installation and the exception happened all of a sudden, maybe some JVM arguments were changed and are hence causing this crash. Always a good candidate is the -Xmx value, which specifies the maximum memory allocation pool for the Java Virtual Machine (JVM). A lot of users still think: The more, the better! Unfortunately they overtune the JVM and the memory allocation is simply too large for the JVM to be successfully created. If changes were made to the JVM settings, please revert those changes back to the default values (you can for example compare them to a working or untouched system). Then try to launch your Eclipse based application again.
Step 4: Create Javacore log files to gain more insight into the root cause of the crash. As Eclipse is still in the start-up process when the crash occurs, the JVM errors cannot be written to the Eclipse error logs under <workspace_root>/.metadata/.log
This should enforce the creation of a javacore.txt file in case of a JVM crash. In our case here, you will find it in the IID root directory after the next restart.
And if nothing helps, take two of these and call me in the morning.
Your Dr. D
Dr.Debug 3100008NWU 8,367 Views
Good morning dear Debuggers,
Tutorial: How to investigate startup issues of Eclipse based products where the Java Process hangs during the start-up
Dr.Debug 3100008NWU Tags:  error equinox iid startup freeze eclipse hang log debug osgi 18,168 Views
Dear fellow Debuggers,
The other day I ran into a problem, where my IBM Integration Designer (IID) did not start properly. I tried to start the application, but the startup never finished and was hung. Frozen - no way to stop it except killing the process. Alternatively I could have waited for it to crash..
If it does crash during the workspace initialization phase, a log file is written to the workspace/.metadata/ directory. The file is named ".log" and is always worth a look. In my case, I saw the following exception:
!ENTRY org.eclipse.osgi 2 0 2015 !MESSAGE While loading class "org.eclipse.core.runtime.RegistryFactory", thread "Thread[Start Level Event Dispatcher,5,main]" timed out waiting (5000ms) for thread "Thread[Component Resolve Thread,5,main]" to finish starting bundle "org.eclipse.equinox.registry_3.5.0.v20100503 ". To avoid deadlock, thread "Thread[Start Level Event Dispatcher,5,main]" is proceeding but "org.eclipse.core.runtime.RegistryFactory" may not be fully initialized. !STACK 0 org.osgi.framework.BundleException: State change in progress for bundle "reference:file:/C:/IBM/SDPShared/plugins/org.eclipse.equinox.registry_3.5.0.v20100503.jar" by thread "Component Resolve Thread". at org.eclipse.osgi.framework.internal.core.AbstractBundle.beginStateChange(AbstractBundle.java:1077) at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:282) at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:417)
If it do Org.eclipse.equinox.registry is one of the first eclipse plugins started and usually indicates that a bundle was not loaded properly.
In my case, a simple restart of my machine and then starting IID with the -clean option resolved the error.
And if nothing helps, take two of these and call me in the morning.
Dr.Debug 3100008NWU Tags:  fixlist bpm server debug ps dbschema process_server fix table lsw_server 7512 post-installation upgrade 12,678 Views
Yesterday I came across an interesting issue.
After upgrading from BPM v184.108.40.206 to v220.127.116.11 the Server List in my Process Center was completely empty. The servers were reachable and available though.
When I clicked the Servers tab, I found the following exception in the log:
[3/11/15 12:16:12:430 GMT+01:00] 00000061 wle E CWLLG1274E: An exception occurred.
I looked at the LSW_SERVER table of my BPM database and the server entries were listed.
Reviewing the table definition of my LSW_SERVER table, I saw that the password column had a definition of varchar(256), although after the installation of FP2 (or JR42774) it was supposed to have varchar(1000). So something must have went wrong during the installation of the fixpack - most likely during the post-install steps where the db schema was upgraded.
After I updated the schema (as described in the post install steps) I nulled the passwords in the LSW_SERVER table, as the heartbeat would simply re-add them afterwards.
Dr.Debug 3100008NWU Tags:  integrationdesigner bpm flow tutorial view iid mediation how-to ibm support debug 9,978 Views
Good morning friends,
Today I would like to show you, how you can set a specific editor to display a certain file type.
As you all may already know, the IBM Integration Designer (IID) is a development tool where you can use a Graphical User Interface to create and test your applications. For this GUI there is a set of editors available that will display your artifacts in a graphical view instead of a file view.
An example would be the mediation flow, which is opened with the "Mediation Flow Editor" by default when you open the flow.
So, you may run into situations where the view is not available, so the mediation flow might open in the XML file view. First attempt would be to open it from the context menu (BI view >> right click on your mediation flow name >> Open With >> Mediation Flow Editor). If you are as unlucky as I am, this option is not available, and the context menu looks like this:
How can I get my Mediation Flow Editor back? Well, it's not hard if you know how. And I am going to show you how.
All you need to do is to register the editor for the related file type again. In our case the mediation flow (associated to the .mfc file type).
To do so, click on Window >> Preferences and click on the little eraser icon to remove all filters.
Then click on General >> Editors >> File Associations
You will find a list of file types where you can associate the correct editor to the according selected file type.
In our example, we will select the file type .mfc. You will now see the editors in the "Associated editors" section. With the "Add..." button we can now add additional editors for this file type. And we will now add the 'Mediation Flow Editor".
All we need to do now is save our changed and go back to the BI view where we will open the context menu to verify the Mediation Flow Editor was added:
And we're there. It worked.
And if all of that does not help, take two of these and call me in the morning.
Your Doc D.
When a Coach validation error occurs in a Client-Side Human Service in BPM 8.5.5 previously updated variables remain unchanged
Dr.Debug 3100008NWU Tags:  client-side v855 bpm human-service cshs error 8.5.5 debug variable coach validation 9 Comments 21,046 Views
Dear fellow Debuggers,
Last week I came accross an interesting issue and I thought I would share my findings with you.
In BPM version 8.5.5 the new Client-Side Human Service (CSHS) was introduced.
The Service looked like this:
It is important to understand, that the validation step was a script named "Validation Script" by the user. The validation service has a green icon (V) leading to the script step. If you don't see it, this is just a regular script step (not true validation). See this Knowledge Center link.
From the Knowledge Center:
Please note the green checkmarks or Vs.
However, to configure this validation service, you would have to set the "fire validation property" to "before" for the line to the boundary event. But the CSHS does not support this kind of validation. This is only available for Heritage Coaches in v8.5.5.
In our usecase, when we clicked the button the values of the variables should have been changed when the validation error was triggered, however, the variables still showed the "old" values.
So why was that?
Well, in v8.5.5 for the CSHS, when validation errors occur, ONLY the validation errors info is returned back to Coach for display. Any changed variables are NOT returned to the Coach. This explains why our variables showed the old values, and not the ones we assigned to them in our Client-Side Script.
With this in mind, we can now understand why our variables remained unchanged.
My dear Debug-Friends,
Today, I would like to explain a topic, that arises from time to time when you need to upload large data to a PMR and email just won't work.
IBM Support offers you to upload the data using FTP. And I will show you how you can utilize that.
You must use the appropriate naming conventions as shown in the examples below to make sure your data makes it to your PMR:
Where the parts have the meaning:
xxxxx = PMR number
bbb = Branch Office (if known)
ccc = IBM Country Code (e.g. Germany 724)
yyy.zzz= Short description and the file type, e.g. myLogs.rar, traces.zip etc.
As the login was done with an anonymous UserID, no file listing is allowed on the server. You will not be able to check what files you have already uploaded, which means that you will need to do this very carefully.
If this does not work for you, take two of these and call me in the morning.
Your Dr. D