Special Weapons!

Automatically Generate Introscope Wily ProbeBuilder Directive (PBD) files to Troubleshoot and Monitor WebSphere Portal Performance

1. Enable the following trace in WebSphere Portal Server

   *=info:com.ibm.wps.=all:com.aptrix.pluto.renderer.*=all:com.aptrix.pluto.cmpnt.*=all:com.ibm.workplace.wcm.services.*=all:com.ibm.websphere.personalization.*=all:com.ibm.dm.pzn.ui.*=all

2. Additional Trace settings for various areas of WebSphere Portal can be also used. These settings can be found at the following URL:

   http://www-10.lotus.com/ldd/portalwiki.nsf/xpDocViewer.xsp?lookupName=IBM+Web+Content+Manager+7+Product+Documentation#action=openDocument&res_title=WebSphere_Portal_runtime_logs_wcm7&content=pdcontent

3. Increase the size of the trace files to 100MB and 20 historical files

4. Run a Manual test for a Page that has a long page load time

5. Stop the tracing by setting the following trace string *=info

6. For AIX, zLinux, Linux and other Unix Platforms, copy the trace files created and run the follow command to use the TraceAllMethodsOfClass Directive
   awk '{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {print "TraceAllMethodsOfClass: " $7 " BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE"} }' $1 | sort -k 2,3 -u | egrep -i '.*\..*\..*' | sed -e 's/~/|/g' | sed -e 's/QUOTE/\"/g'

This will print the following for each class Identified

TraceAllMethodsOfClass: com.ibm.wps.um.DataGraphConverter BlamedMethodTimerHC “Custom Tracers|WPSAll|{classname}|{method}:Response time (ms)"

7. For AIX, zLinux, Linux and other Unix Platforms, copy the trace files created and run the follow command to use the TraceOneMethodOfClass Directive so that only methods that appear in the trace are traced

   awk '{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {if (match($8,/\(/) >0) {methodname = substr($8,1,match($8,/\(/)-1) } else {methodname=$8}; print "TraceOneMethodOfClass: " $7 " " methodname " BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE "}} ' %1% | ssort -k 2,4 -u | egrep -i '.*\..*\..*' | sed -e 's/~/|/g' | sed -e 's/QUOTE/\"/g'

8. For Windows, if you have installed the unix utilities for windows found at http://unxutils.sourceforge.net/UnxUtils.zip

9. For Windows, once the UnixUtils.zip is download and installed you must add the directory containing the executable files to your Windows Path Environment variable and rename the sort.exe to ssort.exe since windows comes with its own sort utility.

10. For Windows, Also Note that the awk command is replaced with gawk and functions very similar to awk

11. For Windows, to use the TraceAllMethodsOfClass directive the command is as follows:

    gawk "{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {print """TraceAllMethodsOfClass: """ $7 """ BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE """} }" %1% | ssort -k 1 | uniq | egrep -i ".*\..*\..*" | sed -e "s/~/|/g" | sed -e "s/QUOTE/\"/g"

12. For Windows, to use the TraceOneMethodOfClass directive which only add traces for the methods found in the trace log the command is as follows

    gawk "{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {if (match($8,/\(/) >0) {methodname = substr($8,1,match($8,/\(/)-1) } else {methodname=$8}; print """TraceOneMethodOfClass: """ $7 """ """ methodname """ BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE """}} " %1% | ssort -k 2,4 -u | egrep -i ".*\..*\..*" | sed -e "s/~/|/g" | sed -e "s/QUOTE/\"/g"

13. The script can be executed against any WebSphere trace file.

14. To write the output of the script to an actual pbd file the command must be added to a Unix Shell script or a Windows batch file and redirected to a file when running the shell script
    
    Sample Script for Unix Platforms
    #!/usr/bin/sh

    #AbStract: This generates wily ProbeBuilderDirective (PBD) Files from Portal trace logs Using the TraceAllMethodsOfClass or TraceOneMethodOfClass

    # depending on which command is commented out

    # chmod 755 generateWilyPBD4Portal.sh

    # Command Format:

    # generateWilyPBD4Portal.sh trace.log > portalTracer.pbd

    # copy the portalTracer.pbd to the directory <WILYAGENTHOME>/wily/hotdeploy

    # Uncomment this to use TraceAllMethodsOfClass Directive

    awk '{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {print "TraceAllMethodsOfClass: " $7 " BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE "} }' %1% | ssort -k 2,3 -u | egrep -i '.*\..*\..*' | sed -e 's/~/|/g' | sed -e 's/QUOTE/\"/g'

    # Uncomment this to use TraceOneMethodOfClass Directive

    #awk '{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {if (match($8,/\(/) >0) {methodname = substr($8,1,match($8,/\(/)-1) } else {methodname=$8}; print "TraceOneMethodOfClass: " $7 " " methodname " BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE "}} ' %1% | ssort -k 2,4 -u | egrep -i '.*\..*\..*' | sed -e 's/~/|/g' | sed -e 's/QUOTE/\"/g'

    Sample Script for Windows Platform
    @echo off

    REM AbStract: This generates wily ProbeBuilderDirective (PBD) Files from Portal trace logs Using the TraceAllMethodsOfClass or TraceOneMethodOfClass

    REM depending on which command is commented out

    REM Command Format

    REM generateWilyPBD4Portal.bat trace.log > portalTracer.pbd

    REM copy the portalTracer.pbd to the directory <WILYAGENTHOME>/wily/hotdeploy

    REM Uncomment this to use TraceAllMethodsOfClass Directive

    gawk "{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {print """TraceAllMethodsOfClass: """ $7 """ BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE """} }" %1% | ssort -k 2,3 -u | egrep -i ".*\..*\..*" | sed -e "s/~/|/g" | sed -e "s/QUOTE/\"/g"

    REM Uncomment this to use TraceOneMethodOfClass Directive

    REM gawk "{if (match($6,/^>$/) == 1 || match($6,/^<$/) == 1) {if (match($8,/\(/) >0) {methodname = substr($8,1,match($8,/\(/)-1) } else {methodname=$8}; print """TraceOneMethodOfClass: """ $7 """ """ methodname """ BlamedMethodTimerHC QUOTECustom Tracers~WPSAll~{classname}~{method}:Response time (ms)QUOTE """}} " %1% | ssort -k 2,4 -u | egrep -i ".*\..*\..*" | sed -e "s/~/|/g" | sed -e "s/QUOTE/\"/g"

15. Note there are many classes and methods that will be traced so you will need to set the Wily property introscope.agent.transactiontrace.componentCountClamp to a large number. To start with set it to a minimum of 75000.

16. You can place the portalTracer.pbd file in the <AgentHome>/wily/hotdeploy

17. Note these traces will impede performance but it will help identify the code that is impacting the page load times.

18. Since there are many classes and methods to be traced, the trace can possibly cause the Java Agent to start incorrectly or hang if this occurs you'll need to use only a subset of the classes or methods to be traced.

The number of installations that don't monitor response time is actually much higher than you might expect.  I had to put together the script below to provide some basic monitoring and notification for an issue that was occurring.  Essentially it's a ping tool for Connections.

There are some fields that need to be updated.  I placed a "*" next to the line where the changes need to be made.  By default it runs every minute for 10 hours.  It saves the response times in a perfAlert.log file.  Any requests that take over 5 seconds to complete will generate an alert.

Dim counter
counter = 0
*user="user"
*pw="password"

*domain="domain.ibm.com"
' Threshold is measured in seconds
dim threshold
threshold=5


'***Time between checking in milliseconds
interval=60000

iterations=600

dim startdate
Dim objXmlHttp
Dim ResponseTime
Dim StartTime
Dim EndTime
Set objXmlHttp = CreateObject("MSXML2.ServerXMLHTTP")
Set objFSO=CreateObject("Scripting.FileSystemObject")
outFile="perfAlert.log"
Set objFile = objFSO.CreateTextFile(outFile,True)


lResolve = 5 * 1000
lConnect = 60 * 1000
lSend = 60 * 1000
lReceive = 60 * 1000
objXmlHttp.setTimeouts lResolve, lConnect, lSend, lReceive
startdate = Now



'***Set host variable to IP address of Connections servers and IHS nodes
addr1 =  domain & "/activities"
addr2 =  domain & "/communities"
addr3 =  domain & "/profiles"
addr4 =  domain & "/homepage"
addr5 =  domain & "/news"
addr6 =  domain & "/blogs"
*directaddr1 = "nodeA.ibm.com:9445/profiles/"
*directaddr2 = "nodeHome.ibm.com:9444/homepage/"




Do
  counter = counter + 1
  objFile.Write counter & ": " & Now  & vbCrLf
 
'***Response Time Check
CheckHost("https://" & addr1 & "/")
CheckHost("https://" & addr2 & "/")
CheckHost("https://" & addr3 & "/")
CheckHost("https://" & addr4 & "/")
CheckHost("https://" & addr5 & "/")
CheckHost("https://" & addr6 & "/")
CheckHost("https://" & directaddr1 & "/")
CheckHost("https://" & directaddr2 & "/")

  WScript.Sleep interval
    
Loop Until counter = iterations


'***Clean up***
Set startdate = Nothing
Set objXmlHttp = Nothing
Set ResponseTime = Nothing
Set StartTime = Nothing
Set EndTime = Nothing



Sub CheckHost(host)
  On Error resume next


  StartTime = Timer()
  objXmlHttp.open "GET", host , False, user, pw
  Const SXH_SERVER_CERT_IGNORE_ALL_SERVER_ERRORS = 13056
  objXmlHttp.setOption 2, SXH_SERVER_CERT_IGNORE_ALL_SERVER_ERRORS
  objXmlHttp.send
  EndTime = Timer()
  ResponseTime = FormatNumber(EndTime - StartTime, 2)


  '***Get the value of the reference variable A***
  A = ResponseTime

  objFile.Write host & " took " & A & " seconds" & vbCrLf
  'WScript.Echo objXmlHttp.responseText
WScript.Echo threshold
WScript.Echo A
if A > 5 then
    Call SendEmail (A, host)
    objFile.Write "Sending email" & vbCrLf
    
End IF
End Sub




sub SendEmail (delay, address)

*strSMTPFrom = "testHttp@ibm.com"
*strSMTPTo = "michael_lee@us.ibm.com"
*strSMTPRelay = "smtpserver"
strTextBody = "EOM"
strSubject = "Alert! " & address & " took " & delay



Set oMessage = CreateObject("CDO.Message")
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSMTPRelay
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
oMessage.Configuration.Fields.Update

oMessage.Subject = strSubject
oMessage.From = strSMTPFrom
oMessage.To = strSMTPTo
oMessage.TextBody = strTextBody



oMessage.Send

end Sub
 

If you run into a situation where the processor utilization is high - the process for determining the cause isn't the easiest thing to find on your typical Google search.  So here it is:

For the sake of brevity I'm just going to do Linux

1) Take a javacore.   This is easy on a Unix-type system.  Just find your process and run "kill -3 <PID>"

2) Collect some top data

# top -bH -d 5 > top.out

3) Find the threads that are using the most processor usage in the top output

12771 user   20   0 6028m 2.3g  59m R 19.6  7.4  48:33.59 java               
11880 huser   20   0 6028m 2.3g  59m R 19.6  7.4  46:41.38 java

4) Open up calculate and convert the PID to a hexadecimal value

For example 12771 -> 31E3

5) Open up the javacore and find your thread via the native thread ID.

3XMTHREADINFO      "WebContainer : 46" J9VMThread:0x00000000C4506C00, j9thread_t:0x00007FA9F66D2E10, java/lang/Thread:0x00000000091B2350, state:CW, prio=5
3XMTHREADINFO1            (native thread ID:0x31E3, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00007FA99EFE0000, to:0x00007FA99F021000, size:0x41000)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/awt/image/DataBufferInt.<init>(DataBufferInt.java:52(Compiled Code))
4XESTACKTRACE                at java/awt/image/Raster.createPackedRaster(Raster.java:469(Compiled Code))
4XESTACKTRACE                at java/awt/image/DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1026(Compiled Code))
4XESTACKTRACE                at java/awt/image/BufferedImage.<init>(BufferedImage.java:323(Compiled Code))

You want to do this on different datasets so that you can confirm the pattern behavior is the same.  So essentially just repeat the process

And that's it.  Now that you have found the culprit, all you have to do is determine why that thread path is using up too much cpu.

Apple iPhone owners please read.

Do you have friends who previously had iPhones and switched to Android, who now ignore all of your texts?

Well maybe you aren't being ignored. Apple has a setting that will not send messages to Android numbers if the connections were previously established via iMessage.

To fix this:

Go to > Settings > Messages > and turn on the feature  "Send as SMS" (Send as SMS when iMessage is unavailable.).

If this is working properly, your SMS texts to Android users should appear green.  iMessage messages are blue.

If you do enough installs or are just lucky - you are going to into the DSX exception.   The symptom of this is that you can usually log into Profiles but you can't log into any of the other Connections applications.

  Caused by: com.ibm.connections.directory.services.exception.DSException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: com.ibm.connections.directory.services.exception.DSOutOfServiceException: CLFRK0004E: Directory Service Extension(DSX) received a HTTP response from URL 'https://something.com/profiles/dsx/instance.do?login=UID' expected XML, received unexpected content-type 'text/html;charset=UTF-8'! 

The problem, for me at least, is that it is not clear at all what the exception means, much less what could be causing it.  The worst part is that there are a lot of examples of people having this error but very few examples of them resolving it.  This can lead to days of downtime trying to get things working

So here are some possible reasons for this

1) SSL key/certificate issues between the IHS plugin and the Websphere cell.

2) The connectionsAdmin account can't authenticate correctly.  It may be necessary to remap some of the application admin roles.  This didn't affect me but apparently it's possible.

3) The LotusConnections-config.xml interservice URL references are misconfigured.  You can try setting them directly to the WAS server and port to avoid whole host of possible communication issues that could come into play

4) The SSO configuration is not properly setup.   Make sure the domain is correctly identified and the SSO is enabled.

edit 4-23-14 ->  5) Almost forgot to add performance issues.  The above issues deal with possible configuration.  But if profiles (or communities) isn't available for performance reasons, you can expect to see the issue occur.

The core of the issue is that secure communication isn't being negotiated correctly for Profiles (or Communities) to validate the user account.  If the above steps don't help, you can put the trace below on the application that's having the problem.

*=info:com.ibm.connections.httpClient.*=all:com.ibm.connections.directory.services.*=all

Today's post is about Lotus Mobile Connect and how it can be used with Traveler to make sure that mobile clients can reach the right Traveler HA Pool or Traveler server, in a secure manner with a clientless VPN connection.

First of all, many of you don't even know what LMC is.  What is Lotus Mobile Connect?  In a nutshell...

VPN solution supports all access points

● Laptop and Desktop (client)

● Handheld mobile device (client)

● Browser (clientless HTTP Access)

• Proven ROI - least cost routing & data optimization

● Network optimizations reduces network data costs

• Secure end-to-end connect to corporate intranets

● Strong authentication and support for RSA Secure ID

● End-to-end encryption: DES, Triple DES, RC5, AES and FIPS 140-2 certified

Clientless: No client software installation, application and platform agnostic!

Single Sign On: LMC Gateway + application authentication

Enhanced Security

– Same strong authentication and encryption algorithms as the LMC Mobility Client

– FIPS 140-2 certified platform. Contains latest secure SSL/TLS ciphers and industry standard authentication mechanisms

– Two way certificate validation available for additional layer of trust

– Configurable to run simultaneously with full VPN sessions. Multifunction, IT configurable remote-access solution (Example: Subset of users restricted to specific apps via clientless access only. Other set of users allowed expanded enterprise access with LMC Mobility Client.)

So what does it mean to be "Traveler aware"?

LMC 6.1.4 Traveler Aware Features

• Recognition of Traveler URLs,
• applies data flow optimizers and HTTP header tags,
• previous password recognition (prevent LMC server lockout).

LMC 6.1.5 (NOW IN BETA!!) Traveler Aware Features

With the current set up of Traveler, a user has to know their URL to their Traveler server or Traveler HA (High Availability) Pool.  When customers are deploying to large numbers of people, it can be a pain to have so many profiles out there pointing to different Traveler servers or pools.  In the deployment that I just did, they are planning on 4 pools (I completed 2), each pool supports about 10,000 users.  However, this customer did not want to have 4 different possible Traveler profiles on their Apple iOS Mobile Devices.

Enter the new LMC 6.1.5 Traveler aware feature...

The new LMC 6.1.5 Traveler aware feature allows LMC to go into a single URL mode.  LMC "learns" which server or pool of servers a user belongs to, and makes sure that the user is routed properly.  So in a High Availablility scenario, you still need a load balancer in front of the pool, but if you have more than one Pool, LMC will route the users to the right pool.  It knows how many users belong to each server or pool of servers, and will divvy up new users to a less loaded pool.  Pretty cool, huh?  Look for LMC 6.1.5 some time in Q3!

Several people have asked me about the Traveler deployment that I've done with a 10 server Traveler HA pool.

This is a large global customer, but we are only rolling Traveler out to the US users (about 40K devices).

The plan is to start with a roll out of about 10K users - one Traveler pool. 

We will be using Lotus Mobile Connect 6.1.4 initially as sort of a pass through.  When LMC 6.1.5 becomes available, we'll be upgrading over the top and activating the Single URL feature.  This is a new feature of LMC that makes LMC "Traveler aware" and will do user assignment and routing from a single URL.

In the past versions of Traveler, users have to know their Traveler URL. The HA version, there is a single hostname (such as a load balancer or Virtual IP) that sits in front of each Traveler pool. The users would need to know their URL for that pool.  If you only have up to 10K devices, that is not a problem, but in the case of this customer, we have 40K devices and thus 4 Traveler pools of 10 Traveler servers.  That means 4 possible Traveler URLs and the users have to know which one.  Now this is where LMC comes into the picture. The new version (eta is October 1ish) will handle routing.  LMC learns where the existing users belong and will route them accordingly.  As new users come to the LMC that are not yet assigned, LMC will route them to a pool that has capacity.

The beta for LMC 6.1.5 should be available this week, however it is only on Linux and with an Oracle back-end database.  The Windows version will be available by the end of July and should support MSSQL.  We'll be deploying that as a proof of concept when it becomes available.

My customer is using a MDM provider that will be in front of the LMC.  The MDM provider will handle the authentication of the user against a Sun One Enterprise Directory.  We have authentication configured to pass the tokens from the MDM provider to LMC and then SSO between LMC and Traveler.  You export the LTPA tokens from the LMC and into Domino to get that working.

OK so that's a glimpse of the project so far. Here is a list of some of the things that I have learned:

Requirements: http://www-01.ibm.com/support/docview.wss?uid=swg27027499&myns=swglotus&mynp=OCSSYRPW&mync=R

Infocenter for Admins: http://www-10.lotus.com/ldd/dominowiki.nsf/xpViewCategories.xsp?lookupName=Administering%20Lotus%20Notes%20Traveler%208.5.3%20Upgrade%20Pack%201

Infocenter for users: http://www-10.lotus.com/ldd/dominowiki.nsf/xpViewCategories.xsp?lookupName=Using%20Lotus%20Notes%20Traveler%208.5.3%20Upgrade%20Pack%201

Known Issues: http://www-01.ibm.com/support/search.wss?lang=en&loc=en_US&r=10&cs=utf-8&rankfile=0&cc=&spc=&stc=&apar=include&atrn1=SWVersion&atrv1=8.5.2&sort=rk&tc=SSYRPW&go.x=13&go.y=13&dc=DB520%20D800%20D900%20DA900%20DA800%20D600%20DB530%20DA600%20D700%20DB500%20DB400%20D100%20DB540%20DB510%20DB55&dtm

Use Domino Enterprise server. We don't support the other types (messaging, utility, etc)

The main thing is sizing.  We are planning on about 1000 users per traveler server.  10 Traveler servers per pool. Each Traveler HA pool can only support about 10,000 devices.  If you have more than 10K devices, you should start a new pool.  

We went 8CPU, 16GB RAM with 64bit OS for each Traveler server to support this many users.  You do not want to use a 32bit OS.

Your Traveler servers should be co-located with the Mail servers for performance sake.  We highly recommend that the database server (db2 or sql) has physical hardware with a RAID 0/1 array.  The bottleneck will be in the disk i/o.  

For best performance on the database server, you should separate the transactional logs from the data and system drives.  We also dedicated a logical disk to temporary files. Example
c:\ - system files

d:\ - data files

e:\ - temporary files

f:\ - transaction logs

g:\ - backup files

The Database server on the back end (DB2 or SQL) should have redundancy otherwise you have a single point of failure.  If the database becomes unavailable for any reason, users will not have any service.

Also the database gets created with a full text index by default.  They should maintain the database as they would any other SQL or DB2 database.

We no longer use the /servlet/traveler path to get to the servlet. It is now moved to /traveler.  Also we don't use the Domino Web Engine for a servlet engine, its some kind of osgi servlet engine outside of Domino.

If you are going to use a MDM provider or LMC for clientless authentication, be sure to use Verisign 3rd party SSL certificates. Otherwise the mobile devices won't have the certificates on them.

Apple devices when used with a VPN will not receive notifications when they get new mail/calendar items. This is a limitation of the Apple protocols that we use.  A clientless solution like LMC or MDM (such as Mobile Iron) will workaround this limitation.

Encrypted emails cannot be read from Traveler.  You will need to use the Traveler Companion app for that. Users will get a message saying that their message is encrypted and they have to read it from Notes, iNotes or the Traveler Companion app.