Dougclectica

with Tags: ihs

All posts

Find long-running IHS (web) requests

DougBreaux Visits (2287)

Quick way to locate or watch for long-running IHS requests:

Logging response times

Enable microsecond response time logging with the %D LogFormat option. For example:

LogFormat "%h %l %u %t \"%r\" %>s %b %D" common

Finding Responses longer than 1 second

Then, if you put it at the end of the line like that, the following egrep pattern could filter out, say, those items that take longer than 1 second:

egrep "[0-9]{7}$" access_log

Where 7 digits means you've gotten past 999,999 microseconds.

If you don't want it at the end of the line, you could add some fixed delimiters to locate it in the log string in another manner.

Actively watching for long-running responses

Or to actively watch for such requests (assuming you have command-line tail on your platform):

tail -f access_log | egrep "[0-9]{7}$"

Tags: httpd apache tuning ihs logging grep

mod_mpmstats to analyze thread usage in IBM HTTP Server

DougBreaux Comment (1) Visits (4637)

(Related to my earlier post on mod_status.)

Found this useful tidbit today from IBM HTTP Server Performance Tuning:

For IBM HTTP Server 7.0 and later, mod_mpmstats is enabled automatically.
Check entries like this in the error log to determine how many simultaneous connections were in use at different times of the day:
[Thu Aug 19 14:01:00 2004] [notice] mpmstats: rdy 712 bsy 312 rd 121 wr 173 ka 0 log 0 dns 0 cls 18
[Thu Aug 19 14:02:30 2004] [notice] mpmstats: rdy 809 bsy 215 rd 131 wr 44 ka 0 log 0 dns 0 cls 40
[Thu Aug 19 14:04:01 2004] [notice] mpmstats: rdy 707 bsy 317 rd 193 wr 97 ka 0 log 0 dns 0 cls 27
[Thu Aug 19 14:05:32 2004] [notice] mpmstats: rdy 731 bsy 293 rd 196 wr 39 ka 0 log 0 dns 0 cls 58

This is a quick way to look at your IHS load over time.

The fields logged are described in the table below:

field description

rdy (ready) the number of web server threads started and ready to process new client connections

bsy (busy) the number of web server threads already processing a client connection

rd (reading) the number of busy web server threads currently reading the request from the client

wr (writing) the number of busy web server threads that have read the request from the client but are either processing the request (e.g., waiting on a response from WebSphere Application Server) or are writing the response back to the client

ka (keepalive) the number of busy web server threads that are not processing a request but instead are waiting to see if the client will send another request on the same connection; refer to the KeepAliveTimeout directive to decrease the amount of time that a web server thread remains in this state

log (logging) the number of busy web server threads that are writing to the access log

dns (dns lookup) the number of busy web server threads that are performing a dns lookup

cls (closing) the number of busy web server threads that are waiting for the client to acknowledge that the entire response has been received so that the connection can be closed

Tags: httpd apache mpmstats stats ihs

Using IBM HTTP Server 6.1 diagnostic capabilities with WebSphere

DougBreaux Comment (1) Visits (4479)

Using IBM HTTP Server 6.1 and later diagnostic capabilities with WebSphere

This could be useful. Apache mod_status is available in IHS, and when it's enabled it provides information about the individual threads running under the httpd processes.

The information is provided as a simple web page via a URL (that of course can be protected by any of the various Apache mechanisms).

With the default httpd.conf file, only the following changes are required:

Uncomment the module loading:

LoadModule status_module modules/mod_status.so

Uncomment the Location setup:

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from .example.com
</Location>

Change the Allow rule to your own domain or IP, or protect the URL with HTTP Basic Auth, or something else.

Tags: httpd plugin apache mod_status debugging websphere diagnostic ihs

Temporarily Disabling WebSphere Cluster Members

DougBreaux Comments (2) Visits (5653)

Background

It seems that in our current configuration the WebSphere plugin does not reliably stop sending requests to a Cluster member when it's been taken down. Or sometimes it seems to take an inordinate amount of time for it to do so, or sometimes it starts sending requests before a restarting member is fully operational.

A quick workaround is to manually edit the plugin's configuration file on each web server to comment-out the unwanted servers. The plugin will automatically detect and apply the changes within a minute or so and stop routing requests to those servers until you manually uncomment them.

Plugin Configuration File Location

On our AIX systems, the plugin configuration file is in a location like:

/usr/IBM/HTTPServer/Plugins/config/<webservername>/plugin-cfg.xml

(If you're ever uncertain you're looking in the correct location, look for the WebSpherePluginConfig directive in the IHS httpd.conf file.)

Plugin Configuration File Contents

Within that file are many configuration items, most should never be manually edited, but the ones we care about for this purpose are the <PrimaryServers> elements within each <ServerCluster> element.

The ServerCluster's "Name" attribute will tell you which element to edit.

Then the PrimaryServers element in that ServerCluster will list each of the individual <Server> Cluster members which are also defined within ServerCluster.

   <ServerCluster ... Name="TestCluster" ...>
      <Server ... Name="serverANode01_TestApp-A" ...>
         ...
      </Server>
      <Server ... Name="serverBNode01_TestApp-B" ...>
         ...
      </Server>
      <PrimaryServers>
         <Server Name="serverANode01_TestApp-A"/>
         <Server Name="serverBNode01_TestApp-B"/>
      </PrimaryServers>
   </ServerCluster>

Removing a specific server from Plugin dispatching

The only change we have to make is to comment-out the specific Server which we want to cease receiving requests. So to disable the TestApp-B server on serverB:

      <PrimaryServers>
         <Server Name="serverANode01_TestApp-A"/>
         <!--Server Name="serverBNode01_TestApp-B"/-->
      </PrimaryServers>

Notes

Don't forget to make this change on each Web Server which is configured to send requests to the Cluster.
Don't forget to make this change for each relevant Cluster within the file. That is, if an entire Node is being taken down, each Cluster's PrimaryServers list must be edited.
If someone knows an easier way to do this, or why stopping a server from the console doesn't automatically, immediately accomplish the same thing, please enlighten me!

Tags: plugin web-server was websphere cluster ihs webserver failover

WebSphere Web Server Plugin

DougBreaux Visits (8287)

After attending a webcast on the WebSphere Application Server 6.1 Plug-in, I thought I'd summarize the content I found helpful.

Overview

The plugin is a native module (DLL or .so) that is installed into the Web Server and is offered each HTTP request to determine whether WAS should handle it. It makes this determination based on the contents of the plugin-cfg.xml file which is normally auto-generated by WAS.

I say, "normally", because it can be manually edited, but doesn't need to be for most operating scenarios. Note that many of these customizations can now be configured within the Administrative Console as well, which significantly reduces the need for manual file editing. This configuration is located under Servers -> Web Servers -> <Server Instance> -> Plug-in Properties. (Visiting this location in the Console is also a useful way to to locate the active plugin configuration file.)

Operation Mechanism

The configuration file specifies virtual hosts (IP names/addresses and ports) and URL paths which are serviced by WAS applications, enabling the plugin to determine which requests it can route to WAS and which it must leave for the HTTP server to handle.

The file also contains values governing performance, maintenance, and failover, such as logging configuration and network timeouts.

Finally, it contains the "transport" information used to forward matched requests to the appropriate Web Containers in WAS. That is, the IP name and port combinations of the different Cluster members' Web Container "Transport Chains".

Affinity

"Affinity" is the notion of sending a user to the same Cluster member (application server JVM) for each subsequent request after the initial one, if the application uses HTTP Sessions. This behavior is required by the JEE specification, and in WebSphere is the responsibility of the Web Server plugin. WebSphere maintains this affinity by appending a "Clone ID" or "Partition ID" to the "Session ID", which the plugin then compares against its configuration to correctly route subsequent requests.

For further detail on the Session ID mechanics, see this post.

New users, those without an existing Session, are distributed to Cluster members in either a weighted round-robin fashion or a random fashion. Round-robin is the default, and it will decrement each cluster member's "weight" each time it sends it a new request, until all members are less than zero, then they will all be reset to their starting values.

Tips or Items of Note

If the Web Server is a 32-bit program, the plugin must also be 32-bit. Same for 64-bit.
The plugin's version must be greater than or equal to the WAS version. However, we know of one case where support said otherwise, so I guess this can be overridden at times.
The RefreshInterval attribute of the Config element is the time, in seconds, between cycles where the plugin checks for changes to its configuration and automatically applies them. Its default value is 60.
The <Log LogLevel> attribute can be used to increase the plugin's logging. It defaults to "Error" but can have values of "Warn", "Stats", or "Trace".
LogLevel "Stats" collects a minimal set of statistics, like data on the number of active connections and on affinity.
All plugin logging is written to the same file, even if your Web Server uses multiple processes. Keep this in mind when tracing the log. The format of a log entry is:
<Date/Time> <Process ID> <Thread ID> - <Message>
In the log, the strings "Bld version" and "Bld date" can be used to determine which version of the plugin is installed by comparing against this list.
The ConnectTimeout attribute of the Server element controls how many seconds the plugin should wait for an initial connection response from a WAS cluster member. All indications are that this should be very fast. Our current setting is 10 seconds, but the webcast presenter suggested that even 5 seconds should be sufficient.
The ServerIOTimeout attribute of the Server element controls how many seconds the plugin should wait for continued IO once a connection has been established. The default is 0, which means wait indefinitely. In other words, don't timeout once communication has begun with a particular server.
The plugin should not return a 500 Response Code (Server Error) unless all cluster members fail to respond. (I'm not sure I believe this empirically, but the presenter clearly asserted it.)
This connection from the plugin to a particular Web Container transport is apparently maintained as an available "stream" even after a particular request is complete. I don't know how quickly it is shut down or refreshed after use, but it's not immediate. Keep that in mind when, say, counting "netstat -an" connections on a particular port.
If all Cluster members have the same round-robin weight, the plugin will factor them down to the value 1.
Since the weighted round-robin or random distribution mechanism only applies to new Sessions, it's theoretically possible that Cluster member loads could become unbalanced if long-running Sessions happen to be allocated to one member and shorter ones to another. That is, new requests will be balanced, but that balancing doesn't take into account existing Sessions allocated to the members. This behavior can be changed by setting the IgnoreAffinityRequests attribute of the ServerCluster element to "false", although I don't know what kind of performance implications this change might have.

Resources

Tags: affinity plugin wlm ihs websphere apache web-server

Tip: Finding exact version and applied patches of IBM HTTP Server

DougBreaux Visits (3821)

apachectl -V

For instance

Server version: IBM_HTTP_Server/6.1.0.31 Apache/2.0.47
Server built:   Mar  3 2010 21:55:48
Build level:    IHS60/webIHS1009.03
Server's Module Magic Number: 20020903:4
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/usr/IBMIHS"
 -D SUEXEC_BIN="/usr/IBMIHS/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
Apache vulnerability fixes included:
  CVE-2002-1156  CVE-2002-0840  CVE-2003-0132  CVE-2003-0134
  CVE-2003-0189  CVE-2003-0245  CVE-2003-0254  CVE-2003-0253
  CVE-2003-0192  CVE-2003-0789  CVE-2003-0542  CVE-2004-0174
  CVE-2004-0493  CVE-2004-0747  CVE-2004-0786  CVE-2004-0809
  CVE-2004-0942  CVE-2003-0020  CVE-2005-2088  CVE-2005-2728
  CVE-2005-2491  CVE-2005-2970  CVE-2005-3352  CVE-2006-3918
  CVE-2006-3747  CVE-2006-5752  CVE-2007-1863  CVE-2007-3847
  CVE-2007-3304  CVE-2007-6203  CVE-2007-5000  CVE-2007-6388
  CVE-2009-0436  CVE-2008-2364  CVE-2008-2939  CVE-2009-1956
  CVE-2009-0023  CVE-2009-1890  CVE-2009-1891  CVE-2009-2412
  CVE-2009-3094  CVE-2009-3095  CVE-2009-3555  CVE-2010-0434

Tags: tips web-server ihs patch httpd apache maintenance

Blogs

About this blog

Tags