(Related to my earlier post on mod_status.)

Found this useful tidbit today from IBM HTTP Server Performance Tuning:

For IBM HTTP Server 7.0 and later, mod_mpmstats is enabled automatically.

• Check entries like this in the error log to determine how many simultaneous connections were in use at different times of the day:

  [Thu Aug 19 14:01:00 2004] [notice] mpmstats: rdy 712 bsy 312 rd 121 wr 173 ka 0 log 0 dns 0 cls 18
  [Thu Aug 19 14:02:30 2004] [notice] mpmstats: rdy 809 bsy 215 rd 131 wr 44 ka 0 log 0 dns 0 cls 40
  [Thu Aug 19 14:04:01 2004] [notice] mpmstats: rdy 707 bsy 317 rd 193 wr 97 ka 0 log 0 dns 0 cls 27
  [Thu Aug 19 14:05:32 2004] [notice] mpmstats: rdy 731 bsy 293 rd 196 wr 39 ka 0 log 0 dns 0 cls 58
  

This is a quick way to look at your IHS load over time.

The fields logged are described in the table below:

field	description
rdy (ready)	the number of web server threads started and ready to process new client connections
bsy (busy)	the number of web server threads already processing a client connection
rd (reading)	the number of busy web server threads currently reading the request from the client
wr (writing)	the number of busy web server threads that have read the request from the client but are either processing the request (e.g., waiting on a response from WebSphere Application Server) or are writing the response back to the client
ka (keepalive)	the number of busy web server threads that are not processing a request but instead are waiting to see if the client will send another request on the same connection; refer to the KeepAliveTimeout directive to decrease the amount of time that a web server thread remains in this state
log (logging)	the number of busy web server threads that are writing to the access log
dns (dns lookup)	the number of busy web server threads that are performing a dns lookup
cls (closing)	the number of busy web server threads that are waiting for the client to acknowledge that the entire response has been received so that the connection can be closed

 

Tags:  httpd apache mpmstats stats ihs

Using IBM HTTP Server 6.1 and later diagnostic capabilities with WebSphere

This could be useful. Apache mod_status is available in IHS, and when it's enabled it provides information about the individual threads running under the httpd processes.

The information is provided as a simple web page via a URL (that of course can be protected by any of the various Apache mechanisms).

With the default httpd.conf file, only the following changes are required:

1. Uncomment the module loading:
   

   LoadModule status_module modules/mod_status.so

2. Uncomment the Location setup:
   

   <Location /server-status>
       SetHandler server-status
       Order deny,allow
       Deny from all
       Allow from .example.com
   </Location>

3. Change the Allow rule to your own domain or IP, or protect the URL with HTTP Basic Auth, or something else.

Tags:  httpd plugin apache mod_status debugging websphere diagnostic ihs

Tags:  plugin web-server was cluster websphere ihs webserver failover

After attending a webcast on the WebSphere Application Server 6.1 Plug-in, I thought I'd summarize the content I found helpful.

Overview

The plugin is a native module (DLL or .so) that is installed into the Web Server and is offered each HTTP request to determine whether WAS should handle it. It makes this determination based on the contents of the plugin-cfg.xml file which is normally auto-generated by WAS.

I say, "normally", because it can be manually edited, but doesn't need to be for most operating scenarios. Note that many of these customizations can now be configured within the Administrative Console as well, which significantly reduces the need for manual file editing. This configuration is located under Servers -> Web Servers -> <Server Instance> -> Plug-in Properties. (Visiting this location in the Console is also a useful way to to locate the active plugin configuration file.)

Operation Mechanism

The configuration file specifies virtual hosts (IP names/addresses and ports) and URL paths which are serviced by WAS applications, enabling the plugin to determine which requests it can route to WAS and which it must leave for the HTTP server to handle.

The file also contains values governing performance, maintenance, and failover, such as logging configuration and network timeouts.

Finally, it contains the "transport" information used to forward matched requests to the appropriate Web Containers in WAS. That is, the IP name and port combinations of the different Cluster members' Web Container "Transport Chains".

Affinity

"Affinity" is the notion of sending a user to the same Cluster member (application server JVM) for each subsequent request after the initial one, if the application uses HTTP Sessions. This behavior is required by the JEE specification, and in WebSphere is the responsibility of the Web Server plugin. WebSphere maintains this affinity by appending a "Clone ID" or "Partition ID" to the "Session ID", which the plugin then compares against its configuration to correctly route subsequent requests.

For further detail on the Session ID mechanics, see this post.

New users, those without an existing Session, are distributed to Cluster members in either a weighted round-robin fashion or a random fashion. Round-robin is the default, and it will decrement each cluster member's "weight" each time it sends it a new request, until all members are less than zero, then they will all be reset to their starting values.

Tips or Items of Note

1. If the Web Server is a 32-bit program, the plugin must also be 32-bit. Same for 64-bit.
2. The plugin's version must be greater than or equal to the WAS version. However, we know of one case where support said otherwise, so I guess this can be overridden at times.
3. The RefreshInterval attribute of the Config element is the time, in seconds, between cycles where the plugin checks for changes to its configuration and automatically applies them. Its default value is 60.
4. The <Log LogLevel> attribute can be used to increase the plugin's logging. It defaults to "Error" but can have values of "Warn", "Stats", or "Trace".
5. LogLevel "Stats" collects a minimal set of statistics, like data on the number of active connections and on affinity.
6. All plugin logging is written to the same file, even if your Web Server uses multiple processes. Keep this in mind when tracing the log. The format of a log entry is:
   <Date/Time> <Process ID> <Thread ID> - <Message>
7. In the log, the strings "Bld version" and "Bld date" can be used to determine which version of the plugin is installed by comparing against this list.
8. The ConnectTimeout attribute of the Server element controls how many seconds the plugin should wait for an initial connection response from a WAS cluster member. All indications are that this should be very fast. Our current setting is 10 seconds, but the webcast presenter suggested that even 5 seconds should be sufficient.
9. The ServerIOTimeout attribute of the Server element controls how many seconds the plugin should wait for continued IO once a connection has been established. The default is 0, which means wait indefinitely. In other words, don't timeout once communication has begun with a particular server.
10. The plugin should not return a 500 Response Code (Server Error) unless all cluster members fail to respond. (I'm not sure I believe this empirically, but the presenter clearly asserted it.)
11. This connection from the plugin to a particular Web Container transport is apparently maintained as an available "stream" even after a particular request is complete. I don't know how quickly it is shut down or refreshed after use, but it's not immediate. Keep that in mind when, say, counting "netstat -an" connections on a particular port.
12. If all Cluster members have the same round-robin weight, the plugin will factor them down to the value 1.
13. Since the weighted round-robin or random distribution mechanism only applies to new Sessions, it's theoretically possible that Cluster member loads could become unbalanced if long-running Sessions happen to be allocated to one member and shorter ones to another. That is, new requests will be balanced, but that balancing doesn't take into account existing Sessions allocated to the members. This behavior can be changed by setting the IgnoreAffinityRequests attribute of the ServerCluster element to "false", although I don't know what kind of performance implications this change might have.

---

Resources

• Webcast: WebSphere Application Server 6.1 Plug-in: Technical Overview
• Common questions about the web server Plug-in
• WebSphere plug-in concerns on Linux and Unix systems
• Verifying the IBM HTTP Server Plug-in build version shipped with IBM WebSphere Application Server fix packs

Tags:  affinity plugin wlm ihs websphere apache web-server

apachectl -V

For instance

Server version: IBM_HTTP_Server/6.1.0.31 Apache/2.0.47
Server built:   Mar  3 2010 21:55:48
Build level:    IHS60/webIHS1009.03
Server's Module Magic Number: 20020903:4
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/usr/IBMIHS"
 -D SUEXEC_BIN="/usr/IBMIHS/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
Apache vulnerability fixes included:
  CVE-2002-1156  CVE-2002-0840  CVE-2003-0132  CVE-2003-0134
  CVE-2003-0189  CVE-2003-0245  CVE-2003-0254  CVE-2003-0253
  CVE-2003-0192  CVE-2003-0789  CVE-2003-0542  CVE-2004-0174
  CVE-2004-0493  CVE-2004-0747  CVE-2004-0786  CVE-2004-0809
  CVE-2004-0942  CVE-2003-0020  CVE-2005-2088  CVE-2005-2728
  CVE-2005-2491  CVE-2005-2970  CVE-2005-3352  CVE-2006-3918
  CVE-2006-3747  CVE-2006-5752  CVE-2007-1863  CVE-2007-3847
  CVE-2007-3304  CVE-2007-6203  CVE-2007-5000  CVE-2007-6388
  CVE-2009-0436  CVE-2008-2364  CVE-2008-2939  CVE-2009-1956
  CVE-2009-0023  CVE-2009-1890  CVE-2009-1891  CVE-2009-2412
  CVE-2009-3094  CVE-2009-3095  CVE-2009-3555  CVE-2010-0434

Tags:  tips web-server ihs patch httpd apache maintenance