I sometimes need to set a custom JVM Property for one or more Application Servers and dislike the tedium of setting them one-at-a-time through the console.
Here's a Jython script for wsadmin to add a JVM Custom Property to a specified Application Server, or to "all" application servers. It will also replace an existing property with a new value and description. Usage:
Update: the above only obtains the heap size if a custom value has been explicitly configured. From this other post, here's a mechanism which will obtain the maximum heap size otherwise (note there doesn't appear to be an attribute to obtain the minimum heap size):
Web Application Configuration
This is the basic annotation-based JAX-RS coding. See the below "Getting Started" article, Part 1 of the "RESTful Web services" article, and the Developing JAX-RS Web applications section of the WebSphere Feature Pack documentation.
It's a straightforward, portable way to define the URI Paths of your Resources, the HTTP Methods supported by them, the Parameters passed to them, the Content Types supplied by them, and more.
I'm sure this information is in hundreds or thousands of other places, but as I had written it up for an internal team Wiki, I thought I'd repost here as well.
Perl is included by default in AIX, but most of the Modules are not, so the rest of this article deals with installing those.
Some Perl scripts make use of additional Modules, and there are Modules with many different capabilities, so be sure to look for existing Modules if you need to do anything that someone else might have had to do already.
Obtaining a Module
For instance, a script we created uses the Date::Simple module. After finding its page, you click the Download link on the right side to obtain what is actually a source package in .tar.gz format.
Module Installation from Source
These instructions are summarized from somewhere, I thought an earlier version of the generic installation instructions on the CPAN site. However, that site now describes a simpler approach using a cpanminus installation module. I'll have to try that out.
Decompress the file
This can be done as any user.
gunzip -c Date
Build the module
These steps can also be performed as any user.
By default, the Makefile will attempt to create an "XS" version of the Module, which means a version that uses native C code to improve performance. This means the "make" command will require access to a local C compiler.
If you don't have a C compiler or want to create a "pure-Perl" Module to copy around to various systems, you use the "noxs" option. The output of "perl Makefile.PL" will actually tell you to use this option if you see errors during the "make" step. If that occurs, the error will look something like:
cc_r -c -D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias
Install the module
This step needs to be performed as a user that can write to the Perl installation direction. Most likely "root".
Also watch for file permissions. By default this could make the module directory, subdirectories, and files be only readable by root.
It seems that in our current configuration the WebSphere plugin does not reliably stop sending requests to a Cluster member when it's been taken down. Or sometimes it seems to take an inordinate amount of time for it to do so, or sometimes it starts sending requests before a restarting member is fully operational.
A quick workaround is to manually edit the plugin's configuration file on each web server to comment-out the unwanted servers. The plugin will automatically detect and apply the changes within a minute or so and stop routing requests to those servers until you manually uncomment them.
Plugin Configuration File Location
On our AIX systems, the plugin configuration file is in a location like:
(If you're ever uncertain you're looking in the correct location, look for the WebS
Plugin Configuration File Contents
Within that file are many configuration items, most should never be manually edited, but the ones we care about for this purpose are the <PrimaryServers> elements within each <ServerCluster> element.
The ServerCluster's "Name" attribute will tell you which element to edit.
Then the PrimaryServers element in that ServerCluster will list each of the individual <Server> Cluster members which are also defined within ServerCluster.
Removing a specific server from Plugin dispatching
The only change we have to make is to comment-out the specific Server which we want to cease receiving requests. So to disable the TestApp-B server on serverB:
Following up on Part 1, here's an additional tip which I use frequently. That is, when I need to tunnel SSH through one machine to reach others, using a background proxy with SSH key authentication for the initial connection simplifies this 2-hop process.
Automatic proxying with Plink
The PuTTY installation also includes a command-line SSH program called Plink which can be used in a "background" mode. The PuTTY help describes how to use Plink as a local proxy program, creating a background tunnel for the main PuTTY window. This configuration is performed in the Window > Proxy tab:
Specify a Proxy type of local, and the standard SSH Port, 22.
In the Proxy hostname field, you enter a host to which you have direct access and on which you've configured key authentication, then you refer to that host via the %proxyhost variable in the plink command you provide as the local proxy:
\path\to\plink -l %user %proxyhost -nc %host:%port
The %host and %port variables represent the ultimate destination Host Name and Port fields from the main PuTTY Session tab (which, as usual, you can enter as needed or save under separate Sessions for each server).
%user and %proxyhost are from this configuration page.
Note: If your Default Settings PuTTY profile has a username (on the Connection > Data tab) or hostname configured in it, plink will use those automatically. Discovering that wasted a couple of hours for myself and a colleague. On the other hand, if the configured default username matches your username on the proxy server, you can completely omit the -l parameter from the plink command.
Tunneling additional Ports
Furthermore, this technique can be used in conjunction with "normal" SSH tunneling (Connection > SSH > Tunnels) in order to tunnel a localhost port through both hops. For instance, to tunnel the default WebSphere Application Server administration console port:
Then you connect your browser to http
You can similarly 2-hop tunnel X-Windows by enabling X11 Forwarding on PuTTY's Connection > SSH > X11 tab. (You'll need Windows X server like XMing.)
Any additional tips you've found useful? Better ways to accomplish these same tasks? I welcome your comments and suggestions.
During investigation of some intermittent problems in one of our web applications, we observed some JSESSIONID cookie behavior that we couldn't explain, so I performed some investigation into the cookie's mechanics. I'll attempt to summarize in this article what I learned.
JSESSIONID Cookie Format
In a clustered environment, the JSESSIONID cookie is composed of the core Session ID and a few other components. Here's an example:
It's still unclear to me what the different values mean here, but searching our Proxy logs indicates that the vast majority of our Sessions, the Cache ID is 0001. For instance, a search of one day's log around 17:00 indicates the following number of cookies which contained a particular Cache ID:
For a particular Session ID, the Cache ID can definitely change mid-session, without any other changes. In particular, without the Clone/Partition ID changing. This does not indicate a switch to another Cluster member, but I don't know exactly what it does indicate. Based on the relative loads of our various systems, it seems possible that changing Cache IDs only occurs under heavy loads.
A Partition ID is appended to the cookie if memory-to-memory replication in peer-to-peer mode is utilized for Distributed Session management. Otherwise, a Clone ID is appended.
This will match one of the CloneID attributes in the Server elements within the web server's plugin-cfg.xml file. For instance 138888kcd in:
<Server CloneID="138888kcd" ConnectTimeout="10" Exte
If you use memory-to-memory Session replication, your cookies will contain Partition IDs rather than Clone IDs.
Partition IDs are similar in function to Clone IDs, but best I can tell there is no direct way to determine which values correspond to which cluster members. They are internally mapped by the WebSphere HAManager to specific Clone IDs, and that mapping is exchanged with the web server plug-in so that it can maintain Session affinity and find additional cluster members for failover. (The exchange takes place in private headers on each response from WAS back to the plug-in, and those headers are removed before the response is sent back to the client.)
Session Affinity and Failover
The Clone/Partition ID corresponds to whichever cluster member creates the Session, and the plug-in is responsible to send that Session to the same cluster member as long as it is available. From the Scalability Redbook:
Since the Servlet 2.3 specification, as implemented by WebSphere Application Server V5.0 and higher, only a single cluster member may control/access a given Session at a time. After a Session has been created, all following requests need to go to the same application server that created the Session. This Session affinity is provided by the plug-in.
If on a subsequent request the specified cluster member is unavailable, the plug-in will choose another cluster member and attempt to connect to that. If Distributed Sessions are configured, via database persistence or memory-to-memory replication, the Session will be resumed in-progress on that new member. If not, a new Session will be created and the user's progress will be lost.
If a new cluster member is able to resume the existing Session, it will append its own Clone/Partition ID to the existing JSESSIONID cookie. For instance:
Now the plug-in knows that 2 different cluster members could potentially service this Session. If the original member becomes available again, the Session will switch back to it.
Finally, note that according to the System Management Redbook:
WebSphere provides session affinity on a best-effort basis. There are narrow windows where session affinity fails. These windows are:
Referenced articles and Redbooks
Using IBM HTTP Server 6.1 and later diagnostic capabilities with WebSphere
This could be useful. Apache mod_status is available in IHS, and when it's enabled it provides information about the individual threads running under the httpd processes.
The information is provided as a simple web page via a URL (that of course can be protected by any of the various Apache mechanisms).
With the default httpd.conf file, only the following changes are required:
HTML comments are not displayed by a user's browser, but they're available in the source view. JSP comments are removed before the source HTML is returned to the browser.
In many cases where we use comments in our JSPs, we prefer they not appear in the browser-viewable page source. Best case, they're confusing and useless to anybody who looks at them. Worst case, they actually reveal something about our implementation which we'd rather not reveal.
So without further ado:
<!-- HTML Comment -->
I've been trying to connect to a customer's Web Service (from within WebSphere 6.1, but that's a different topic) and having some problems getting the security to work (another different topic), so I've desperately wanted an https-capable traffic monitor/proxy.
After trying the one built into RAD (which successfully passes the https requests, but displays them as encrypted garbage), Apache tcpmon (no SSL), membrane-monitor (seemed to not respond at all and I couldn't tell why), and Wireshark (looked too tedious to get working with SSL), I eventually got soapUI's HTTP Monitor working, although not without some non-obvious setup there either. So I'm going to document that setup here.
Trust the Service Endpoint
First, you'll need a jks TrustStore where you can place the public key of the server which hosts the https-protected Service you're going to call. You can use a tool like the GUI ikeyman, included with WebSphere, or the command-line keytool, included with your JRE, to do this.
Launch the HTTP Monitor
Next, it seemed fairly obvious what to do in soapUI. Right-click on a project and "Launch HTTP Monitor". You're presented with the following dialog:
To do https/SSL, you need to select HTTP Tunnel rather than HTTP Proxy. Set the Port to the local port your client is going to hit, and the endpoint to the remote service URL you're ultimately calling.
Then go to the Security tab:
Initially, all I had was the aforementioned TrustStore configured here in the HTTP tunnel - TrustStore and TrustStore Password fields. This allows the HTTP Monitor to talk SSL to the remote Service.
However, when I started the monitor with those settings, I received a cryptic dialog, "Error starting monitor: C:\Program File
I suspected Windows authority problems or path problems, but nothing seemed to resolve those, so I tried running the soapui.bat script from a command-prompt rather than the soapUI desktop shortcut, and then I saw the following prompts:
Ahah! Google does find something relevant there.
HTTP Monitor Server Certificate
HTTP Monitor needs its own SSL certificate (private key) and KeyStore in order to also talk SSL to your client application, which I should have realized. Then, of course, your client environment will have to import the public portion of that certificate into its TrustStore as well. But the less obvious detail is that this private key must use the alias "jetty" (as soapUI is using the Jetty server under the covers).
So again using keytool (or ikeyman), this time to generate a private/public key certificate pair with the alias "jetty", create the KeyStore that is referenced in the HTTP tunnel - KeyStore, Password, and KeyPassword fields. (Note the two properties listed in the prompts above.) Here's the keytool command I actually used:
jre\bin\keytool -genkeypair -alias jetty -keystore soapui.jks
Now complete the remaining Security fields and click "OK":
You'll have an SSL Tunnel listening on your local port, and you'll be able to see SOAP requests and responses sent through that local URL.
This post is intended to document procedures which can be used to simplify and increase security of remote login procedures to UNIX systems through use of SSH private and public keys.
PuTTY for Windows
Creating an SSH private key
PuTTYgen is used for this step. Use the "Generate" button and follow the instructions.
You can choose RSA or DSA key types, and you can change the key size.
Enter a passphrase and "Save private key" somewhere on your local system. Note: Use a passphrase that you can remember but that is stronger than a normal password. Security people usually suggest using whole sentences or combinations of words/phrases. Later we'll configure another program so that you don't have to type this passphrase very often.
Installing the public key on the remote system
There are various ways to do this, but the PuTTYgen window explains what I've found to be the easiest. That is, copy the text from the Key text area at the top of the window and manually add it to the $HOM
The authorized_keys file probably doesn't yet exist, and the .ssh directory may or may not. (The same directory is where ssh places the known_hosts file that contains the public keys for hosts which you have trusted for ssh connections in the past.)
If either doesn't exist and has to be created, ensure that the permissions are as follows
Each line in authorized_keys can also be configured with further options, including restricting a key's use to specific hosts, for instance. The best documentation I found on those options is at this Free BSD man page.
You can, and probably will want to, install the exact same public key on each system on which you want to use key-based authentication.
Using the key pair to login with puTTY
This is the "manual" approach, which isn't necessary if you follow the next step, but I wanted to document it for completeness. Here we explicitly tell a PuTTY session that we'll be authenticating with the private key file we saved earlier.
If you use this approach, when you connect to the remote system, you'll be prompted to enter your private key's passphrase:
You could argue that's actually worse usability than the user ID/password solution since you'll be typing a much longer passphrase, which is where the next step comes in handy.
Automatic key usage with Pageant
Another of the PuTTY programs, Pageant, can act as an agent providing access to private keys and only requiring you to authenticate once for each key.
Once you've added this key file to Pageant and entered your passphrase there, you can leave Pageant running, and all the PuTTY programs will be able to authenticate with your key without your further involvement. In fact, with Pageant running, any attempts to connect to a host which trusts your key will automatically connect even if you haven't explicitly configured your PuTTY session to use key authentication. (See the default, enabled "Attempt authentication using Pageant" checkbox in the above PuTTY screenshot.)
Furthermore, the Pageant tray icon can actually be used to directly launch any saved PuTTY sessions you've created. Right-click on the tray icon and select "Saved Sessions".
Finally, you can use the command-line to pass to Pageant any private keys you want it to automatically load when it starts. This allows you to create a shortcut icon that will prompt you for the necessary passphrases and then start a copy of Pageant ready to be used for subsequent, key-authenticated SSH sessions:
If you just stop an application to make it unavailable, it will still be automatically started whenever the server where it resides is restarted. To disable an application entirely, but leave it installed, you can change the "autostart" settings under the application's "Target specific application status" (WebSphere 6.1).
Have you ever discovered that you no longer have the EAR file for the specific version of an Enterprise Application which is deployed on one of your systems?
It turns out WebSphere keeps a copy around that can be retrieved and re-deployed elsewhere. For the Network Deployment edition of WAS v6.1, you can find a deployed EAR file in:
(Note the ear name twice, first as a directory, then as a file.)