Now that you're here, I confess to a little bait and switch. Rather than discuss things that people are afraid to ask regarding WebSphere Application Server (quite frankly, there isn't much you're afraid to ask about), I thought I would take this opportunity instead to address a few questions that I seem to get asked over and over again.
With apologies to Douglas Adams, I would prefer to answer "42" or almost anything other than "it depends" when approached with these questions, but as you will soon see, often the most accurate response really is: it depends (on your application, on your infrastructure, on your performance or scalability requirements, and so on). That said, however, I really will try to provide some guidance to help you determine the best answer for your specific situation, as well as provide some definitive answers to a couple of common questions. So, let's get started.
A: It's been just over five years since I was part of the team that co-authored the IBM Redbook WebSphere Scalability: WLM and Clustering Using WebSphere Application Server Advanced Edition, and one of my contributions was a discussion entitled "How Many Clones," in which I attempted to address the question of how many application servers do I need to run my application? The CPU question is a related one that I encounter more often now, and this is one of those classic questions for which the answer is, in fact, it depends, so let's look at why it depends and the factors that it depends on.
Modern operating systems generally do an excellent job of process scheduling, which minimizes context switching. Context switching occurs when the operating system scheduler replaces one process operating on a CPU with another process and is the result of a variety of factors, such as job (or process) priority, waiting for other resources (like I/O), if the process is using up all of its allocated CPU cycles (time slice), and so on. However, "excellent" in this context is not "perfect," since each time a process is context-switched it stops operating -- which then results in throughput delays (and performance degradation). If we really wanted to eliminate the possibility of context switching, we would need to have more CPUs on a system than processes running on that system. Now, this isn't exactly practical, since few organizations can afford to put that many CPUs on one system, but moreover it is unlikely to even be necessary, since most processes don't require continuous CPU access. Therefore, we should look at this instead in terms of most important processes, which in this context are the JVMs for the application servers running on a system.
As a starting point, I plan on having at least one CPU per application server JVM; that way I have likely minimized the number of times that a context switch will occur -- at least as far as using up a time slice is concerned (although, as mentioned, there are other factors that can result in a context switch). Unless you run all your servers at 100% CPU, more than likely there are CPU cycles available as application requests arrive at an application server, which in turn are translated into requests for operating system resources. Therefore, we can probably run more application servers than CPUs.
Arriving at the precise number that you can run in your environment, however, brings us back to it depends. This is because that number will in fact depend on the load, application, throughput, and response time requirements, and so on, and the only way to determine a precise number is to run tests in your environment.
It is certainly reasonable to expect that you can run a higher multiple of application server instances per CPU in a development environment (where the load is likely only a single user per application server) than in a production environment. Beyond that, it is difficult to be more specific, though I should add that adequate physical memory for all the application server processes is an equally important factor. A good rule of thumb is that the sum of all the WebSphere Application Server JVM processes should not exceed 80% of the otherwise unused physical memory on the server. When calculating the largest number that this figure can grow to, you must not only consider the maximum heap size, but also the process size of the Java™ bytecode interpreter (which is reflected in the OS process table) over and above the maximum heap size. The bytecode interpreter adds about 65MB to the process table (over the maximum heap size for a 128MB heap), and increases in size as does the maximum heap size does.
Having addressed the minimum number of CPUs, you might then ask, What about the maximum? You won't be surprised to learn that the answer here also it depends. A very efficient and well written application can use multiple CPUs with just a single application server process. In fact, the WebSphere Performance Lab has fully utilized 12 CPUs (and in some cases more) when running tests involving the Trade performance benchmark. While it is probably not reasonable to expect most applications to be this scalable, most well written applications should be able to use more than one CPU per application server (in fact, only using one CPU is often the sign of an application bottleneck). In any event, the guidance that I penned more than five years ago in WebSphere Scalability on determining the optimal number of clones (or, application server instances) still applies:
|In general one should tune a single instance of an application server for throughput and performance, then incrementally add clones testing performance and throughput as each clone is added. By proceeding in this manner one can determine what number of clones provide the optimal throughput and performance for their environment. In general once CPU utilization reaches 75% little, if any, improvement in throughput will be realized by adding additional clones.|
And, as you can see, the bottom line answer to this question is, it depends.
A: This question actually has a definitive answer, so I can't get away with it depends here.
When using Local OS for any operating system as the user registry, WebSphere Application Server makes native OS calls for authentication to the underlying user registry. In the case of Windows, "Act as part of the operating system" is required to make the native OS calls specific to Windows. (To digress slightly, to use Local OS on UNIX®, you would have to run WebSphere Application Server as root because root is the only user ID that can query the registry for all user IDs on the system.) I believe one reason why operating systems restrict access to password checking APIs is because any user could call them from any user level program and potentially use a dictionary attack to try to learn a user's password. Forcing them to first gain root (or act as part of the OS) access limits the risk. Of course it's not perfect, but every little bit helps. Don't forget, this is not a "WebSphere requirement." This is simply a requirement of the underlying operating system.
It is due to operating system specifics of this nature that lead to the general recommendation to use LDAP for the security registry. This enables you to avoid the differences that may exist between operating systems where different operating systems are being used for development, test, and production. An additional advantage of using LDAP is that you need not use the LDAP administrative user ID and password for WebSphere Application Server to authenticate with the LDAP server (via an LDAP bind); any valid user ID and password in the LDAP directory will suffice for this purpose. Another alternative is to use a custom user registry employing a file with the user ID and passwords for development environments. As a result, both LDAP and custom user registries are good alternatives for negating the requirement to grant OS privileges over and above that of an unprivileged user.
A: Performance does not differ significantly between database persistence and memory-to-memory replication. This is because 95% of the cost of replicating or persisting sessions is incurred in the serialization/deserialization of the session object -- which must occur regardless of how the session is distributed. Also, as the size of the session object increases, performance degrades -- again, about equally for both session distribution options.
In IBM WebSphere: Deployment and Advanced Configuration, I wrote:
|Instead, the decision will be based partially on how the two technologies differ:
To this, I would like to add another thought. With memory-to-memory replication, the amount of session information you can store is bounded by the JVM heap size of your application server(s). Even with the advent of 64-bit JVM support in WebSphere Application Server V6.01, the maximum application server heap size is going to be significantly smaller than the amount of disk space you have available on a database server that is serving as a session store. Therefore, I am still of the opinion that database persistence remains the best option, although I know that in many organizations it is more expedient to use memory-to-memory replication to avoid conflicts over roles and responsibilities between system and database administrators.
Finally, I was also going to discuss another common question regarding the optimal size of a session object, but one of my colleagues beat me to it last month in this very column. If this is something you'd be inclined to ask about, then check out what Bill Hines had to say.
These are my ramblings. I hope they have provided some insight into how you can go about determining the best answers to these questions for your environment, if not some direct answers to a couple of common queries.
- IBM WebSphere Deployment and Advanced Configuration, by Roland Barcia, Bill Hines, Tom Alcott, Keys Botzum
- WebSphere Scalability: WLM and Clustering Using WebSphere Application Server Advanced Edition
- Get involved in the developerWorks community, participate in developerWorks blogs.
- Browse for books on these and other technical topics.
Tom Alcott is consulting IT specialist for IBM in the United States. He has been a member of the Worldwide WebSphere Technical Sales Support team since its inception in 1998. In this role, he spends most of his time trying to stay one page ahead of customers in the manual. Before he started working with WebSphere, he was a systems engineer for IBM's Transarc Lab supporting TXSeries. His background includes over 20 years of application design and development on both mainframe-based and distributed systems. He has written and presented extensively on a number of WebSphere run time issues.