This article concludes our two-part series describing how IBM's product development team lay the foundation for high performance and scalability growth in IBM Workplace Collaboration Services 2.5. The first article focused on the process adjustments and methodology enhancements we implemented to ensure high performance in the product. This article examines the lessons we learned from a development perspective. This includes best practices we applied to ensure components of Workplace Collaboration Services 2.5 met release goals. These strategies bring obvious benefit to our customers in the form of a highly reliable, quality product. In addition, our experiences can be useful for improving your own development practices, making them more efficient and productive.
As you read this article, pay particular attention to the illustrations. These graphically demonstrate how our development techniques and practices improved and simplified the architecture of the product, significantly streamlining it and reducing its complexity.
As with part 1 of this article series, we assume you're an experienced product manager, architect, application developer, tester, or performance engineer. If you haven't read part 1, we suggest you do so now to help familiarize yourself with some of the terminology and concepts discussed in this article.
In the early stages of Workplace Collaboration Services 2.5 development, we established a process that could identify risks early and accurately. This involved profiling, using tools to execute and analyze our code against a set of previously defined requirements. The profiling process helps developers determine which subroutines (or even just snippets of code) take the longest time to execute, and which subroutines are called most often. This code is then examined to determine whether or not it can be optimized. Profiling is especially useful when you suspect that some part of your code is called very often, and therefore any optimization that can be performed on it might significantly improve the overall performance of the application.
Identifying and addressing schedule, performance, and reliability risks early in the product development cycle enables product teams and organizations to develop workable solutions in time to prevent and/or minimize a potential problem. By also identifying these risks, we were able to control the costs and influence the solution paths more effectively. After we defined this process, we communicated it across the development team, to ensure all participants were on the same page.
One goal usually addressed at the end of a typical product development cycle is ensuring that performance requirements are met. We moved up the performance analysis and engineering phases to be earlier, to identify (and hopefully correct) any performance issues well before product release. This involved allocating resources for analysis and corrections early in the product cycle. It's also important to bear in mind that performance tuning is an iterative process that can span multiple releases. This means that potential performance enhancements and suggestions identified in one release may end up being fully implemented in a subsequent release.
Identifying risks and issues early in the product cycle, especially with accelerated product deliverables, also means overlap of phases between product releases. As one product release is in its final phases of systems and performance testing, the next product release cycle should be starting where requirements are identified, and the process of risk identification can also start. This overlap increases the need to identify risks earlier.
Using "functional pushdown" to identify code flow
"Functional pushdown" is a process for product design and implementation with the overall goal of improving performance. It identifies key software activities in a program and determines whether or not they can be "pushed down" so that they are initiated and executed in the lower tiers of the software application architecture (see the following section). Functional pushdown is especially important during the early design phase, to help define what the code is doing and where we can look for and avoid duplication. It also helps ensure successful movement of key logic components in the software execution architecture.
To help you better understand how the suggestions and best practices discussed in this article work, let's take a quick look at IBM Workplace architecture. IBM Workplace products such as Workplace Collaboration Services are built upon a "four-tier" architecture, supported by WebSphere Application Server. This architecture allows application functionality to be distributed across different systems. The four tiers are:
- Presentation (user) tier. This consists of client access components, including the HTTP server and WebSphere Portal server. This tier presents the user interface, allowing users to interact with the other three tiers.
- Workplace tier. This consists of the WebSphere Portal server. This tier handles the user's actions, tracks session details, and coordinates and secures interactions between the presentation tier and business logic tier (see the following).
- Business logic (service) tier. This consists of processes running on remote servers, including WebSphere Application Server and IBM Workplace products. This tier (which is also referred to as the application logic tier) manages the business logic of the application. This is where most of the actual processing activity occurs. Multiple client components can simultaneously access the processes in this tier, so this tier must manage its own transactions.
- Data/resource tier. This consists of a set of relational databases and resource managers. For security reasons, this tier is protected from direct access by clients. All interactions with this tier must occur through the business logic tier.
In the following sections, we describe ways in which we took advantage of this architecture to help improve the performance of Workplace Collaboration Services 2.5.
To begin the Workplace Collaboration Services design effort, we critically examined and defined a critical piece of its logic flow, a basic application Web transaction. We considered all the pieces and process this entailed, and charted this graphically (see figure 1):
Figure 1. Application Web transaction logic flow
As you can see, our original view of the Web transaction is rather complex and involved. To help simplify this and make it more efficient, we applied functional pushdown techniques to several key components of this process. The remainder of this article describes how we did this. In each section, we discuss a single best practice, and then show graphically how it impacted the flow shown in figure 1. By the end of this article, you may be surprised by our results!
As part of Workplace Collaboration Services application requirements, data is retrieved from multiple fields and/or tables. Typically, the program goes to the database and retrieves data, often in the form of a key value. Then an additional call is made to the database by another section of the code to look up data contents based on that key value. Next, the code collates the data returned from these different methods to satisfy a client request. This process is referred to as a "data join," and it can be inefficient because the services offered by the database software itself may not be fully leveraged.
To avoid data joins, we decided to "push down" this processing and let the services provided in the database/resource tier deliver what they are designed to do, and return data from one or more record sources. This allowed us to execute a single call to collect the data. The join request was carried out on the database server to combine data from the tables. The database optimizer determines the most optimal method for joining, another reason why this responsibility should be handled at the database/resource tier. There are a variety of different types of join strategies that are likely offered, and at this tier it can be determined which should be used and execute on it. No duplicate logic needed to be developed (and this would probably not measure up to the functionality provided by the database services).
So taking the code flow shown in figure 1, we removed the "data joins" and pushed down this activity so it would be executed in the database/resource tier (see figure 2):
Figure 2. Removing data joins
This represented our first step towards streamlining our code path's execution.
Most applications devote part of their user interfaces to displaying information corresponding to a collection of elements. (By "collection," we mean a set of objects that satisfy given conditions.) To retrieve these objects, the application executes some retrieval logic. There are different strategies for this retrieval logic. One strategy involves calling the same method of components in a loop with elements of a collection. We've observed that it is often better to pass the collection to a single call, rather than passing its elements one at a time. A good reference point is the cost of interactions with the database server.
This "loop optimization" approach is a positive conservation strategy for the database server, where one call is made for many elements, which it is optimized to handle. Unless the entire collection is passed in on the call, it is very hard for the called component to optimize its approach.
We needed to identify methods that are called from within a loop, especially with a large number of iterations (one indicator is the number of records in the database). Any improvements made to the method calls will likely be effective. For example, in one program we observed a new DateFormat object being instantiated, and then immediately orphaned. We pulled the DateFormat object out of the loop and reused the same object in the loop. So if you're cycling through data returned from a loop, review how the call can be made once and only once.
We also needed to define what type of calls to make (coarse-grained or fine-grained). We decided to use coarse-grained calls for remote operations and fine-grained calls for local operations. This reduced incidental overhead.
Based on our experiences, we recommend passing the whole collection in one call, so it can be optimized in one place. Don't pass the data back in a loop. Instead, design for a collection approach. This allowed us to further simplify our code logic flow, where multiple calls are made in a loop to retrieve needed data. With code optimizations and redesign, we reduced the loops to a single call, avoiding multiple nested segments of code (see figure 3):
Figure 3. Loop optimization
Another thing we noticed in our code was that the same information was referenced and defined in different ways by the individual components. Each component placed its own definition requirements upon the data content. For example, one lower-level component required a different key, which then forced the calling components to translate to that specification. (This was frequently the case with access control.) This extra layer of definitions can be avoided with proper design and planning.
As part of defining an interface utilized by another part of the application, we needed to ensure that the calling interface could take the key of the caller and use it. This eliminates the need to repeatedly translate keys. Therefore we could remove an extraneous code path, including data content translation (see figure 4).
Figure 4. Key mapping
Reducing access control checks
Software developers often "overdevelop" the executing logic in their area of responsibility, doing more than they really have to. This is usually due to:
- Making sure the areas under their control are robust and complete.
- Not communicating enough with the other components (either by review of what they support, or making sufficient demands in the form of requirements) about what they will accomplish.
- Appreciating the bigger picture approach -- thinking about where services should be located in the hierarchy.
One example we encountered while developing Workplace Collaboration Services was redundancy in the areas of access control checks and security. Security, of course, is a serious matter. But the same software repeating the same validation and screening logic is costly and wasteful in terms of time. Logic for access control checking can be complex, leading to inefficiencies.
One thing we noticed was that developers had included duplicate access control checks at different tiers. To avoid this, we recommended that access control checks be executed once at the appropriate software level. Our goal was to keep the logic and interface as simple as possible, by checking only when we had to and avoiding redundant checks.
Although this may appear obvious, it's important to know which software tiers have responsibility for what service offerings. A theme throughout this article is the pushdown concept, where as much activity as reasonably possible should be executed at the lowest possible tier. This especially applies to access control checks. So, design in a way that ensures the lowest tiers handle these responsibilities. By avoiding additional security checks, we gained additional performance time by eliminating duplicate work. We also reduced the number of calls, as shown in Figure 5:
Figure 5. Reducing access control checks
Bear in mind that Workplace Collaboration Services is a set of "federated" components that can be combined in any number of ways. As such, it may not always be simple for each individual component to understand the context in which it is operating. Infrastructure support components (such as access control and persistence services) need to take this fact into account. In complex systems where common services are provided for, common services must understand how they are going to be used by applications.
"Struts" comprise a base controller in the application architecture. This provides building blocks and infrastructure components to build a Web-based application. It also provides other important items, including a consistent mechanism for passing dynamic data between the struts action and JSP. The controller servlet gives the application a flow control mechanism. It acts as the central "traffic cop" for requests, insuring that the requests get properly routed down the correct path and handled by the right component.
In our efforts to optimize performance and scalability, we found it important to have a single method invocation into the business logic (service) tier. A great deal of analysis and optimization can be better addressed by working with a single method. Otherwise, we might end up with multiple different calls to business methods. This translates to multiple lower level calls to the database/resource tier.
Figure 6 illustrates the results of rearchitecting our code so the struts action makes a single service call, rather than branch across several:
Figure 6. Struts action calling a single service method
The important point to note here is the reduction of layers between the JSPs and/or the controller, and the application's business logic (service) tier. Our goal is to push the logic down to the lowest possible tier. Object-oriented APIs that strictly express application semantics can be a very costly approach in terms of performance. Well-designed coarse-grained APIs at lower tiers are usually more efficient.
Properly implemented caching can help reduce the runtime and load on your server(s). It will help avoid expensive I/O operations (for instance, database calls, LDAP calls, and file reads). Based on this experience, we recommend the following:
- Cache at the lowest level (right above I/O). Do not cache at high levels up the call stack; request that the called component implement the cache.
- Cache compact objects (not large memory inefficient structures).
- Prevent the cache from growing on demand; make sure there are boundaries.
- Consider off-loading the cache to disk.
- When returning cached objects, try to ensure they are immutable. This way they can be shared by multiple threads.
- If the object must be copied (to avoid concurrent access), do this via a clone (not a serialization).
One last point to keep in mind is that planning and designing for use of caches needs to be done at the beginning of the development project. Identifying these key optimization opportunities early in the cycle, where much of development work will take place around it, is the most time and resource efficient approach. Beyond cache design, you also need to plan for cache management -- services should be provided for validation of data, expiration of stale data, and identification and management of infrequently utilized data (analysis of size, contents, and usage patterns). These guidelines helped influence our cache design (both internally and externally tunable), and should also be considered in your application design. When caches are implemented, you need to monitor and verify them for effective usage patterns within the current product release and afterwards. Usage patterns may change as the application changes.
Debug code behind logging check
It is extremely helpful to the development/test team when the debug code is implemented to report out on response codes, variable amounts, and locations reached within the code. This same logic can sometimes also be utilized by the support and consulting teams when the product is released. However, we need to be careful to avoid situations where the division between pre-release code is not roped-off or removed before product release. This could produce extraneous debug messages, causing additional overhead for processing (especially if verbose in nature) and log messages displayed that may not be informative to users.
A logging mechanism should be put in place that lets you switch off logging in the production environment to reduce logging overhead. This makes it simpler and easier to move from the development environment to the test and production environments. The Boolean variable will control the logging, and you can instruct the entire component to turn logging on or off through the value specification or one variable. For instance:
if (LOG_SWITCH_ENABLED)
{
log_call( Log this message );
}
|
Ensure the component returns no more than the information required Perhaps the most important thing we learned was to ensure that all application components return no more than the information required for the needs of the particular use case. This results in:
- Not doing more work than you know you have to.
- Only returning information required. Anticipating future needs may seem like a good idea, but it can be time costly and wasteful.
- Extra resource utilization.
To achieve this goal, we considered the needs of the end-users and how they related to our processing. We then frequently monitored and reviewed the product, to make sure the data retrieved is actually used (and not discarded). As we discussed earlier, we try to accomplish this within one call whenever possible. This further simplified our code logic flow, as shown in figure 7:
Figure 7. Optimized information return
One point to consider when designing database architectures for storage and retrieval of the data is the associated indexes that are created to access the data uniquely and as quickly as possible. The index allows the database software to find elements in a table without scanning the table repeatedly. This saves a great amount of system resources -- CPU in particular, and also memory and network bandwidth utilization.
The advantage of using indexes within your application logic is that no direct access of the data is required. Data can be read via the index access, rather than a scan through the data stored in the tables. Note that if all data required by a database select statement is in the index, there is no reason for the database to access the table data (this is called index-only access). This can produce queries that are an order of magnitude faster than those that access the table, and simply underscores the point that only the data that is absolutely required for the transaction should be retrieved from the database.
The final result of implementing all the optimizations and best practices discussed in the preceding sections is illustrated by figure 8:
Figure 8. Streamlined application Web transaction logic flow
Compare this to figure 1, and you can see how our functional pushdown approach produced a reduction in calls and codepaths executed while still accomplishing the same goal. Making this happen, of course, required significant cooperation and cohesion across the development team.
In conclusion, we learned the resource and cost/benefits of:
- Mitigating risk by defining it earlier in the product development cycle.
- Fixing key issues as early in the cycle as possible, resulting in more test and analysis coverage.
- Making sure that each component participating in the use case should be called once and only once.
- Ensuring that each component return only the data needed by the calling component.
- Not having lower level components ask for extra data.
Performance begins at the application, but eventually affects the entire customer production environment. As you consider good J2EE coding practices, think about the development tips we've discussed in this article. You can also consult the IBM Workplace Collaboration Services documentation and WebSphere Portal documentation for additional information. These should provide a lot of good suggestions and advice for developing Web applications for IBM Workplace.
- Read the first part of this article series, "Developing IBM Workplace Collaboration Services 2.5: Part 1: Building IBM Workplace Collaboration Services for performance," to learn about the process adjustments and methodology enhancements we implemented to ensure high performance in IBM Workplace Collaboration Services 2.5.
- To learn more about IBM Workplace Collaboration Services, see its product page.
- Visit the IBM Rational products page to read about IBM Rational profiling and performance tools.
- Read the developerWorks: Lotus article, "New features in IBM Workplace Collaboration Services 2.5," to learn about the new features provided in IBM Workplace Collaboration Services 2.5, the latest addition to the IBM Workplace product family.
- For a complete list of supported platforms for Workplace Collaboration Services, see the IBM Workplace Collaboration Services documentation.
- Get involved in the developerWorks community by participating in
developerWorks blogs.
Carol Zimmet, Lead Technical Analyst on IBM's Workplace, Portal, and Collaboration Performance Team team, is striking a balance between managing multiple projects and developing performance analysis skills on the new Workplace platform. She continues to evangelize on behalf of performance, both promoting product accomplishments as well as advocating on behalf of our customer requirements. Previously, Carol served on the development and test teams for many different Lotus products.
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
