![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Editor's note: Know a lot about this topic? Want to share your expertise? Participate in the IBM Lotus software wiki program today.
| WebSphere Portal wiki |
|---|
“To measure is to know. If you cannot measure it, you cannot improve it.” -- Lord Kelvin
At the core of WebSphere Portal effectiveness is the ability to gauge that the portal is fulfilling its objectives and to measure that planned improvements on the portal have their desired impact. In practice, however, measuring portal effectiveness is a gray area. Often portal sites lack the the tools, the techniques, and the empirical metrics needed for meaningful measurement.
This article describes the available tools and technologies that you can use to monitor portal usage, including WebSphere Portal site analytics, IBM Lotus® Web Content Management rendering portlet analytics, WebSphere Portal personalization logging, and various open source and commercial offerings.
This article also compares these tools and technologies, providing usage guidelines in various scenarios and the design patterns and best practices for tracking portal usage.
The article also address how to interpret portal usage statistics to build quantitative portal effectiveness scores through a number of techniques and examples. These techniques enable mappings across the usage statistics, the prioritized portal objectives, and the nature of portal usage.
Finally, it wraps up by illustrating how to analyze the WebSphere Portal effectiveness scores for determining the next steps that you can take to improve effectiveness.
To get the most from this article, you should have at least have a basic understanding of the concepts and the use of WebSphere Portal and, preferably, some real-life experience with how the WebSphere Portal affects various constituencies within the organization.
This article can be helpful to anyone working in WebSphere Portal management, particularly with respect to administrative and organizational aspects.
Measuring effectiveness: An overview
A portal is often perceived to provide significant benefits, but frequently these benefits are thought to be intangible. This article demonstrates how to assess, quantify, and assign impact to those benefits.
The measurement of portal effectiveness can be viewed conceptually as the difference between having the portal available and not having a portal available. This measurement incorporates the impact that WebSphere Portal can have on multiple communities and its overall benefits to the organization.
Table 1 illustrates the typical portal constituencies and their potential benefits.
Table 1. Portal constituencies and their impact
| Portal constituencies | Overall benefits |
|---|---|
| Users |
|
| Business stakeholders |
|
| IT stakeholders |
|
| Help desk team |
|
The process for measuring WebSphere Portal effectiveness consists of: these tasks:
- Monitoring usage and satisfaction
- Determining key impact criteria and priorities to construct metrics
- Building the WebSphere Portal value model
Figure 1 illustrates this process.
Figure 1. Strategy for measuring portal effectiveness
Tracking portal usage and satisfaction
After the portal is live, there are important questions to address as to whether the portal is meeting its expectations and how it's being used:
- How satisfied are the users with the portal?
- What are the most frequent user complaints about the portal? Are the complaints growing, steady, or declining?
- How many users are accessing the portal and when is the peak usage? Is the usage growing, steady, or declining?
- What are the profiles of the users who access the portal most frequently? How much time do the users spend on the Dashboard page? Can user activity be correlated with their profiles or with any known external events?
- What is the average time for a content item to get stale? What is the average number of search queries for users to find what they're looking for?
- What WebSphere Portal applications and functions do users access the most?
There is no single means to answer these questions; rather, they can be addressed through a composite of techniques. Table 2 describes some of the available techniques.
Table 2. Techniques to track WebSphere Portal usage and satisfaction
| Technique | Description | When to use |
|---|---|---|
| Server-side site analytics (based on site analytics file analysis) |
| For portal sites requiring visibility into the user behaviors but not requiring any real-time feedback. |
| Real-time active analytics |
| For e-commerce sites supporting user campaigns. |
| Feedback database |
| For sites actively supporting user personalization. |
| User satisfaction surveys |
| All sites. |
| User complaint tracking |
| Sites supporting an active user help desk. |
| Observations on live portal usage |
| Mission-critical sites. |
Figure 2 shows a snippet of the WebSphere Portal site analytics log file for server-side analytics.
Figure 2. Log file snippet
Refer to the developerWorks® article, “IBM WebSphere Developer Technical Journal: 'Using portal analytics with open-source reporting tools',” for open source-tool approaches to processing the log file to generate usage reports. These reporting tools provide a low-cost means to gain insights into both the overall portal web-traffic characteristics and the portions of the portal that are being used heavily.
There are also more advanced commercial products available to perform real-time monitoring of WebSphere Portal and to deliver on a dashboard the detailed analysis of portal activity. Some examples of vendors that offer such products are Webtrends and Coremetrics, as discussed in the paper, “IBM WebSphere Portal version 6.1.5 Tracking and Reporting Using Webtrends.”
Five-step process for measuring effectiveness
WebSphere Portal usage and satisfaction data can be correlated against business goals to determine effectiveness. This correlation can be viewed as a five-step process:
- Map business goals to their supporting WebSphere Portal capabilities.
- Develop WebSphere Portal capabilities into design artifacts that can lead to specific portal activities to be monitored.
- Assign impact formulas to the monitored WebSphere Portal activities based on the traffic and associated business value.
- Construct impact ratios based on anticipated impact and measured impact.
- Develop an overall effectiveness ranking based on multiple criteria, including impact ratios, user satisfaction surveys, problem reports, and usability studies.
Figure 3 illustrates these five steps.
Figure 3. Model for measuring portal effectiveness
The first step in determining WebSphere Portal effectiveness is to identify the business goals, with the priorities of those goals, and to map them to the applicable portal technologies.
Let's demonstrate this concept by using the example of the hypothetical Financial Representatives Insurance portal that delivers information and services to independent field representatives.
Table 3 illustrates an example of mapping the business goals for the insurance portal to WebSphere Portal server capabilities.
Table 3. Mapping business goals to WebSphere Portal server capabilities
| Sample business goals | Business goal priority | Portal capabilities | Sample applicable WebSphere Portal technology |
|---|---|---|---|
| Increase revenue for each representative | High | Enable delivery of full suite of customer relationship management applications | Application integration Custom portlets |
| Enhance the effectiveness of information and services | High | Deliver targeted content | Content personalization Content management |
| Reduce support costs | Medium | Provide self-service capabilities and help functions | Application integration, custom portlets, and content delivery |
| Increase the appeal against competitive offerings | Medium | Provide rich interaction capabilities | Asynchronous Java and XML (Ajax)-based user interface and semantic tags Search |
| Improve communications and interactions with representatives | Medium | Deliver blogs and forums | Collaboration suite |
Step 2. Identify the WebSphere Portal activities to be measured
The next step is to map the requirements and portal capabilities derived from Step 1 to facets of portal design so that you can identify the WebSphere Portal activities to be monitored.
Table 4 illustrates an example of mapping aspects of portal design to the monitored WebSphere Portal activities with the sample insurance portal.
Table 4. Identify portal activities to be monitored
| Portal capability | Portal design | Sample monitored WebSphere Portal activity |
|---|---|---|
| Customer relationship management (CRM) application delivery | Deliver role-based access to the full suite of eService applications | Launch of eMyClient application Launch of Client Insight portlet |
| Targeted content | Deliver personalized content for products and services | Portfolio and Annuities content access |
| Self-service and help | Deliver user profile application and help content | Launch of Profile application Help content access |
| Rich interactions | Web 2.0 theme and search engine | Search engine hits (user satisfaction scores in portal surveys) |
| Blogs and forums |
| Number of blogs Forum posts |
In this step, the activities monitored in Step 2 are incorporated into a set of impact formulas. You can do this task by associating a proportional relevance derived from the value of the business goal associated with the monitored WebSphere Portal activity and combining it with the portal traffic figures for the activity.
Table 5 outlines this step for the sample insurance portal.
Table 5. Building WebSphere Portal impact formulas
| Monitored activity | Business goal and value | Proportional relevance (adjusted across scale of 100) | Sample impact formulas |
|---|---|---|---|
| eMyClient application launch | Increase revenue (High, score 3) | 14 | <Average daily eMyClient launches> |
| Client Insight portlet launch | Increase revenue (High, score 3) | 14 | <Average daily Insight launches > |
| Content access - Portfolio | Enhance information delivery (High, score 3) | 13 | <Average daily Portfolio access > |
| Content access - Annuities | Enhance information delivery (High, score 3) | 13 | <Average daily Annuities access > |
| User-profile application access | Reduce support cost (Medium, score 2) | 9 | <Average daily Profile access > |
| Search engine hits | Increase appeal (Medium, score 2) | 10 | <Average daily search queries> |
| Content access - Help | Reduce support cost (Medium, score 2) | 9 | <Average daily help access > |
| Blog posts | Improve communications (Medium, score 2) | 9 | <Average daily blog access> |
| Forum posts | Improve communications (Medium, score 2) | 9 | <Average daily forum posts> |
Step 4. Construct WebSphere Portal activity-impact ratios
We construct the quantitative scores to assess the impact of the activities by using the formulas in table 5. Both the anticipated impact score, based on the original expectations of the portal, and the measured impact score are constructed, so that you can build an impact ratio of the measured as opposed to the anticipated.
Table 6 outlines the process of generating impact ratios with the sample insurance portal. NOTE: The impact formulas and porportional relevance are defined in table 5.
NOTE: The impact ratios of the monitored activities are averaged to construct the overall portal impact ratio.
Table 6. Constructing impact scores and ratios for the portal
| Monitored activity | Anticipated impact | Measured impact | Impact ratio (Measured/anticipated) * (proportional relevance / 100) |
|---|---|---|---|
| eMyClient application launch | 12000 | 4000 | 5% |
| Client Insight Portlet launch | 12000 | 10000 | 12% |
| Content access - Portfolio | 36000 | 12000 | 5% |
| Content access - Annuities | 36000 | 15000 | 6% |
| User-profile application access | 4000 | 3000 | 9% |
| Search engine hits | 24000 | 15000 | 6% |
| Content access - Help | 2000 | 1000 | 6% |
| Blog posts | 1000 | 600 | 5% |
| Forum posts | 1000 | 650 | 6% |
| Overall portal-activity impact ratio | N/A | N/A | 60% |
Step 5. Build WebSphere Portal effectiveness ranking
Measuring the impact of the activities as described previously is only one of several dimensions to assessing WebSphere effectiveness. The portal activity impact ratios can be combined with user satisfaction measurements, problem ticket reports, and usability studies to build an overall effectiveness ranking (see table 7).
Table 7. Constructing the effectiveness ranking with the sample insurance portal
| Ranking | Goal | Measured | Impact ratio [(measured / goals) * 100] |
|---|---|---|---|
| Activity impact | 100 | 60 (mean impact score from table 6) | 60 |
| User satisfaction surveys | High satisfaction (Ranking: 3) | Medium-high satisfaction (Ranking: 2) | 66 |
| Problem reports | Average daily reports less than 20 | Average daily reports around 40 | 50 |
| Usability studies | Very usable (Ranking: 3) | Medium-low usability (Ranking: 1.5) | 50 |
| Overall portal effectiveness ratio | N/A | N/A | 57 |
The value of 57 for the portal effectiveness ratio is below expectations, leading to a “B-” ranking for our sample portal; a 90 to 100 percent portal-effectiveness-ratio range in our example would earn an “A” ranking.
Now let's analyze how to improve the effectiveness of the sample Insurance portal to at least a “B” ranking.
Tuning WebSphere Portal effectiveness
The data in tables 1–7 provide some key metrics to enable you to improve WebSphere Portal effectiveness. For example, table 6 shows that the eMyClient application launch from the portal is one-third of its anticipated value; in addition, there are significant shortfalls in the levels of anticipated content access to Portfolio and Annuities products.
Some new strategies to improving effectiveness can include the following:
- Improving access to the eMyClient application. Place the eMyClient application launch icon on the portal home page for ease of access, and add new content highlighting the availability of eMyClient.
- Enhancing the quality of content for Portfolio and Annuities. Increase the available new content for Portfolio and Annuities products by speeding up the content approval process.
- Enhancing the portal usability. Refine the portal navigation to improve on the portal usability experience.
These two strategies are just a start; the objectives of measuring WebSphere Portal effectiveness are to build a continuous refinement and measurement cycle.
It is possible to systematically measure WebSphere Portal effectiveness by correlating business goals with the monitored activities on the portal. This approach enables the compiling of metrics that can help guide you toward specific actions to improve effectiveness, which is an integral part of ensuring that the portal is aligned for success.
- Participate in the discussion forum.
-
Read the WebSphere Portal wiki article, “Site analytics and optimization."
-
Read the developerWorks white paper, "Site analytic support in IBM WebSphere Portal 6.1.5."
-
Refer to the IBM WebSphere Portal Version 6.1.5 Information Center.
-
Refer to the IBM developerWorks WebSphere Portal zone.
-
Contribute to the WebSphere Portal Family wiki.
Pradeep Behl is a Senior Consulting IT Architect with IBM Software Services for Lotus who assists customers with design, evaluation, and best practices of architectures based on WebSphere Portal. He has presented in the WebSphere Portal Technical Conference. His current areas of technical interest include Portal Governance and Virtual Portal best practices. You can reach him at pbehl@us.ibm.com.
Table of contents
Next steps from IBM
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
