Measuring IBM WebSphere Portal effectiveness

This article explains the tools and technologies available to monitor IBM® WebSphere® Portal V6.1 or later usage, including WebSphere Portal site analytics and various open source and commercial offerings. It compares these tools and technologies, providing usage guidelines in various scenarios with the design patterns and best practices on tracking portal usage.

Share:

Pradeep Behl (pbehl@us.ibm.com), Senior Consulting IT Architect, WSO2 Inc

Pradeep Behl is a Senior Consulting IT Architect with IBM Software Services for Lotus who assists customers with design, evaluation, and best practices of architectures based on WebSphere Portal. He has presented in the WebSphere Portal Technical Conference. His current areas of technical interest include Portal Governance and Virtual Portal best practices. You can reach him at pbehl@us.ibm.com.



13 July 2010 (First published 06 July 2010)

Also available in Chinese Portuguese Spanish

Editor's note: Know a lot about this topic? Want to share your expertise? Participate in the IBM Lotus software wiki program today.

Introduction

“To measure is to know. If you cannot measure it, you cannot improve it.” -- Lord Kelvin

At the core of WebSphere Portal effectiveness is the ability to gauge that the portal is fulfilling its objectives and to measure that planned improvements on the portal have their desired impact. In practice, however, measuring portal effectiveness is a gray area. Often portal sites lack the the tools, the techniques, and the empirical metrics needed for meaningful measurement.

This article describes the available tools and technologies that you can use to monitor portal usage, including WebSphere Portal site analytics, IBM Lotus® Web Content Management rendering portlet analytics, WebSphere Portal personalization logging, and various open source and commercial offerings.

This article also compares these tools and technologies, providing usage guidelines in various scenarios and the design patterns and best practices for tracking portal usage.

The article also address how to interpret portal usage statistics to build quantitative portal effectiveness scores through a number of techniques and examples. These techniques enable mappings across the usage statistics, the prioritized portal objectives, and the nature of portal usage.

Finally, it wraps up by illustrating how to analyze the WebSphere Portal effectiveness scores for determining the next steps that you can take to improve effectiveness.

Prerequisites

To get the most from this article, you should have at least have a basic understanding of the concepts and the use of WebSphere Portal and, preferably, some real-life experience with how the WebSphere Portal affects various constituencies within the organization.

This article can be helpful to anyone working in WebSphere Portal management, particularly with respect to administrative and organizational aspects.


Measuring effectiveness: An overview

A portal is often perceived to provide significant benefits, but frequently these benefits are thought to be ​intangible. This article demonstrates how to assess, quantify, and assign impact to those benefits.

The measurement of portal effectiveness can be viewed conceptually as the difference between having the portal available and not having a portal available. This measurement incorporates the impact that WebSphere Portal can have on multiple communities and its overall benefits to the organization.

Table 1 illustrates the typical portal constituencies and their potential benefits.

Table 1. Portal constituencies and their impact
Portal constituenciesOverall benefits
Users
  • Consolidation, personalization, and customization of content and application delivery
  • Collaboration
  • Social networking
  • Self-service
Business stakeholders
  • Rapid adaptability to market and user community needs
  • User tracking and feedback
  • Direct control
  • Improved tracking and analysis
IT stakeholders
  • IT infrastructure consolidation
  • Leverage of standards-based Plug and Play
  • Availability of new platform capabilities
Help desk team
  • Consolidated user service functions

The process for measuring WebSphere Portal effectiveness consists of: these tasks:

  • Monitoring usage and satisfaction
  • Determining key impact criteria and priorities to construct metrics
  • Building the WebSphere Portal value model

Figure 1 illustrates this process.

Figure 1. Strategy for measuring portal effectiveness
Strategy for measuring portal effectiveness

Tracking portal usage and satisfaction

After the portal is live, there are important questions to address as to whether the portal is meeting its expectations and how it's being used:

  • How satisfied are the users with the portal?
  • What are the most frequent user complaints about the portal? Are the complaints growing, steady, or declining?
  • How many users are accessing the portal and when is the peak usage? Is the usage growing, steady, or declining?
  • What are the profiles of the users who access the portal most frequently? How much time do the users spend on the Dashboard page? Can user activity be correlated with their profiles or with any known external events?
  • What is the average time for a content item to get stale? What is the average number of search queries for users to find what they're looking for?
  • What WebSphere Portal applications and functions do users access the most?

There is no single means to answer these questions; rather, they can be addressed through a composite of techniques. Table 2 describes some of the available techniques.

Table 2. Techniques to track WebSphere Portal usage and satisfaction
TechniqueDescriptionWhen to use
Server-side site analytics (based on site analytics file analysis)
  • User interactions and metadata are derived from fine-grained usage information stored in the site analytics file by the WebSphere Portal server.
  • Also, the Java™ Portlet Specification (JSR 286) Web Content Viewer portlet generates usage statistics on instances of the rendered web content.
  • A number of report-generation packages are available to process the WebDAV format file, including open source packages (such as AWStats) and commercial analysis packages.
For portal sites requiring visibility into the user behaviors but not requiring any real-time feedback.
Real-time active analytics
  • JavaScript within the portal page (for example, theme and skin or portlet markup) notifies the analytics server about user activity that is collected online.
  • Analytics systems, such as Coremetrics, gather and process the data, allowing the reporting of page and portlet views and of specific portlet business events.
  • Enable live viewing and drill-down into the portal analytics data.
For e-commerce sites supporting user campaigns.
Feedback database
  • Collects data on the personalization rules that are triggered and on user actions and behaviors.
For sites actively supporting user personalization.
User satisfaction surveys
  • Survey forms (delivered using WebSphere Portal dashboard or email) enabling portal users to periodically rank and evaluate their experience across a range of criteria. Surveys can be combined with a report-generation package for analysis of the survey results.
All sites.
User complaint tracking
  • Analysis of the help desk tickets on the portal to gauge the nature and the frequency of reported problems.
Sites supporting an active user help desk.
Observations on live portal usage
  • User groups can be monitored in a controlled environment to track their live interactions with the portal, to help gauge whether usage is meeting business and usability expectations.
Mission-critical sites.

Figure 2 shows a snippet of the WebSphere Portal site analytics log file for server-side analytics.

Figure 2. Log file snippet
Log file snippet

Refer to the developerWorks® article, “IBM WebSphere Developer Technical Journal: 'Using portal analytics with open-source reporting tools',” for open source-tool approaches to processing the log file to generate usage reports. These reporting tools provide a low-cost means to gain insights into both the overall portal web-traffic characteristics and the portions of the portal that are being used heavily.

There are also more advanced commercial products available to perform real-time monitoring of WebSphere Portal and to deliver on a dashboard the detailed analysis of portal activity. Some examples of vendors that offer such products are Webtrends and Coremetrics, as discussed in the paper, “IBM WebSphere Portal version 6.1.5 Tracking and Reporting Using Webtrends.”


Five-step process for measuring effectiveness

WebSphere Portal usage and satisfaction data can be correlated against business goals to determine effectiveness. This correlation can be viewed as a five-step process:

  1. Map business goals to their supporting WebSphere Portal capabilities.
  2. Develop WebSphere Portal capabilities into design artifacts that can lead to specific portal activities to be monitored.
  3. Assign impact formulas to the monitored WebSphere Portal activities based on the traffic and associated business value.
  4. Construct impact ratios based on anticipated impact and measured impact.
  5. Develop an overall effectiveness ranking based on multiple criteria, including impact ratios, user satisfaction surveys, problem reports, and usability studies.

Figure 3 illustrates these five steps.

Figure 3. Model for measuring portal effectiveness
Model for measuring portal effectiveness

Step 1. Map business goals to portal capabilities: Financial Representatives Insurance portal example

The first step in determining WebSphere Portal effectiveness is to identify the business goals, with the priorities of those goals, and to map them to the applicable portal technologies.

Let's demonstrate this concept by using the example of the hypothetical Financial Representatives Insurance portal that delivers information and services to independent field representatives.

Table 3 illustrates an example of mapping the business goals for the insurance portal to WebSphere Portal server capabilities.

Table 3. Mapping business goals to WebSphere Portal server capabilities
Sample business goalsBusiness goal priorityPortal capabilitiesSample applicable WebSphere Portal technology
Increase revenue for each representativeHighEnable delivery of full suite of customer relationship management applicationsApplication integration

Custom portlets
Enhance the effectiveness of information and services HighDeliver targeted content Content personalization

Content management
Reduce support costsMediumProvide self-service capabilities and help functionsApplication integration, custom portlets, and content delivery
Increase the appeal against competitive offeringsMediumProvide rich interaction capabilitiesAsynchronous Java and XML (Ajax)-based user interface and semantic tags

Search
Improve communications and interactions with representatives MediumDeliver blogs and forumsCollaboration suite

Step 2. Identify the WebSphere Portal activities to be measured

The next step is to map the requirements and portal capabilities derived from Step 1 to facets of portal design so that you can identify the WebSphere Portal activities to be monitored.

Table 4 illustrates an example of mapping aspects of portal design to the monitored WebSphere Portal activities with the sample insurance portal.

Table 4. Identify portal activities to be monitored
Portal capabilityPortal designSample monitored WebSphere Portal activity
Customer relationship management (CRM) application deliveryDeliver role-based access to the full suite of eService applicationsLaunch of eMyClient application

Launch of Client Insight portlet
Targeted contentDeliver personalized content for products and servicesPortfolio and Annuities content access
Self-service and helpDeliver user profile application and help contentLaunch of Profile application

Help content access
Rich interactionsWeb 2.0 theme and search engineSearch engine hits (user satisfaction scores in portal surveys)
Blogs and forums
  • Web Content Management blogs
  • Lotus Quickr™
  • Lotus Connections
Number of blogs

Forum posts

Step 3. Build impact formulas

In this step, the activities monitored in Step 2 are incorporated into a set of impact formulas. You can do this task by associating a proportional relevance derived from the value of the business goal associated with the monitored WebSphere Portal activity and combining it with the portal traffic figures for the activity.

Table 5 outlines this step for the sample insurance portal.

Table 5. Building WebSphere Portal impact formulas
Monitored activityBusiness goal and valueProportional relevance (adjusted across scale of 100)Sample impact formulas
eMyClient application launchIncrease revenue (High, score 3)14<Average daily eMyClient launches>
Client Insight portlet launchIncrease revenue (High, score 3)14<Average daily Insight launches >
Content access - PortfolioEnhance information delivery (High, score 3)13<Average daily Portfolio access >
Content access - Annuities Enhance information delivery (High, score 3)13<Average daily Annuities access >
User-profile application accessReduce support cost (Medium, score 2)9<Average daily Profile access >
Search engine hitsIncrease appeal (Medium, score 2)10<Average daily search queries>
Content access - HelpReduce support cost (Medium, score 2)9<Average daily help access >
Blog postsImprove communications (Medium, score 2)9<Average daily blog access>
Forum postsImprove communications (Medium, score 2)9<Average daily forum posts>

Step 4. Construct WebSphere Portal activity-impact ratios

We construct the quantitative scores to assess the impact of the activities by using the formulas in table 5. Both the anticipated impact score, based on the original expectations of the portal, and the measured impact score are constructed, so that you can build an impact ratio of the measured as opposed to the anticipated.

Table 6 outlines the process of generating impact ratios with the sample insurance portal. NOTE: The impact formulas and porportional relevance are defined in table 5.

NOTE: The impact ratios of the monitored activities are averaged to construct the overall portal impact ratio.

Table 6. Constructing impact scores and ratios for the portal
Monitored activityAnticipated impactMeasured impactImpact ratio (Measured/anticipated) * (proportional relevance / 100)
eMyClient application launch1200040005%
Client Insight Portlet launch120001000012%
Content access - Portfolio36000120005%
Content access - Annuities 36000150006%
User-profile application access400030009%
Search engine hits24000150006%
Content access - Help200010006%
Blog posts10006005%
Forum posts10006506%
Overall portal-activity impact ratioN/AN/A60%

Step 5. Build WebSphere Portal effectiveness ranking

Measuring the impact of the activities as described previously is only one of several dimensions to assessing WebSphere effectiveness. The portal activity impact ratios can be combined with user satisfaction measurements, problem ticket reports, and usability studies to build an overall effectiveness ranking (see table 7).

Table 7. Constructing the effectiveness ranking with the sample insurance portal
RankingGoalMeasuredImpact ratio [(measured / goals) * 100]
Activity impact 10060 (mean impact score from table 6)60
User satisfaction surveysHigh satisfaction (Ranking: 3) Medium-high satisfaction (Ranking: 2)66
Problem reportsAverage daily reports less than 20Average daily reports around 4050
Usability studiesVery usable (Ranking: 3)Medium-low usability (Ranking: 1.5)50
Overall portal effectiveness ratioN/AN/A57

The value of 57 for the portal effectiveness ratio is below expectations, leading to a “B-” ranking for our sample portal; a 90 to 100 percent portal-effectiveness-ratio range in our example would earn an “A” ranking.

Now let's analyze how to improve the effectiveness of the sample Insurance portal to at least a “B” ranking.


Tuning WebSphere Portal effectiveness

The data in tables 1–7 provide some key metrics to enable you to improve WebSphere Portal effectiveness. For example, table 6 shows that the eMyClient application launch from the portal is one-third of its anticipated value; in addition, there are significant shortfalls in the levels of anticipated content access to Portfolio and Annuities products.

Some new strategies to improving effectiveness can include the following:

  • Improving access to the eMyClient application. Place the eMyClient application launch icon on the portal home page for ease of access, and add new content highlighting the availability of eMyClient.
  • Enhancing the quality of content for Portfolio and Annuities. Increase the available new content for Portfolio and Annuities products by speeding up the content approval process.
  • Enhancing the portal usability. Refine the portal navigation to improve on the portal usability experience.

These two strategies are just a start; the objectives of measuring WebSphere Portal effectiveness are to build a continuous refinement and measurement cycle.


Conclusion

It is possible to systematically measure WebSphere Portal effectiveness by correlating business goals with the monitored activities on the portal. This approach enables the compiling of metrics that can help guide you toward specific actions to improve effectiveness, which is an integral part of ensuring that the portal is aligned for success.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere, Lotus
ArticleID=499029
ArticleTitle=Measuring IBM WebSphere Portal effectiveness
publish-date=07132010