Today's marketplace requires that successful companies be able to respond in a timely fashion to customer needs and priorities. The Web server industry is highly competitive. With major companies such as IBM, Sun Microsystems, and Hewlett-Packard in the market, the need to distinguish products and provide superior service becomes a key requirement for attracting new customers and retaining established ones. Customer needs and priorities become more important, since the company that best meets these needs the fastest will attract and retain customers. Another issue is that of the differing priorities of customers based on market segments. When designing products, companies must take into account the needs of the various market segments, and how well their strategies and products match customer expectations. To do this efficiently, companies must obtain valid customer information in a timely and cost-effective manner.

Traditional solutions

The User-Centered Design (UCD) process provides numerous options for gathering customer and user input. Some of the methods used include focus groups, surveys, and interviews. Focus groups allow participants to express their individual views without necessarily identifying themselves. It encourages brainstorming and is a good way to collect a vast amount of unfiltered data. Surveys and interviews can also be used to collect customer information. However, each of these methods has some significant drawbacks that affect obtaining timely information in a cost-effective manner. Some drawbacks are listed in the following table:

Back to top

The "Fly on the Wall" technique

The "Fly on the Wall" (FOTW) technique is an unobtrusive observation technique where the observer has no interaction with the customers. FOTW enables the observer to obtain verbatim quotes from customers, while remaining unobtrusive and not interfering with the presentation. With time and resources at a premium, development teams should seriously consider using this method as part of an overall UCD approach to aid in developing products that will satisfy users while controlling costs.

Pilot study

To test the validity of this method, a study was conducted at one company's Executive Briefing Center. The pilot was conducted to gather valid customer data concerning a product line of network servers, while alleviating the drawbacks listed in the previous table, by doing the following:

• Minimizing costs by leveraging the Executive Briefing Center as a potential source of direct and immediate access to the thoughts, feelings, needs, problems, and priorities of customers and users
• Maximizing the amount of customer inputs per briefing session
• Eliminating setup logistics by leveraging scheduled marketing presentations

Methodology

A UCD practitioner attended various briefings on network servers held over a span of two months (August and September of 2000). The practitioner's role was to listen to the presentations and observe customer reactions, concerns, and feedback without interfering with the presentations or asking questions. Before attending the briefing sessions, the UCD practitioner became familiar with the network servers product line, as part of the duties of a good observer.

Setting

The briefing center where the pilot was conducted hosts a variety of customers and presents them with information on the company's latest offerings, as well as future technologies and strategy. Part of the mission of the briefing center presentations is to improve customer satisfaction with products. As a result, customer objectives and customer concerns are a key part of the briefing agenda. This is where the UCD practitioner is able to collect the most data. Additionally, customers are encouraged to ask questions during the presentations on the various product offerings and future technologies. The UCD practitioner also uses these questions and their context to extract more information about customer concerns, and to determine how well the company's strategy fits in with the customer's goals.

Market segment

Customers were categorized into market segments in order to reveal how well customer needs across different segments were being met. Fourteen customers represented seven market segments. Table 2 shows the market segments and the associated companies. The companies are identified by their industry or organization type to protect their anonymity.

Study limitations

FOTW requires that the observer be silent during the presentation without asking questions or making suggestions. The inability to ask questions was a limitation of this study, since the observer could not delve deeply into customer concerns and ask questions to determine the root cause of the problem. However, the context of the comments and questions that were asked by customers made it easy to have a feel for the problem, and possible root causes. Additionally, since the observer could not ask questions, she was unable to address issues that might be important, but that neither the customer nor briefing host addressed. However, in most cases, the majority of the important issues were discussed and addressed appropriately. For in-depth elaboration and discussion of specific issues, it is more appropriate to have a predetermined agenda and to use special software that collects group responses while reserving anonymity.

Back to top

Results

The UCD practitioner attended 14 customer briefings from August through September. A layout of the number of customers and the time it took to observe the customers is included in Table 3.

A total of 14 customer sets (71 individual customers) were observed in 30.5 hours, over the course of 33 days. Information about customer objectives, customer concerns, and customer needs and priorities were collected. The "observation time" is the amount of time the UCD practitioner spent at the briefing session collecting customer information. This time varied for the different customers and was dependent, not on the number of customers present, but on the issues and concerns of the customers.

In each briefing session, the agenda included a discussion of customer objectives (the reason they came for the briefing) and customer concerns. The customer objectives are shown in Table 4, organized by market segment. The number in parentheses indicates the total number of companies with a particular objective, while the number of "X"s in each box corresponds to the number of companies with that objective for a particular market segment. For example, a total of 18 companies specified operations and operation requirements as a concern, and it had the highest concern in the "Health and insurance" segment (five companies).

Customer concerns were derived from questions and statements made during or after the presentations. The Customer concerns table shows the number of each concern within a market segment. The number in parentheses indicates the total number of companies concerned with a particular topic, while the number of "X"s in each box relates to the number of companies interested in that topic for a particular market segment. For example, a total of nine companies specified quality and reliability as a concern, and it had the highest concern in the finance and banking segment (three companies). Customer concerns are organized by market segment in Table 5.

Table 5 shows that quality and reliability are a major concern in a majority of the market segments. Storage is also a major concern. Common concerns across the market segments are listed below.

• Storage: Many companies require large storage space for the operations and services they provide. It is a concern across all market segments, especially in the "CRM" and "Health and insurance" segment.
• Reliability: Customers stated that reliability was a key issue, because lack of it reflects negatively on them to their customers.
• Scalability/hardware upgrades: Companies are continuously growing and expanding their services. As a result, companies want to ensure that the products they buy now are scalable for future expansion. Some customers also cite their intentions for moving to Web-models in the near future as another reason for scalability and flexibility.
• Security: This is an extreme concern for the finance and banking market segment, where services require high security.
• Server consolidation: Customers are looking to server consolidation to maximize space and reduce server management complexity by minimizing the number of servers that need to be managed.
• Memory: Memory is an extreme concern in the health and insurance market segment, where databases and applications are memory intensive.
• Quality of support: The quality of support was a major source of frustration.

Using the common customer concerns, some tentative ease of use objectives were developed. Developers can also use briefing centers to highlight the key concerns they would like usability professionals to assess in further detail. The ease of use objectives are shown in Table 6.

Back to top

So does FOTW work?

The results demonstrate that FOTW is a viable option for collecting timely data on customer needs and priorities. Compared to other methods (such as focus groups), setup, data collection, and processing are much faster with FOTW. For example, 14 customer groups were observed in 30.5 hours; an average focus group session can only have a maximum of 14 people per session. It takes approximately six weeks to set up one focus group session. With 14 customer groups and 71 individuals, this pilot study was conducted in eight weeks. Additionally, a budget for a focus group would, in most cases, have to include travel and meals. With FOTW, the catering costs were already absorbed by the Executive Briefing Center, and customers paid their own travel expenses, so no additional costs were incurred.

Table 7 contrasts the benefits and drawbacks of FOTW with those of other methods:

FOTW has proven to be a good way to obtain customer data that directly relates to usability issues. FOTW is not only helpful for usability purposes, it can also provide marketing departments with information on how to better address customer needs. For example, finance and banking presentations should contain specific information on the quality and reliability of different network servers. In addition, sales and marketing should address the distinct advantages their company has over its competitors in the areas that customers are most interested in. FOTW also provides a method for compiling vast amounts of verbatim customer data available at these briefing sessions. The information can then be analyzed by various segments in order to improve overall customer satisfaction. For example, the customer concern data shows that service and support were of major concern. With this information, the service and support team can focus on re-engineering the current support process, and creating better integration between the different service and support areas required by customers.

There are some limitations with the FOTW technique, most notably that the observer cannot ask questions or make comments during the presentations. Depending on the depth of customer comments and explanations given, additional steps might be needed to get a complete customer perspective and determine a root cause. Using FOTW in conjunction with other techniques might help provide an overall picture of the customer's needs and concerns. For example, using FOTW to get the initial data, an interviewer can ask specific questions based on that data. This provides for a focused interview with a specific goal in mind. Other suggestions include using the FOTW in conjunction with occasional customer site visits, surveys, etc. Conducting focus groups with several customers is also another potential use of the FOTW data. In the focus group, issues raised during the FOTW data collection can be elaborated on and follow-up strategies can be addressed.

Overall, FOTW met the objectives of the pilot: to gather valid customer data concerning network server products while alleviating some of the drawbacks of other data-collection methods, such as cost, response time, etc. Not only can usability issues be addressed, but marketing, development, and design can now make specific modifications based on customer concerns and frustrations.

Back to top

Summary

This pilot study demonstrated that FOTW is a viable option for leveraging the marketing environment for UCD customer input. FOTW offers the following advantages:

• Verbatim customer input
• Objective observations
• Low overhead, if any
• No setup necessary

Additionally, UCD practitioners and development team members can use this method to get exposure to customers and listen first-hand to some of their major concerns at no extra cost (at companies where briefing centers exist).

However, in using FOTW, development teams should be cognizant of the following limitations:

• Lack of interaction by the observer
• No interruption allowed
• Observer cannot obtain details on customer comments during a session

Using FOTW in conjunction with other data-collection methods, (for example, interviewing, customer visits, etc.) can prove to be a complete method for staying abreast of customer's changing needs and priorities, and thus maintaining or attaining a profitable market share. Development teams using a user-centered design process with access to customer briefing sessions should strongly consider adding FOTW to their repertoire of methodologies for gaining customer requirements.

Resources

• Jared Spool's usability site, User Interface Engineering places special emphasis on research methods.
  
  
• Usability First provides information and resources for a range of issues related to usability in Web site and software design.
  
  
• James Hom's Usability Methods Tool Box features information on usability methods and evaluation techniques.
  
  
• Guide to Usability for Software Engineers is a collection of pages created by engineering students at the University of Maryland. The collection is intended for software engineers and usability engineering practitioners to help them find relevant resources on the Web.
  
  
• For more on FOTW, check out Task Analysis Methods for Instructional Design by David H. Jonassen, Martin Tessmer, and Wallace H. Hannum.
  
  
• Preparing your Web site for machine translation: Improve the usability of your Machine Translation (MT) output with these preparatory tactics (developerWorks, July 2001).
  
  
• developworks Web architecture zone: Get more dW usability resources.
  
  
• IBM offers training courses in user-centered design -- find out more here.
  
  
• IBM's Ease of Use site offers new innovations, user-centered design, guidelines, stories, technologies, and other resources to help improve the total user experience for your products and services.
  
  

About the authors

Rate this article

Comments

Back to top

Table of contents

• Traditional solutions
• The "Fly on the Wall" technique
• Results
• So does FOTW work?
• Summary
• Resources
• About the authors
• Comments