Fly on the Wall

Get to know what your users want with this low-cost method for collecting valid customer data

Today's marketplace requires that successful companies be able to respond in a timely fashion to customer needs and priorities. The Web server industry is highly competitive. With major companies such as IBM, Sun Microsystems, and Hewlett-Packard in the market, the need to distinguish products and provide superior service becomes a key requirement for attracting new customers and retaining established ones. Customer needs and priorities become more important, since the company that best meets these needs the fastest will attract and retain customers. Another issue is that of the differing priorities of customers based on market segments. When designing products, companies must take into account the needs of the various market segments, and how well their strategies and products match customer expectations. To do this efficiently, companies must obtain valid customer information in a timely and cost-effective manner.

Traditional solutions

The User-Centered Design (UCD) process provides numerous options for gathering customer and user input. Some of the methods used include focus groups, surveys, and interviews. Focus groups allow participants to express their individual views without necessarily identifying themselves. It encourages brainstorming and is a good way to collect a vast amount of unfiltered data. Surveys and interviews can also be used to collect customer information. However, each of these methods has some significant drawbacks that affect obtaining timely information in a cost-effective manner. Some drawbacks are listed in the following table:

Table 1. Drawbacks of common methods for obtaining customer input
Focus groupsSurveysInterviews
At least six weeks to set upCheap but limited dataInvolves travel or phone cost; labor intensive
Travel costs involvedUsually no follow-up questionsLimited number of customers per interview
Limited number of customers per sessionTypically low response percentageFormality of interview might hinder information received

The "Fly on the Wall" technique

The "Fly on the Wall" (FOTW) technique is an unobtrusive observation technique where the observer has no interaction with the customers. FOTW enables the observer to obtain verbatim quotes from customers, while remaining unobtrusive and not interfering with the presentation. With time and resources at a premium, development teams should seriously consider using this method as part of an overall UCD approach to aid in developing products that will satisfy users while controlling costs.

Pilot study

To test the validity of this method, a study was conducted at one company's Executive Briefing Center. The pilot was conducted to gather valid customer data concerning a product line of network servers, while alleviating the drawbacks listed in the previous table, by doing the following:

  • Minimizing costs by leveraging the Executive Briefing Center as a potential source of direct and immediate access to the thoughts, feelings, needs, problems, and priorities of customers and users
  • Maximizing the amount of customer inputs per briefing session
  • Eliminating setup logistics by leveraging scheduled marketing presentations


A UCD practitioner attended various briefings on network servers held over a span of two months (August and September of 2000). The practitioner's role was to listen to the presentations and observe customer reactions, concerns, and feedback without interfering with the presentations or asking questions. Before attending the briefing sessions, the UCD practitioner became familiar with the network servers product line, as part of the duties of a good observer.


The briefing center where the pilot was conducted hosts a variety of customers and presents them with information on the company's latest offerings, as well as future technologies and strategy. Part of the mission of the briefing center presentations is to improve customer satisfaction with products. As a result, customer objectives and customer concerns are a key part of the briefing agenda. This is where the UCD practitioner is able to collect the most data. Additionally, customers are encouraged to ask questions during the presentations on the various product offerings and future technologies. The UCD practitioner also uses these questions and their context to extract more information about customer concerns, and to determine how well the company's strategy fits in with the customer's goals.

Market segment

Customers were categorized into market segments in order to reveal how well customer needs across different segments were being met. Fourteen customers represented seven market segments. Table 2 shows the market segments and the associated companies. The companies are identified by their industry or organization type to protect their anonymity.

Table 2. Market segment and respective companies
Market segmentCompanies
Customer relationship management (CRM)Customer services firm
e-businesse-business infrastructure company
Finance and bankingBank 1
Bank 2
Bank 3
Health and insuranceHealth organization 1
Health organization 2
Health organization 3
Retail services and distributionRetail and distribution company
Science and technologyUniversity 1
University 2
Research organization
TelecommunicationsTelecommunications company 1
Telecommunications company 2

Study limitations

FOTW requires that the observer be silent during the presentation without asking questions or making suggestions. The inability to ask questions was a limitation of this study, since the observer could not delve deeply into customer concerns and ask questions to determine the root cause of the problem. However, the context of the comments and questions that were asked by customers made it easy to have a feel for the problem, and possible root causes. Additionally, since the observer could not ask questions, she was unable to address issues that might be important, but that neither the customer nor briefing host addressed. However, in most cases, the majority of the important issues were discussed and addressed appropriately. For in-depth elaboration and discussion of specific issues, it is more appropriate to have a predetermined agenda and to use special software that collects group responses while reserving anonymity.


The UCD practitioner attended 14 customer briefings from August through September. A layout of the number of customers and the time it took to observe the customers is included in Table 3.

Table 3. Briefing session observations
DateCustomerNumber of individualsObservation time (hours)
August 18E-business infrastructure company43
August 21Bank 133
August 22Bank 273
August 23Health organization 161
August 24Retail and distribution company41.5
August 25Telecommunications company 151.5
August 30University 153
August 31Health organization 253
September 14Health organization 391.5
September 18Telecommunications company 2103
September 19University 241.5
September 20Bank 341
September 21Research organization33
September 27Customer services firm21.5

A total of 14 customer sets (71 individual customers) were observed in 30.5 hours, over the course of 33 days. Information about customer objectives, customer concerns, and customer needs and priorities were collected. The "observation time" is the amount of time the UCD practitioner spent at the briefing session collecting customer information. This time varied for the different customers and was dependent, not on the number of customers present, but on the issues and concerns of the customers.

In each briefing session, the agenda included a discussion of customer objectives (the reason they came for the briefing) and customer concerns. The customer objectives are shown in Table 4, organized by market segment. The number in parentheses indicates the total number of companies with a particular objective, while the number of "X"s in each box corresponds to the number of companies with that objective for a particular market segment. For example, a total of 18 companies specified operations and operation requirements as a concern, and it had the highest concern in the "Health and insurance" segment (five companies).

Table 4. Customer objectives by market segment
Market segment
Objectivee-businessFinance and bankingHealth and insuranceScience and technologyCRMTelecommunicationsRetail services and distribution
Operations/Operation requirements (18)XXXXXXXXXXXXXXXXXX
Operating system (7)XXXXXXX 
Future technology/Web environments (5) XX XX  X

Customer concerns were derived from questions and statements made during or after the presentations. The Customer concerns table shows the number of each concern within a market segment. The number in parentheses indicates the total number of companies concerned with a particular topic, while the number of "X"s in each box relates to the number of companies interested in that topic for a particular market segment. For example, a total of nine companies specified quality and reliability as a concern, and it had the highest concern in the finance and banking segment (three companies). Customer concerns are organized by market segment in Table 5.

Table 5. Customer concerns by market segment
Market segment
Objectivee-businessFinance and bankingHealth and insuranceScience and technologyCRMTelecommunicationsRetail services and distribution
Quality and reliability (9)XXXXXXX XX
Storage/Manageability (8)XXXX XXX X
Security (3)XX X   
Performance (3)  XXX   
Technology (2)   XX   
Hardware upgrades (1)   X   

Table 5 shows that quality and reliability are a major concern in a majority of the market segments. Storage is also a major concern. Common concerns across the market segments are listed below.

  • Storage: Many companies require large storage space for the operations and services they provide. It is a concern across all market segments, especially in the "CRM" and "Health and insurance" segment.
  • Reliability: Customers stated that reliability was a key issue, because lack of it reflects negatively on them to their customers.
  • Scalability/hardware upgrades: Companies are continuously growing and expanding their services. As a result, companies want to ensure that the products they buy now are scalable for future expansion. Some customers also cite their intentions for moving to Web-models in the near future as another reason for scalability and flexibility.
  • Security: This is an extreme concern for the finance and banking market segment, where services require high security.
  • Server consolidation: Customers are looking to server consolidation to maximize space and reduce server management complexity by minimizing the number of servers that need to be managed.
  • Memory: Memory is an extreme concern in the health and insurance market segment, where databases and applications are memory intensive.
  • Quality of support: The quality of support was a major source of frustration.

Using the common customer concerns, some tentative ease of use objectives were developed. Developers can also use briefing centers to highlight the key concerns they would like usability professionals to assess in further detail. The ease of use objectives are shown in Table 6.

Table 6. Common concerns and ease of use objectives
Common concernsEase of use objectives
StorageEnsure ease of use in storage management. Research possible ways customers can maximize storage systems we have available.
Scalability/hardware upgradesEnsure ease of use in upgrades and installation.
ReliabilityNo unscheduled downtime means easy to manage.
SecurityEnsure ease of use in security management.
Server consolidationIncrease ease of use for server management.
MemoryResearch ways in which customers can maximize the current amount of memory they have. Ensure ease of use by making it easy to manage.
Quality of supportEnsure customer satisfaction with telephone and onsite support. Conduct process re-engineering to increase customer satisfaction and regain customer trust.

So does FOTW work?

The results demonstrate that FOTW is a viable option for collecting timely data on customer needs and priorities. Compared to other methods (such as focus groups), setup, data collection, and processing are much faster with FOTW. For example, 14 customer groups were observed in 30.5 hours; an average focus group session can only have a maximum of 14 people per session. It takes approximately six weeks to set up one focus group session. With 14 customer groups and 71 individuals, this pilot study was conducted in eight weeks. Additionally, a budget for a focus group would, in most cases, have to include travel and meals. With FOTW, the catering costs were already absorbed by the Executive Briefing Center, and customers paid their own travel expenses, so no additional costs were incurred.

Table 7 contrasts the benefits and drawbacks of FOTW with those of other methods:

Table 7. FOTW versus other methods
FOTWFocus groupsSurveysInterviews
Little to no costTravel, meal, and other setup costsLow cost, but limited dataInvolves travel or phone cost or both; labor intensive
Verbatim customer commentsVerbatim customer commentsComments might be modifiedComments might be modified
Does not allow for follow-up questionsAllows for exploration of customer commentsUsually no follow-up questionsAllows for exploration of customer comments
Can observe a large number of customersLimited number of customers per sessionTypically low response percentageLimited number of customers per interview
Unobtrusive observationOvert interactionLittle interactionFormality of interview might hinder information received

FOTW has proven to be a good way to obtain customer data that directly relates to usability issues. FOTW is not only helpful for usability purposes, it can also provide marketing departments with information on how to better address customer needs. For example, finance and banking presentations should contain specific information on the quality and reliability of different network servers. In addition, sales and marketing should address the distinct advantages their company has over its competitors in the areas that customers are most interested in. FOTW also provides a method for compiling vast amounts of verbatim customer data available at these briefing sessions. The information can then be analyzed by various segments in order to improve overall customer satisfaction. For example, the customer concern data shows that service and support were of major concern. With this information, the service and support team can focus on re-engineering the current support process, and creating better integration between the different service and support areas required by customers.

There are some limitations with the FOTW technique, most notably that the observer cannot ask questions or make comments during the presentations. Depending on the depth of customer comments and explanations given, additional steps might be needed to get a complete customer perspective and determine a root cause. Using FOTW in conjunction with other techniques might help provide an overall picture of the customer's needs and concerns. For example, using FOTW to get the initial data, an interviewer can ask specific questions based on that data. This provides for a focused interview with a specific goal in mind. Other suggestions include using the FOTW in conjunction with occasional customer site visits, surveys, etc. Conducting focus groups with several customers is also another potential use of the FOTW data. In the focus group, issues raised during the FOTW data collection can be elaborated on and follow-up strategies can be addressed.

Overall, FOTW met the objectives of the pilot: to gather valid customer data concerning network server products while alleviating some of the drawbacks of other data-collection methods, such as cost, response time, etc. Not only can usability issues be addressed, but marketing, development, and design can now make specific modifications based on customer concerns and frustrations.


This pilot study demonstrated that FOTW is a viable option for leveraging the marketing environment for UCD customer input. FOTW offers the following advantages:

  • Verbatim customer input
  • Objective observations
  • Low overhead, if any
  • No setup necessary

Additionally, UCD practitioners and development team members can use this method to get exposure to customers and listen first-hand to some of their major concerns at no extra cost (at companies where briefing centers exist).

However, in using FOTW, development teams should be cognizant of the following limitations:

  • Lack of interaction by the observer
  • No interruption allowed
  • Observer cannot obtain details on customer comments during a session

Using FOTW in conjunction with other data-collection methods, (for example, interviewing, customer visits, etc.) can prove to be a complete method for staying abreast of customer's changing needs and priorities, and thus maintaining or attaining a profitable market share. Development teams using a user-centered design process with access to customer briefing sessions should strongly consider adding FOTW to their repertoire of methodologies for gaining customer requirements.

Downloadable resources

Related topics

  • Jared Spool's usability site, User Interface Engineering places special emphasis on research methods.
  • Usability First provides information and resources for a range of issues related to usability in Web site and software design.
  • James Hom's Usability Methods Tool Box features information on usability methods and evaluation techniques.
  • Guide to Usability for Software Engineers is a collection of pages created by engineering students at the University of Maryland. The collection is intended for software engineers and usability engineering practitioners to help them find relevant resources on the Web.
  • For more on FOTW, check out Task Analysis Methods for Instructional Design by David H. Jonassen, Martin Tessmer, and Wallace H. Hannum.
  • IBM Accessibility site offers new innovations, user-centered design, guidelines, stories, technologies, and other resources to help improve the total user experience for your products and services.
Zone=Web development
ArticleTitle=Fly on the Wall