Fly on the Wall

Get to know what your users want with this low-cost method for collecting valid customer data

For developers to make products that delight customers, they need adequate information about who exactly the customers are and what their requirements are. The User-Centered Design (UCD) process provides numerous options for gathering both customer and user input, with wide variation regarding the time involved, labor required, overhead costs, and validity of the information collected. The "Fly on the Wall" (FOTW) technique is a low-cost, low-overhead method of collecting valid customer data. The method is illustrated here through a pilot study that used first-hand, unobtrusive observations by UCD practitioners to collect valid customer data in a timely, cost-effective manner in collaboration with development and marketing staff.

Share:

Eyitope St. Matthew-Daniel (eyitope@us.ibm.com), Software Engineer, IBM Solution Technologies Group

Eyitope St. Matthew-Daniel is a Software Engineer with the IBM Solution Technologies group in Austin, TX. She has worked on projects utilizing the various UCD customer requirement gathering methods (FOTW, focus groups, interviews, etc). You can contact her at eyitope@us.ibm.com.



Robert J Kamper, PhD. (rkamper@us.ibm.com), Advisory Software Engineer, IBM Solution Technologies Group

Robert Kamper is an Advisory Software Engineer with the IBM Solution Technologies group in Austin, TX. His interests include software and hardware human-computer interface design and accessibility. He is currently developing the "Lead, Follow, and Get Out of the Way" methodology for design and evaluation of ease of use. Dr. Kamper can be contacted at rkamper@us.ibm.com.



01 August 2001

Today's marketplace requires that successful companies be able to respond in a timely fashion to customer needs and priorities. The Web server industry is highly competitive. With major companies such as IBM, Sun Microsystems, and Hewlett-Packard in the market, the need to distinguish products and provide superior service becomes a key requirement for attracting new customers and retaining established ones. Customer needs and priorities become more important, since the company that best meets these needs the fastest will attract and retain customers. Another issue is that of the differing priorities of customers based on market segments. When designing products, companies must take into account the needs of the various market segments, and how well their strategies and products match customer expectations. To do this efficiently, companies must obtain valid customer information in a timely and cost-effective manner.

Traditional solutions

The User-Centered Design (UCD) process provides numerous options for gathering customer and user input. Some of the methods used include focus groups, surveys, and interviews. Focus groups allow participants to express their individual views without necessarily identifying themselves. It encourages brainstorming and is a good way to collect a vast amount of unfiltered data. Surveys and interviews can also be used to collect customer information. However, each of these methods has some significant drawbacks that affect obtaining timely information in a cost-effective manner. Some drawbacks are listed in the following table:

Table 1. Drawbacks of common methods for obtaining customer input
Focus groupsSurveysInterviews
At least six weeks to set upCheap but limited dataInvolves travel or phone cost; labor intensive
Travel costs involvedUsually no follow-up questionsLimited number of customers per interview
Limited number of customers per sessionTypically low response percentageFormality of interview might hinder information received

The "Fly on the Wall" technique

The "Fly on the Wall" (FOTW) technique is an unobtrusive observation technique where the observer has no interaction with the customers. FOTW enables the observer to obtain verbatim quotes from customers, while remaining unobtrusive and not interfering with the presentation. With time and resources at a premium, development teams should seriously consider using this method as part of an overall UCD approach to aid in developing products that will satisfy users while controlling costs.

Pilot study

To test the validity of this method, a study was conducted at one company's Executive Briefing Center. The pilot was conducted to gather valid customer data concerning a product line of network servers, while alleviating the drawbacks listed in the previous table, by doing the following:

  • Minimizing costs by leveraging the Executive Briefing Center as a potential source of direct and immediate access to the thoughts, feelings, needs, problems, and priorities of customers and users
  • Maximizing the amount of customer inputs per briefing session
  • Eliminating setup logistics by leveraging scheduled marketing presentations

Methodology

A UCD practitioner attended various briefings on network servers held over a span of two months (August and September of 2000). The practitioner's role was to listen to the presentations and observe customer reactions, concerns, and feedback without interfering with the presentations or asking questions. Before attending the briefing sessions, the UCD practitioner became familiar with the network servers product line, as part of the duties of a good observer.

Setting

The briefing center where the pilot was conducted hosts a variety of customers and presents them with information on the company's latest offerings, as well as future technologies and strategy. Part of the mission of the briefing center presentations is to improve customer satisfaction with products. As a result, customer objectives and customer concerns are a key part of the briefing agenda. This is where the UCD practitioner is able to collect the most data. Additionally, customers are encouraged to ask questions during the presentations on the various product offerings and future technologies. The UCD practitioner also uses these questions and their context to extract more information about customer concerns, and to determine how well the company's strategy fits in with the customer's goals.

Market segment

Customers were categorized into market segments in order to reveal how well customer needs across different segments were being met. Fourteen customers represented seven market segments. Table 2 shows the market segments and the associated companies. The companies are identified by their industry or organization type to protect their anonymity.

Table 2. Market segment and respective companies
Market segmentCompanies
Customer relationship management (CRM)Customer services firm
e-businesse-business infrastructure company
Finance and bankingBank 1
Bank 2
Bank 3
Health and insuranceHealth organization 1
Health organization 2
Health organization 3
Retail services and distributionRetail and distribution company
Science and technologyUniversity 1
University 2
Research organization
TelecommunicationsTelecommunications company 1
Telecommunications company 2

Study limitations

FOTW requires that the observer be silent during the presentation without asking questions or making suggestions. The inability to ask questions was a limitation of this study, since the observer could not delve deeply into customer concerns and ask questions to determine the root cause of the problem. However, the context of the comments and questions that were asked by customers made it easy to have a feel for the problem, and possible root causes. Additionally, since the observer could not ask questions, she was unable to address issues that might be important, but that neither the customer nor briefing host addressed. However, in most cases, the majority of the important issues were discussed and addressed appropriately. For in-depth elaboration and discussion of specific issues, it is more appropriate to have a predetermined agenda and to use special software that collects group responses while reserving anonymity.


Results

The UCD practitioner attended 14 customer briefings from August through September. A layout of the number of customers and the time it took to observe the customers is included in Table 3.

Table 3. Briefing session observations
DateCustomerNumber of individualsObservation time (hours)
August 18E-business infrastructure company43
August 21Bank 133
August 22Bank 273
August 23Health organization 161
August 24Retail and distribution company41.5
August 25Telecommunications company 151.5
August 30University 153
August 31Health organization 253
September 14Health organization 391.5
September 18Telecommunications company 2103
September 19University 241.5
September 20Bank 341
September 21Research organization33
September 27Customer services firm21.5
TOTALS7130.5

A total of 14 customer sets (71 individual customers) were observed in 30.5 hours, over the course of 33 days. Information about customer objectives, customer concerns, and customer needs and priorities were collected. The "observation time" is the amount of time the UCD practitioner spent at the briefing session collecting customer information. This time varied for the different customers and was dependent, not on the number of customers present, but on the issues and concerns of the customers.

In each briefing session, the agenda included a discussion of customer objectives (the reason they came for the briefing) and customer concerns. The customer objectives are shown in Table 4, organized by market segment. The number in parentheses indicates the total number of companies with a particular objective, while the number of "X"s in each box corresponds to the number of companies with that objective for a particular market segment. For example, a total of 18 companies specified operations and operation requirements as a concern, and it had the highest concern in the "Health and insurance" segment (five companies).

Table 4. Customer objectives by market segment
Market segment
Objectivee-businessFinance and bankingHealth and insuranceScience and technologyCRMTelecommunicationsRetail services and distribution
Operations/Operation requirements (18)XXXXXXXXXXXXXXXXXX
Operating system (7)XXXXXXX 
Future technology/Web environments (5) XX XX  X

Customer concerns were derived from questions and statements made during or after the presentations. The Customer concerns table shows the number of each concern within a market segment. The number in parentheses indicates the total number of companies concerned with a particular topic, while the number of "X"s in each box relates to the number of companies interested in that topic for a particular market segment. For example, a total of nine companies specified quality and reliability as a concern, and it had the highest concern in the finance and banking segment (three companies). Customer concerns are organized by market segment in Table 5.

Table 5. Customer concerns by market segment
Market segment
Objectivee-businessFinance and bankingHealth and insuranceScience and technologyCRMTelecommunicationsRetail services and distribution
Quality and reliability (9)XXXXXXX XX
Storage/Manageability (8)XXXX XXX X
Security (3)XX X   
Performance (3)  XXX   
Technology (2)   XX   
Hardware upgrades (1)   X   

Table 5 shows that quality and reliability are a major concern in a majority of the market segments. Storage is also a major concern. Common concerns across the market segments are listed below.

  • Storage: Many companies require large storage space for the operations and services they provide. It is a concern across all market segments, especially in the "CRM" and "Health and insurance" segment.
  • Reliability: Customers stated that reliability was a key issue, because lack of it reflects negatively on them to their customers.
  • Scalability/hardware upgrades: Companies are continuously growing and expanding their services. As a result, companies want to ensure that the products they buy now are scalable for future expansion. Some customers also cite their intentions for moving to Web-models in the near future as another reason for scalability and flexibility.
  • Security: This is an extreme concern for the finance and banking market segment, where services require high security.
  • Server consolidation: Customers are looking to server consolidation to maximize space and reduce server management complexity by minimizing the number of servers that need to be managed.
  • Memory: Memory is an extreme concern in the health and insurance market segment, where databases and applications are memory intensive.
  • Quality of support: The quality of support was a major source of frustration.

Using the common customer concerns, some tentative ease of use objectives were developed. Developers can also use briefing centers to highlight the key concerns they would like usability professionals to assess in further detail. The ease of use objectives are shown in Table 6.

Table 6. Common concerns and ease of use objectives
Common concernsEase of use objectives
StorageEnsure ease of use in storage management. Research possible ways customers can maximize storage systems we have available.
Scalability/hardware upgradesEnsure ease of use in upgrades and installation.
ReliabilityNo unscheduled downtime means easy to manage.
SecurityEnsure ease of use in security management.
Server consolidationIncrease ease of use for server management.
MemoryResearch ways in which customers can maximize the current amount of memory they have. Ensure ease of use by making it easy to manage.
Quality of supportEnsure customer satisfaction with telephone and onsite support. Conduct process re-engineering to increase customer satisfaction and regain customer trust.

So does FOTW work?

The results demonstrate that FOTW is a viable option for collecting timely data on customer needs and priorities. Compared to other methods (such as focus groups), setup, data collection, and processing are much faster with FOTW. For example, 14 customer groups were observed in 30.5 hours; an average focus group session can only have a maximum of 14 people per session. It takes approximately six weeks to set up one focus group session. With 14 customer groups and 71 individuals, this pilot study was conducted in eight weeks. Additionally, a budget for a focus group would, in most cases, have to include travel and meals. With FOTW, the catering costs were already absorbed by the Executive Briefing Center, and customers paid their own travel expenses, so no additional costs were incurred.

Table 7 contrasts the benefits and drawbacks of FOTW with those of other methods:

Table 7. FOTW versus other methods
FOTWFocus groupsSurveysInterviews
Little to no costTravel, meal, and other setup costsLow cost, but limited dataInvolves travel or phone cost or both; labor intensive
Verbatim customer commentsVerbatim customer commentsComments might be modifiedComments might be modified
Does not allow for follow-up questionsAllows for exploration of customer commentsUsually no follow-up questionsAllows for exploration of customer comments
Can observe a large number of customersLimited number of customers per sessionTypically low response percentageLimited number of customers per interview
Unobtrusive observationOvert interactionLittle interactionFormality of interview might hinder information received

FOTW has proven to be a good way to obtain customer data that directly relates to usability issues. FOTW is not only helpful for usability purposes, it can also provide marketing departments with information on how to better address customer needs. For example, finance and banking presentations should contain specific information on the quality and reliability of different network servers. In addition, sales and marketing should address the distinct advantages their company has over its competitors in the areas that customers are most interested in. FOTW also provides a method for compiling vast amounts of verbatim customer data available at these briefing sessions. The information can then be analyzed by various segments in order to improve overall customer satisfaction. For example, the customer concern data shows that service and support were of major concern. With this information, the service and support team can focus on re-engineering the current support process, and creating better integration between the different service and support areas required by customers.

There are some limitations with the FOTW technique, most notably that the observer cannot ask questions or make comments during the presentations. Depending on the depth of customer comments and explanations given, additional steps might be needed to get a complete customer perspective and determine a root cause. Using FOTW in conjunction with other techniques might help provide an overall picture of the customer's needs and concerns. For example, using FOTW to get the initial data, an interviewer can ask specific questions based on that data. This provides for a focused interview with a specific goal in mind. Other suggestions include using the FOTW in conjunction with occasional customer site visits, surveys, etc. Conducting focus groups with several customers is also another potential use of the FOTW data. In the focus group, issues raised during the FOTW data collection can be elaborated on and follow-up strategies can be addressed.

Overall, FOTW met the objectives of the pilot: to gather valid customer data concerning network server products while alleviating some of the drawbacks of other data-collection methods, such as cost, response time, etc. Not only can usability issues be addressed, but marketing, development, and design can now make specific modifications based on customer concerns and frustrations.


Summary

This pilot study demonstrated that FOTW is a viable option for leveraging the marketing environment for UCD customer input. FOTW offers the following advantages:

  • Verbatim customer input
  • Objective observations
  • Low overhead, if any
  • No setup necessary

Additionally, UCD practitioners and development team members can use this method to get exposure to customers and listen first-hand to some of their major concerns at no extra cost (at companies where briefing centers exist).

However, in using FOTW, development teams should be cognizant of the following limitations:

  • Lack of interaction by the observer
  • No interruption allowed
  • Observer cannot obtain details on customer comments during a session

Using FOTW in conjunction with other data-collection methods, (for example, interviewing, customer visits, etc.) can prove to be a complete method for staying abreast of customer's changing needs and priorities, and thus maintaining or attaining a profitable market share. Development teams using a user-centered design process with access to customer briefing sessions should strongly consider adding FOTW to their repertoire of methodologies for gaining customer requirements.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Web development on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Web development
ArticleID=11595
ArticleTitle=Fly on the Wall
publish-date=08012001