Solving common challenges in sentiment analysis with help from Project Debater

By and Sean Sodha | 6 minute read | August 10, 2020

Today, we are going to talk about a feature that isn’t always talked a lot in the AI space: sentiment analysis.

What is sentiment analysis?

Sentiment analysis is a subset of natural language processing (NLP) capabilities that provides high level filters for users when exploring and evaluating data. Popularly, sentiment analysis is used to construct an enhanced perspective on customer experiences and the voice of the customer. Sentiment analysis is typically used in combination with other natural language processing features, such as natural language classification.

At IBM Watson, we offer sentiment analysis as a key feature inside Watson Natural Language Understanding (NLU). Here is a deep dive into some recent and new announcements.

Why is sentiment analysis important?

Sentiment analysis is by far one of the most important and commonly used NLP features. Sentiment is the classification of emotions extracted from a piece of text, speech, or document. While we typically look at emotion to capture feelings, such as anger, sadness, joy, fear, hesitation, sentiment is a higher-level classifier that divides the spectrum of emotions into positive, negative, and neutral.

While it can be used as a high-level filter to weed out comments, tweets, and reviews, sentiment is typically used in tandem with other NLP features such as emotion, entity extraction, text classification and keyword detection. NLP users will typically use two or three of these features depending on their use case. Below is a chart showing a few examples of where different industries use different slices of NLP.

When to use sentiment analysis

Common Use Cases:

  • Customer feedback — Analyze written feedback to understand how customers feel about your products and services
  • Customer care — Optimize your customer care messaging and customer engagement
  • Contact Center Insights — Lighten the load of your agents and analysts, and enable them to determine at scale the trends, patterns and anomalies in your customer’s needs using text analytics and sentiment analysis
  • Brand intelligence — Monitor the language your audience, customers, experts, influencers, and media are using to talk about your brand online

There are a variety of use cases that need NLP to extract key insights and determine key business decisions. Think of the millions of calls that occur each day in call centers around the world, each customer calling about a product issue, a billing issue, a return order, a status update or a question. As terabytes of recorded data are stored, call centers can use speech-to-text services to convert the audio file into a text file, and then continue to apply NLP to the conversations. Within each conversation, one can retrieve the intent (this is known as intent classification, a type of text classifier), the emotion of each customer’s statements, and most importantly, the sentiment of the customer throughout the call. From these insights, call centers can optimize the number of representatives per issue, draft better scripts to ease customer tension, and deliver feedback to managers about general product issues, such as “customers seem to have a problem with our billing system” or “this is one of the things customers do not like about the product.”

This is just one use case for the inherent abilities of NLP. From the chart above, you can see other features that correspond with other use cases.

Challenges in current sentiment models

Sentiment analysis is a very powerful and simple tool to gain great insights from your data. But there are a few factors that can affect the accuracy of your results: bias and context.

Bias can be introduced during model training, when labeled data inadvertently includes biased examples. For example, if the sentence “My boss is older than me” is labeled negative and “My boss is younger than me” is labeled neutral, that is indicative that there is inherent bias against age in the data. This can be remedied by evaluating the training data for such bias and updating the labels.

As in real life, without appropriate context, sentiment can be incorrectly interpreted. Context errors show up in a few general classes: polysemy, multi-polarity and sarcasm and irony.

Polysemy refers to a word that has several meanings. For instance, “return” can mean a profit or something that has been sent back. So the word “return” could indicate a positive sentiment in an investment context but negative sentiment in a retail context. This can be remedied by having more customized models that use domain-specific examples in the training data.

Multi-polarity is when the text of interest contains varying sentiment, which may lead to a confusing overall sentiment of the text. For example, if a restaurant review stated “I really loved the food. The service was terrible, though.” Deducing an overall sentiment for these two sentences here is a little tricky. A better approach is to look at the sentiment for the keywords of interest: in this case, food and service.

Sarcastic statements intentionally imply a different sentiment than what is stated when taken at face value. “Wow, what a well-written book” could on the surface seem genuine, but in context can be sarcastic. Here context is very important. Sarcasm can be difficult even for humans to detect, and it’s all the more challenging in sentiment analysis. There are several algorithmic approaches to tackling this problem.

Sentiment analysis in Watson NLU

NLU provides a sentiment model that returns a sentiment score ranging from -1 to 1, with -1 being negative, 0 being neutral and 1 being positive. Out of the box, our Sentiment analysis feature informs the user if sentiment of the data is “positive” or “negative” and presents an associated score. For any given text it will also provide the sentiment of the keywords and entities present.

New improvements

Custom Sentiment Beta Feature
The Natural Language Understanding team is proud to announce the launch of a new beta feature called Custom Sentiment. One challenge in sentiment analysis is differentiating a phrase’s context. A common example is: “We had a lot of returns this month.”

From an investment banking perspective, this seems like a very positive sentiment — an investment seemed to have made positive returns. However, from a retail perspective, this would be a negative sentiment — there were many customers that returned products to this retail store. When using an out-of-the-box sentiment model, it is extremely challenging to analyze the sentiment of this statement without any context.

The new Custom Sentiment beta feature tackles this exact issue. By uploading an easy-to-build CSV file listing labeled examples with a good balance of positives, negatives and neutrals, you are able to train Watson to understand the language and nuances within your own domain or industry. Fill in your email below to receive documentation on how to build a custom sentiment model.

Project Debater (Sentiment Analysis)
We are also providing enhanced sentiment analysis in our out-of-the-box sentiment model by integrating technology from Project Debater. NLU has implemented idiom understanding to better understand sentiment associated with idioms such as “under the weather.” NLU will also be implementing sentiment shifter understanding, which are combinations of words that together have a different meaning and sentiment, such as “you gave it your best shot.”

Sentiment analysis is a more powerful tool when applied with sophistication and attention to the specific use case. Watson NLU’s new features for custom and out-of-the-box sentiment models will give you more precise, reliable results. Discover its value today.