What is sentiment analysis?

Companies now have access to more data about their customers than ever before, presenting both an opportunity and a challenge: analyzing the vast amounts of textual data available and extracting meaningful insights to guide their business decisions.

From emails and tweets to online survey responses, chats with customer service representatives and reviews, the sources available to gauge customer sentiment are seemingly endless. Sentiment analysis systems help companies better understand their customers, deliver stronger customer experiences and improve their brand reputation.

With more ways than ever for people to express their feelings online, organizations need powerful tools to monitor what’s being said about them and their products and services in near real time. As companies adopt sentiment analysis and begin using it to analyze more conversations and interactions, it will become easier to identify customer friction points at every stage of the customer journey.

The latest artificial intelligence (AI) sentiment analysis tools help companies filter reviews and net promoter scores (NPS) for personal bias and get more objective opinions about their brand, products and services. For example, if a customer expresses a negative opinion along with a positive opinion in a review, a human assessing the review might label it negative before reaching the positive words. AI-enhanced sentiment classification helps sort and classify text in an objective manner, so this doesn’t happen, and both sentiments are reflected.

Sentiment analysis enables companies with vast troves of unstructured data to analyze and extract meaningful insights from it quickly and efficiently. With the amount of text generated by customers across digital channels, it’s easy for human teams to get overwhelmed with information. Strong, cloud-based, AI-enhanced customer sentiment analysis tools help organizations deliver business intelligence from their customer data at scale, without expending unnecessary resources.

Modern enterprises need to respond quickly in a crisis. Opinions expressed on social media, whether true or not, can destroy a brand reputation that took years to build. Robust, AI-enhanced sentiment analysis tools help executives monitor the overall sentiment surrounding their brand so they can spot potential problems and address them swiftly.

Sentiment analysis uses natural language processing (NLP) and machine learning (ML) technologies to train computer software to analyze and interpret text in a way similar to humans. The software uses one of two approaches, rule-based or ML—or a combination of the two known as hybrid.Each approach has its strengths and weaknesses; while a rule-based approach can deliver results in near real-time, ML based approaches are more adaptable and can typically handle more complex scenarios.

In the rule-based approach, software is trained to classify certain keywords in a block of text based on groups of words, or lexicons, that describe the author’s intent. For example, words in a positive lexicon might include “affordable,” “fast” and “well-made,” while words in a negative lexicon might feature “expensive,” “slow” and “poorly made”. The software then scans the classifier for the words in either the positive or negative lexicon and tallies up a total sentiment score based on the volume of words used and the sentiment score of each category.

With a machine learning (ML) approach, an algorithm is used to train software to gauge sentiment in a block of text using words that appear in the text as well as the order in which they appear. Developers use sentiment analysis algorithms to teach software how to identify emotion in text similarly to the way humans do. ML models continue to “learn” from the data they are fed, hence the name “machine learning”. Here are a few of the most commonly used classification algorithms:

• Linear regression: A statistics algorithm that describes a value (Y) based on a set of features (X).
   

• Naive Bayes: An algorithm that uses Bayes’ theorem to categorize words in a block of text.
   

• Support vector machines: A fast and efficient classification algorithm used to solve two-group classification problems.
   

• Deep learning (DL): Also known as an artificial neural network, deep learning is an advanced machine learning technique that links together multiple algorithms to mimic human brain function.

A hybrid approach to text analysis combines both ML and rule-based capabilities to optimize accuracy and speed. While highly accurate, this approach requires more resources, such as time and technical capacity, than the other two.

In addition to the different approaches used to build sentiment analysis tools, there are also different types of sentiment analysis that organizations turn to depending on their needs. The three most popular types, emotion based, fine-grained and aspect-based sentiment analysis (ABSA) all rely on the underlying software’s capacity to gauge something called polarity, the overall feeling that is conveyed by a piece of text.

Generally speaking, a text’s polarity can be described as either positive, negative or neutral, but by categorizing the text even further, for example into subgroups such as “extremely positive” or “extremely negative,” some sentiment analysis models can identify more subtle and complex emotions. The polarity of a text is the most commonly used metric for gauging textual emotion and is expressed by the software as a numerical rating on a scale of one to 100. Zero represents a neutral sentiment and 100 represents the most extreme sentiment.

Here are the three most widely used types of sentiment analysis:

Fine-grained, or graded, sentiment analysis is a type of sentiment analysis that groups text into different emotions and the level of emotion being expressed. The emotion is then graded on a scale of zero to 100, similar to the way consumer websites deploy star-ratings to measure customer satisfaction.

Aspect based sentiment analysis (ABSA) narrows the scope of what’s being examined in a body of text to a singular aspect of a product, service or customer experience a business wishes to analyze.

For example, a budget travel app might use ABSA to understand how intuitive a new user interface is or to gauge the effectiveness of a customer service chatbot. ABSA can help organizations better understand how their products are succeeding or falling short of customer expectations.

Emotional detection sentiment analysis seeks to understand the psychological state of the individual behind a body of text, including their frame of mind when they were writing it and their intentions. It is more complex than either fine-grained or ABSA and is typically used to gain a deeper understanding of a person’s motivation or emotional state. Rather than using polarities, like positive, negative or neutral, emotional detection can identify specific emotions in a body of text such as frustration, indifference, restlessness and shock.

Organizations conduct sentiment analysis for a variety of reasons. Here are some of the most popular use cases.

While sentiment analysis and the technologies underpinning it are growing rapidly, it is still a relatively new field. According to “Sentiment Analysis,” by Liu Bing (2020) the term has only been widely used since 2003.¹ There is still much to be learned and refined, here are some of the most common drawbacks and challenges.

Context is a critical component for understanding what emotion is being expressed in a block of text and one that frequently causes sentiment analysis tools to make mistakes. On a customer survey, for example, a customer might give two answers to the question: “What did you like about our app?” The first answer might be “functionality” and the second, “UX”. If the question being asked was different, for example, “What didn’t you like about our app?” it changes the meaning of the customer’s response without changing the words themselves. To correct this problem, the algorithm would need to be given the original context of the question the customer was responding to, a time-consuming tactic known as pre or post processing.

Regardless of the level or extent of its training, software has a hard time correctly identifying irony and sarcasm in a body of text. This is because often when someone is being sarcastic or ironic it’s conveyed through their tone of voice or facial expression and there is no discernable difference in the words they’re using. For example, when analyzing the phrase, “Awesome, another thousand-dollar parking ticket—just what I need,” a sentiment analysis tool would likely mistake the nature of the emotion being expressed and label it as positive because of the use of the word “awesome”.

Negation is when a negative word is used to convey a reversal of meaning in a sentence. For example, consider the sentence, “I wouldn’t say the shoes were cheap." What’s being expressed, is that the shoes were probably expensive, or at least moderately priced, but a sentiment analysis tool would likely miss this subtlety.

Idiomatic language, such as the use of—for example—common English phrases like “Let’s not beat around the bush,” or “Break a leg,” frequently confounds sentiment analysis tools and the ML algorithms that they’re built on. When human language phrases like the ones above are used on social media channels or in product reviews, sentiment analysis tools will either incorrectly identify them—the “break a leg” example could be incorrectly identified as something painful or sad, for example—or miss them completely.

Organizations who decide they want to deploy sentiment analysis to better understand their customers have two options for how they can go about it: either purchase an existing tool or build one of their own.

Businesses opting to build their own tool typically use an open-source library in a common coding language such as Python or Java. These libraries are useful because their communities are steeped in data science. Still, organizations looking to take this approach will need to make a considerable investment in hiring a team of engineers and data scientists.

Acquiring an existing software as a service (SaaS) sentiment analysis tool requires less initial investment and allows businesses to deploy a pre-trained machine learning model rather than create one from scratch. SaaS sentiment analysis tools can be up and running with just a few simple steps and are a good option for businesses who aren’t ready to make the investment necessary to build their own.