Sentiment (targeted)

At a glance

The Targets Sentiment model contains algorithms for the task of extracting the sentiments expressed in text and identifying the targets of those sentiments.

For example, given the input:

The served food was delicious, yet the service was slow.

The model identifies that there is a positive sentiment expressed towards the target “food”, and a negative sentiment expressed towards “service”.

The model offers implementations with transformer-based algorithms.

Class definition
`watson_nlp.blocks.targeted_sentiment.SequenceTransformerTSA`

What’s the difference between the Sentiment (targeted) model and the Sentiment (aggregated) model?

The major difference between this model and the "targets sentiment classification" functionality provided in the Sentiment (aggregated) model, is whether the user has to specify the targets in the input.

In the Sentiment (aggregated) model, the user has to specify one or more target terms in the input, and the model classifies the sentiment of the input target terms.

For the Sentiment (targeted) model, the user only inputs the text. The model extracts the sentiments expressed in text and identifies the targets of those sentiments.

For example, consider the text:

The served food was delicious, yet the service was slow.

For this Sentiment (targeted) model, the only required input is the above sentence. The model does not require the user to manually specify any target terms that should be analyzed for their sentiment. All it requires is the input text.

For the Sentiment (aggregated) model, the user must also specify which targets or substrings to classify by sentiment, for example, both “food” and “service” have to be specified by the user in the example sentence used above.

A significant advantage of this Sentiment (targeted) model is that it can handle multiple targets with different sentiments in one sentence. For the above example, it automatically identifies the positive sentiment towards “food” and the negative sentiment towards “service”. In comparison, the Sentiment (aggregated) model induces the sentence sentiment to the input target terms. It is thus limited to classifying all targets within a sentence with the same sentiment.

Pretrained models

Model names are listed below. For language support, see Supported languages.

Model ID	Container Image
Transformer models
targets-sentiment_transformer-workflow_lang_multilingual_slate.153m.distilled	cp.icr.io/cp/ai/watson-nlp_targets-sentiment_transformer-workflow_lang_multilingual_slate.153m.distilled:1.4.1
targets-sentiment_transformer-workflow_lang_multilingual_slate.153m.distilled-cpu	cp.icr.io/cp/ai/watson-nlp_targets-sentiment_transformer-workflow_lang_multilingual_slate.153m.distilled-cpu:1.4.1
targets-sentiment_transformer-workflow_lang_multilingual_slate.270m	cp.icr.io/cp/ai/watson-nlp_targets-sentiment_transformer-workflow_lang_multilingual_slate.270m:1.4.1
targets-sentiment_transformer-workflow_lang_multilingual_slate.270m-cpu	cp.icr.io/cp/ai/watson-nlp_targets-sentiment_transformer-workflow_lang_multilingual_slate.270m-cpu:1.4.1

All models were fine-tuned and evaluated on labeled data from IBM internal datasets, covering user reviews from multiple domains: restaurants, electronics, automotives, movies, and other products or services.

All models are multilingual: they can analyze input texts from multiple languages. Since the models were fine-tuned solely on English data, they support non-English languages in a zero-shot manner.

Pre-trained `transformer` models

(T1): targets-sentiment_transformer_multilingual_slate.270m is based on the WatBERT model. It obtains the best TSA results quality, out of all available out-of-the-box models.

(T2): targets-sentiment_transformer_multilingual_slate.270m-cpu is a variant of (T1) optimized for running on a CPU.

(T3): targets-sentiment_transformer_multilingual_slate.153m.distilled is based on the distilled WatBERT model. It improves runtime, while slightly degrading the quality of the results (details below).

(T4): targets-sentiment_transformer_multilingual_slate.153m.distilled-cpu is a variant of (T3) optimized for running on a CPU.

Which transformer model should I use?

The choice of the model should depend on the importance of quality or runtime; higher-quality models are slower. Here are some additional details:

When running on a GPU:
- Use T1 or T3.
- Do not use T2 or T4 because they cannot take advantage of GPU hardware.
- Quality:
- English: The quality of the results produced by T1 is slightly better, it is better than the quality produced by T3 by around 5%-10%, depending on the domain of the analyzed data (i.e., whether it's a restaurant review, movie review, car review, etc.).
- Other languages: Since the models were only trained on English data, the quality gap between them is higher for non-English. T1 obtains better quality than T3 by about 10%-20%, on the tested languages (French, Spanish and Dutch). The quality also depends on the domain of the analyzed texts.
- Runtime: On a GPU, T3 is faster than T1 by a factor of ~1.5x.
When running on a CPU:
- Quality:
- English: The quality of the results produced by T1 is best, it is better than the quality produced by T2/T3 by around 5%-10%, depending on the domain of the analyzed data (i.e., whether it's a restaurant review, movie reviews car review, etc.). T2 and T3 are comparable in quality, with sometimes a slight advantage to T2. The quality of the results produced by T1 is better than the quality produced by T4 by 10%-15%.
- Other languages: Since the models were only trained on English data, the quality gap between them is higher for non-English. T1 obtains better quality than T2/T3 by about 10%-20%, on the tested languages (French, Spanish and Dutch). The quality also depends on the domain of the analyzed texts. T2 and T3 are comparable in quality, with sometimes a slight advantage to T2. The quality of the results produced by T1 is better than the quality produced by T4 by 10%-20%.
- Runtime On a CPU: T2 is faster than T1 by a factor of 2.5x. T3 is faster than T1 by ~5.5x. T4 is faster than T1 by ~13.5x.

Creating a custom model

For the current release of Watson Natural Language Processing Library for Embed, you can work with Python notebooks in Watson Studio to train some Watson NLP models with your own data. See Creating custom models for information.

Running models

The Targets Sentiment model request accepts the following fields:

Field	Type	Required Optional Repeated	Description
`raw_document`	`watson_core_data_model.nlp.RawDocument`	required	The input document on which to perform targeted sentiment predictions
`language_code`	`str`	optional	An optional language code for the input text, required if the model is multi-lingual

Example requests

REST API

curl -s \
  "http://localhost:8080/v1/watson.runtime.nlp.v1/NlpService/TargetsSentimentPredict" \
  -H "accept: application/json" \
  -H "content-type: application/json" \
  -H "Grpc-Metadata-mm-model-id: targets-sentiment_transformer-workflow_lang_multilingual_slate.270m" \
  -d '{ "raw_document": { "text": "The served food was delicious, yet the service was slow." } }'

Response

{"targetedSentiments":{"served food":{"score":0.9896019, "label":"SENT_POSITIVE", "mixed":false, "sentimentMentions":[{"span":{"begin":4, "end":15, "text":"served food"}, "sentimentprob":{"positive":0.9896019, "neutral":0, "negative":0.01039809}}]}, "service":{"score":-0.98677325, "label":"SENT_NEGATIVE", "mixed":false, "sentimentMentions":[{"span":{"begin":39, "end":46, "text":"service"}, "sentimentprob":{"positive":0.0132267475, "neutral":0, "negative":0.98677325}}]}}, "producerId":{"name":"Transformer-based Targets Sentiment Extraction Workflow", "version":"0.0.1"}}

Python

import grpc

from watson_nlp_runtime_client import (
    common_service_pb2,
    common_service_pb2_grpc,
    syntax_types_pb2,
)

channel = grpc.insecure_channel("localhost:8085")

stub = common_service_pb2_grpc.NlpServiceStub(channel)

request = common_service_pb2.TargetsSentimentRequest(
    raw_document=syntax_types_pb2.RawDocument(text="The served food was delicious, yet the service was slow.")
)

response = stub.TargetsSentimentPredict(
    request, metadata=[("mm-model-id", "targets-sentiment_transformer-workflow_lang_multilingual_slate.270m")]
)

print(response)

Response

targeted_sentiments {
  key: "service"
  value {
    score: -0.925867736
    label: SENT_NEGATIVE
    sentiment_mentions {
      span {
        begin: 39
        end: 46
        text: "service"
      }
      sentimentprob {
        positive: 0.0741322637
        negative: 0.925867736
      }
    }
  }
}
targeted_sentiments {
  key: "served food"
  value {
    score: 0.961341143
    label: SENT_POSITIVE
    sentiment_mentions {
      span {
        begin: 4
        end: 15
        text: "served food"
      }
      sentimentprob {
        positive: 0.961341143
        negative: 0.0386588573
      }
    }
  }
}
producer_id {
  name: "Transformer-based Targets Sentiment Extraction Workflow"
  version: "0.0.1"
}

Sentiment (targeted)

At a glance

What’s the difference between the Sentiment (targeted) model and the Sentiment (aggregated) model?

Pretrained models

Pre-trained transformer models

Which transformer model should I use?

Creating a custom model

Running models

Example requests

REST API

Response

Python

Response

Pre-trained `transformer` models