This is the fourth article in a series of blog posts on how the summer interns at IBM Watson are training robots to play poker with humans.
In the previous article, we have seen how we use the Watson Speech to Text service convert speech from an audio stream to text. In this blog post, we will see how the Natural Language Classifier service helps in interpreting the meaning of a sentence.
The Natural Language Classifier Service
The Natural Language Classifier service makes it really simple to determine the intent of any given text by identifying its classification with corresponding confidence values. Though it might seem like a difficult task, the service makes it easy by letting the developers focus on creating a good set of training data and forget about the rest.
To use this service, you will need to create an instance of the Natural Language Classifier service in your Bluemix account. The blog post on Speech to Text service has detailed instructions on how this is done. Once this step is completed, you will receive your credentials to the service which will be required to authenticate with the service.
Interactions with the Natural Language Classifier service happen via a REST API. We will now see details on creating training data for the classifier and also how we use the service from Watson Developer Cloud with code examples in Python. There is also documentation available on using this service with other programming platforms such as cURL, Node, Java: here.
Creating Training Data for the Classifier
In our project, we are not only training the robots to play poker but also making them conversational. For the robots to act as human as possible, we should make them respond to commands from the dealer, decisions of the players, and casual questions from anyone. For this purpose, we have classifiers to distinguish casual questions from the poker commands, to identify categories of casual speech and so on.
To create a new classifier, you will first need a good amount of training data. This data is given in .CSV format wherein each line in the file is referred as a record. Each record is a combination of a string of text followed by different classes to which this text can be mapped. For example, one record might be I decide to call,pokerwhich means the text I decide to call should be classified as a poker statement. Also, it is required that your training file has a minimum of five records. You can refer to the guidelines on constructing training data to see the different constraints and supported formats for training data.
Let us create training data for a scenario that requires a classifier to distinguish poker commands from casual sentences. This use case would require two classes in the training data namely: poker and casual. The classifier that we build using this training set should classify text that is a poker command as poker and anything that is not related to poker should be classified as casual.
Create a new file trainingData.csv that contains data such as:
I decide to call,poker
I decide to check,poker
I decide to fold,poker
Here are the community cards,poker
This is a flop,poker
This is a pre flop,poker
"Hey, how are you",casual
How is your family,casual
Can you tell me a joke?,casual
Fold your chair,casual
I will call you in sometime,casual
You can add many more records and include different variations of text, for the classifier to determine correct classification with
good confidence values. In this scenario, we can see that it is important to classify only sentences that mean 'I will fold' to poker,
but any other sentence like 'Fold your chair' should not be classified to poker. To handle these ambiguous cases, you have to add
other variations of text that don't mean the same under appropriate class type.
Now that we have our training data ready, all we need is to train a classifier with this data and use it to classify any test data to the appropriate class.
Creating training data for classifier is a continuous process. Writing training data for the first time, might require a lot of guess work based on the user inputs you are anticipating. Once you start using the classifier to classify test data, it good to log its results. This will be of great help to evaluate the performance of classifier, identify the areas of improvement and refine training data accordingly.
Using Natural Language Classifier Service with Watson Developer Cloud
To work with the Natural Language Classifier service, you should have watson-developer-cloud installed. Refer to the first article on Solution Overview to find instructions on how this can be done. Once you are done with the installation, create a file classifier.py and import the Natural Language Classifier client:
from watson_developer_cloud import NaturalLanguageClassifierV1
This import should succeed if the installation was done correctly. Now that we have everything ready, let’s create a classifier and train it on the data we just wrote :
from watson_developer_cloud import NaturalLanguageClassifierV1
natural_language_classifier = NaturalLanguageClassifierV1(
with open('trainingData.csv', 'rb') as training_data:
classifier = natural_language_classifier.create(
Here, we used english as the language format, but the service also supports many other languages that are listed here.
The method train_classifier() will create new classifier and trains it on the input data provided. As you can see below, this method outputs various details about the classifier such as current status, assigned name, a unique classifier ID and so on. It is important to save the value of classifier_id returned here, in order to access the classifier later on.
"name": "My Classfier",
"status_description": "The classifier instance is in its training phase, not yet ready to accept classify requests",
This process of training takes ten minutes or longer depending on the size of your training data. To keep track of its status, we will create a method to query the status of the classifier:
status = natural_language_classifier.status('<ClassifierID>')
print (json.dumps(status, indent=2))
Once the classifier is trained, the status displays as Available. At this point, you are all set to use the classifier to classify test data and evaluate its performance. So, we define a method that can take any text as input and use the classifier to find the class it belongs to:
classes = natural_language_classifier.classify('<ClassifierID>', 'I am going to fold')
This function will return confidence values with which the text classifies to each class in the training data. Obvious as it is, the class that is returned with the highest confidence is the top_class in the JSON response.
"text": "I will be folding now",
These are some of the basic methods you will need to work with the Natural Language Classifier service. You might want to browse through the source code for this service to see various other functionality provided.
Here is the classifier we have created in action!
Using Results of Classifiers
In our project, when the robots listen to some speech, they interact with the Speech to Text client to convert user input to text. Then, a classifier is used to understand the intent of this text.
If it is a poker statement, we determine what poker command it is and make necessary changes to the game state. When they hear a command like, ‘Megatron, it is your turn to act’, they need to determine the strength of their cards and resolve their decision for the round. For this purpose, we have created a custom service that takes card details as input and returns the strength for that card combination.
If it is a casual statement, we make the robot give an appropriate response. IBM Watson came up a new API called the Conversation Service which is makes it really easy to enable an application converse with user and guide users through any sequence of steps. We have been using Dialog Service for this purpose, as the Conversation service was only available for use sometime after we started our project.
This is how the Natural Language Classifier service can be used to interpret user inputs and generate responses accordingly. In the next article we will see how IBM Watson’s Visual Recognition Service helps in recognizing images and determines its classification. If you have any questions, feel free to post on DeveloperWorks Answersor join the Slack community (get an invite here).
How can AI improve the employee experience? In this episode of thinkPod, we are joined by x.ai co-founder & CEO Dennis Mortensen and Ben Jackson, founder of For the Win. We talk to Dennis and Ben about hiring algorithms and the danger of bias, whether HR teams are equipped to make data-driven decisions, Inbox Zero versus Inbox Infinity, and the possibility of cultural change.
Introducing, smart document understanding (SDU), a powerful feature now available in Watson Discovery, that can give you accurate answers faster, from 10 days to 2 minutes for one leading bank. Building on advances from IBM Research and drawing on the corpus conversion service released in late 2018, smart document understanding allows you visually train AI to understand your documents.
IBM's Think 2019 was an industry-leading conference that took place February 12-15 in San Francisco and included 2,000+ technical and business sessions with execs from ExxonMobil, Sprint, Honda, KPM, and others. In addition, more than 800 leaders, 400 developers and 200 distinguished engineers from IBM appeared on stage and in private sessions. Speakers ranged from football great Joe Montana, skateboard pioneer Tony Hawk and astronaut Taylor Richardson.