Part II: Wimbledon Facebook Bot on IBM Cloud

5 min read

By: David Provan

Identifying intents and broadcasting

This series of articles was contributed and developed along with Alyssa Small and Brian Adams

This series of posts will detail how IBM iX designed, developed, and delivered the Facebook Messenger Bot available at the The Championships, Wimbledon 2018.  The series is broken into three pieces:

  • Part IThis article will focus on the purpose of the Bot, our high-level approach, and how we developed the integration with the Facebook Messenger Platform.

  • Part II: This piece will focus more on the broadcast integration within Facebook and how we persisted user preferences using IBM Cloudant and Compose for Redis.

  • Part III: Finally, we will review our integrations with on-site systems at the All England Club and how we used Multi-Region within IBM Cloud to ensure scale and availability.

Moving beyond the basics

In Part I of this series, we detailed how IBM iX set up our liberty runtime to be able to communicate with the Facebook platform. In this piece, we are going to focus on two primary areas:

  • How we identified which service to execute upon receiving contact from a messenger client

  • How we developed a broadcast system for sending alerts

Broadcasting updates to our users

One of the core offerings we wanted our Chat Bot to have was to utilize Subscription Messaging on Facebook. This functionality would allow to us to let users subscribe to scoring and news updates for certain players in the Wimbledon Championships

Subscription messages make use of labels on Facebook; our solution would need to maintain a list of which users were subscribed to which players and create associated labels on the Facebook platform to subscribe to. When a player broadcast was then sent, we would need to create the message and post it to the label.

We identified Cloudant as our preferred datastore and designed a solution utilizing it as our subscription messaging datastore.

This blog post will detail our approach to the Cloudant implementation along with the architectural approaches.

Cloudant multi-region development

This experience was not our first Cloudant solution, and one of the primary lessons we have learned through various Cloudant implementations is the importance of designing in data integrity into your application architecture. This becomes more pronounced when you start to utilize multiple Cloudant clusters for availability.

The solution below helps us deal with our requirement to implement a multi-region Cloudant solution. We would highly recommend adhering to these lessons regardless of your intended solution scale. In our experience, it’s far easier to plan in data integrity rather than try and retrospectively fit it into the solution.

We’ll start with a very basic overview of how Cloudant replication works. As we said earlier, we intended to use a multi-region Cloudant set-up. We would then set up two-way replication between the instances to ensure that they were kept in sync.  This is demonstrated in the diagram below:

Cloudant multi-region development

The _rev field is used by Cloudant replication to identify changes to documents, and then they are replicated between the instances. If your application makes extensive use of Document updates along with high volume, you can find documents that end up in “conflict.” Our experience tells us that the use of “conflict” here causes more confusion than it helps. It can have the implication that something broke, when in fact Cloudant is trying to help in this case. It could merge the documents before potentially using the data. Instead, the documents are held in “conflict” so that you can identify the correct data and create the final version with the most up-to-date data. If you want this done automatically, you need to design your application in such a way as to identify the correct data, create a new revision, and remove the previous conflicted documents. We won’t attempt to rewrite the excellent advice and descriptions available on developerworks for this. We have found that when people transition from SQL-like databases to noSQL, there is a tendency to keep the same thinking around updates rather than checking the data model and adapting it a more suitable format for Cloudant.

Our approach, therefore, has been to move our approach to utilizing Create and Delete requests and Views to make the best of what Cloudant offers us. For this implementation, we decided that every subscription would be it’s own unique document in Cloudant, and if a user unsubscribed from the alert, we would DELETE the document from the DB to mark that preference as removed.

As such we created a document structure as follows per subscription:

{
  "psid": ,
  "classifier": "scores",
  "label": {
    "labelId": 1416024498498316,
    "labelName": "atpf324_scores",
    "classifier": "scores",
    "type": "label",
    "created": 1530358229775
  },
  "type": "subscription",
  "lastUpdated": 1531252352667,
  "created": 1531252352667
}

When a user clicked subscribe within the Facebook Messenger platform, we would create the label if it didn’t already exist and associate the user’s PS ID with it before persisting that in our DB.

When a user requested to see all things, they were subscribed for it with a simple view that listed all subscriptions by PS ID:

function (doc) {
  if (doc.type == 'subscription'){
    emit(doc.psid, 1);
  }
}

Intent identification

Our implementation was looking to reuse the “Fred” bot that was able to understand natural language queries and respond back to the user accordingly. However, we also wanted to be able to surface some of the Facebook custom features with natural language where it was appropriate to do so.

Our basic approach would be as follows. Upon receiving text from a user, we would pass the text to Watson Assistant for analysis. If that text was identified with high confidence that it was a “basic command” to shortcut the user to scores, latest news, or players, we would present the according Facebook layout. If we weren’t confident on the intent match or found no matching intent in Watson Assistant, the text was passed unaltered into the Fred implementation for review and response.

To implement this, we used the Watson Java SDK and added the Watson Assistant implementation to our gradle build file:

compile group: 'com.ibm.watson.developer_cloud', name: 'assistant', version: '6.1.0'

This allowed us to only import the part of the SDK we needed, instead of the whole thing.

We then took the text from Facebook and used the SDK to query the Assistant service (we used VCAP Services to get the credentials as we had a UK and US South instance for resiliency and availability).

public IntentIdentificationService() {

        service = new Assistant("2018-02-16");

        String VCAP_SERVICES = System.getenv("VCAP_SERVICES");

        String user;
        String password;

        if (VCAP_SERVICES != null) {
            // parse the VCAP JSON structure
            JsonObject obj = (JsonObject) new JsonParser().parse(VCAP_SERVICES);
            Entry<String, JsonElement> dbEntry = null;
            Set<Entry<String, JsonElement>> entries = obj.entrySet();
            // Look for the VCAP key that holds the cloudant no sql db information
            for (Entry<String, JsonElement> eachEntry : entries) {
                if (eachEntry.getKey().toLowerCase().contains("conversation")) {
                    dbEntry = eachEntry;
                    break;
                }
            }
            if (dbEntry == null) {
                throw new RuntimeException("Could not find conversation key in VCAP_SERVICES env variable");
            }

            obj = (JsonObject) ((JsonArray) dbEntry.getValue()).get(0);
            obj = (JsonObject) obj.get("credentials");

            user = obj.get("username").getAsString();
            password = obj.get("password").getAsString();

        }
        else {
            user = “local-test-user”
            password = "local-test-password";

        }
        service.setUsernameAndPassword(user, password);

    }

    public IntentIdentifcationResult identify(String text) {
        final String methodName = "identify";
        logger.entering(className, methodName, text);

        IntentIdentifcationResult result = null;

        try {
            InputData input = new InputData.Builder(text).build();
            MessageOptions options = new MessageOptions.Builder(WORKSPACE_ID).input(input).build();
            MessageResponse response = service.message(options).execute();

            // System.out.println("Assistant response:" + response);
            if (null != response.getOutput().getText() && !response.getOutput().getText().isEmpty()) {
                result = new IntentIdentifcationResult(response.getIntents(), response.getEntities(), response.getOutput().getText(), response.getContext().toString());
            }
            else {
                result = new IntentIdentifcationResult(response.getIntents(), response.getEntities(), response.getContext().toString());
            }
        }
        catch (Exception e) {
            logger.logp(Level.SEVERE, className, methodName, String.format("Error identifying intent for %s", text), e);
            result = new IntentIdentifcationResult();
        }

        logger.exiting(className, methodName, result);
        return result;
    }

Wrap-up

This article focused on our Cloudant implementation and the broadcast integration within Facebook. We made some significant decisions early on that allowed us to make the use of what Cloudant could provide us “out of the box.” Cloudant helped us greatly as we implemented and delivered our solution, allowing us to create a scalable and highly available solution on IBM Cloud.

Finally, our use of Watson Assistant as a simple intent identification service allowed a simple integration for Natural Language Queries. It also our allowed our team to retrain and do better intent identification in real time with the Watson Assistant training and test tools.

Be the first to hear about news, product updates, and innovation from IBM Cloud