Reporting transcription events from SMS Gateway

Configuring SMS Gateway to publish transcription events in an IBM Cloudant database

Before configuring SMS Gateway to publish to IBM Cloudant, set up a IBM® Cloudant® for IBM Cloud. When you generate your service credentials from the IBM Cloudant dashboard, you can choose either IAM or a combination of API key and username and password.

Set the following environment variables in your configuration to publish transcription events in an IBM Cloudant database. You can use both IBM Cloudant and another reporting service simultaneously by adding the IBM Cloudant configuration variables to your configuration.

The following examples show the configuration for publishing turn events to IBM Cloudant.

• Single-tenant environment: Set the following environment variables in your configuration. You can configure authentication for IBM Cloudant either with an API key by adding the REPORTING_TRANSCRIPTION_CLOUDANT_APIKEY variable or by username and password by adding REPORTING_TRANSCRIPTION_CLOUDANT_USERNAME and REPORTING_TRANSCRIPTION_CLOUDANT_PASSWORD.

REPORTING_TRANSCRIPTION_CLOUDANT_URL =http://mytranscriptioncloudant.ibm.com/
REPORTING_TRANSCRIPTION_CLOUDANT_APIKEY=a1b2c3d3fgh1jk1mn0
REPORTING_TRANSCRIPTION_CLOUDANT_USERNAME=myCloudantUsername
REPORTING_TRANSCRIPTION_CLOUDANT_PASSWORD=myCloudantPassword
REPORTING_TRANSCRIPTION_CLOUDANT_DB_NAME=myCloudantDB
REPORTING_TRANSCRIPTION_CLOUDANT_EVENT_INDEX=transcription

• Multi-tenant JSON configuration: In the multi-tenant JSON configuration file, for each tenant where you want to enable transcription events, configure a reporting object that contains the following properties. If you use non-default account configurations to IBM Cloudant, you might need to include additional environment variables. See transcriptionCloudant in Properties for the reporting object. You can configure authentication for IBM Cloudant either with an API key by adding the apiKey variable or by username and password by adding username and password.

  "reporting": {
  "transcriptionCloudant": {
    "url": "http://transcriptionCloudantURL.ibm.com/",
    "dbName": "myDBName",
    "username": "myAdmin",
    "password": "myTokenOrPassword",
    "apiKey": "myApiKey",
    "eventInd": "transcription"
  }
  }
  

Configuring SMS Gateway to publish transcription events in CouchDB

To set up CouchDB, clone the repository and follow the instructions on GitHub. If you use CouchDB, you can't use an API key to authenticate.

Set the following environment variables in your configuration to publish transcription events in a CouchDB database. You can use both CouchDB and another reporting service simultaneously by adding the CouchDB configuration variables to your configuration.

The following examples show the configuration for publishing turn events to IBM Cloudant.

• Single-tenant environment: Set the following environment variables in your configuration.

REPORTING_TRANSCRIPTION_CLOUDANT_URL="http://<svc-couchdb>:5984/"
REPORTING_TRANSCRIPTION_CLOUDANT_USERNAME=myCouchDBUsername
REPORTING_TRANSCRIPTION_CLOUDANT_PASSWORD=myCouchDBPassword
REPORTING_TRANSCRIPTION_CLOUDANT_DB_NAME=myCouchDB
REPORTING_TRANSCRIPTION_CLOUDANT_EVENT_INDEX=transcription

• Multi-tenant JSON configuration: In the multi-tenant JSON configuration file, for each tenant where you want to enable transcription events, configure a reporting object that contains the following properties. See transcriptionCloudant in Properties for the reporting object.

  "reporting": {
  "transcriptionCloudant": {
    "url": "http://<svc-couchdb>:5984/",
    "dbName": "myDBName",
    "username": "myAdmin",
    "password": "myTokenOrPassword",
    "eventInd": "transcription"
  }
  }
  

Configuring SMS Gateway to publish transcription events

1. Set up a Splunk HTTP Event Collector (HEC) or a REST server that can store the events, for example in a noSQL database.

2. Configure your REST server URL and authorization credentials and enable transcription events in SMS Gateway.

   • Single-tenant environment: Set the following environment variables in your configuration.

   REPORTING_URL=http://myresteventserver.ibm.com/
   REPORTING_USERNAME=myRestAdmin
   REPORTING_PASSWORD=myRestTokenOrPassword
   REPORTING_TRANSCRIPTION_EVENT_INDEX=transcription
   

   • Multi-tenant JSON configuration: In the multi-tenant JSON configuration file, for each tenant where you want to enable transcription reporting, configure a reporting object that contains the following properties. You can configure indexes for other types of reporting events within the same object.

     "reporting": {
      "url": "http://myresteventserver.ibm.com/",
      "username": "myRestAdmin",
      "password": "myRestTokenOrPassword",
      "transcriptionEventInd": "transcription"
     }
   

   The transcription event index value is included as the index value in all transcription events so that the REST server that consumes them can differentiate the type of event. The index field is required for Splunk HEC, but it's also useful for anyone who is building their own REST server to handle these events.

After you redeploy SMS Gateway, it will publish event records to the configured REST server every time an utterance is detected.

Transcription event format

All SMS Gateway reporting events are based on the Splunk HTTP Event Controller JSON format.

Important: Transcription events include text transcriptions and other information that could potentially contain Protected Health Information (PHI), personally identifiable information (PII), or PCI Data Security Standard (PCI DSS) data. Therefore, it's critical that the consuming server properly stores these events to prevent exposure of personal information.

In the following example, the transcription event report is generated for an inbound message to Watson Assistant from the user or SMS Provider.

{
  "time": 1532551497,
  "host": "a2b4cd3f09",
  "source": "tenantID",
  "sourceType": "smsgw",
  "index": "transcription",
  "event": {
    "transcription": "Hello",
    "tenantID": "+18887776666",
    "receiver": "+18887776666",
    "sender": "+18889990000",
    "sessionID": "fe98d765-4ccc-3210-9bca-876zz54y3xw2",
    "direction": "inbound",
    "destination": "a2bc3def-4g56-78h9-01i2-34jk55l67m8n",
    "destinationType": "conversation",
    "conversationID": "d3ef4ghi-5j67-89k0-12l3-45mn67o89p0q"
  }
}

In the following example, the transcription event report is generated for an outbound MMS message to Watson Assistant from the user or SMS Provider.

{
  "time": 1558536481,
  "host": null,
  "source": "+18887776666",
  "sourcetype": "smsgw",
  "index": "transcription",
  "event": {
    "transcription": "How can I help you today?",
    "tenantID": "+18887776666",
    "receiver": "+18889990000",
    "sender": "+18887776666",
    "sessionID": "1f2dfec3-c99d-4b53-a87e-d7ecec03d3fd",
    "direction": "outbound",
    "destination": "+18889990000",
    "destinationType": "sms-provider",
    "conversationID": "7ec88882-b832-44c7-91d3-b1c97032fa6f",
    "mediaURLs": [
      "https://media.com/media/l4JyRqcDU93S334KQ/first.gif",
      "https://media.com/media/l4JyRqcDU93S334KQ/second.gif"
    ]
  }
  }

In the following example, the transcription event report is generated for an inbound MMS message to Watson Assistant from the user or SMS Provider.

{
  "time": 1532551497,
  "host": "a2b4cd3f09",
  "source": "tenantID",
  "sourceType": "smsgw",
  "index": "transcription",
  "event": {
    "transcription": "Hello",
    "tenantID": "+18887776666",
    "receiver": "+18887776666",
    "sender": "+18889990000",
    "sessionID": "fe98d765-4ccc-3210-9bca-876zz54y3xw2",
    "direction": "inbound",
    "destination": "a2bc3def-4g56-78h9-01i2-34jk55l67m8n",
    "destinationType": "conversation",
    "conversationID": "d3ef4ghi-5j67-89k0-12l3-45mn67o89p0q",
    "mediaURLs": [
      "https://blah.blah.com/2010-04-01/Accounts/AC32b5411d925440xxxxa6f05427c4xxxx/Messages/MM7d78f699365a682f2441cd45496cb3d9/Media/MEaaf3424e479605cbd6c9c87360xxxxxx"
    ]
  }
}

In cases where an MMS message is sent or received without any text (SMS) content, "transcription" does not show up in the report.