Entity-mentions

The generic entity models

The models for entity type systems have been trained and tested on labeled data from news reports. These models have two parts:

• A rule-based model (the rbr models), which handles syntactically regular entity types such as number, email and phone.

• A model trained on labeled data for the more complex entity types such as person, organization, or location.

The rbr sire, and bilstm models are monolingual: each model knows how to analyze input text in a single language.

The bert model is multilingual: the single model can analyze input texts from multiple languages.

The bilstm models use GloVe embeddings trained on the Wikipedia corpus in each language.

The bert model uses the Google Multilingual BERT model (Large, Cased, 104 languages).

The entity-mentions_transformer-workflow_lang_multilingual_slate.153m.distilled model is optimized for GPU, but supports CPU usage. entity-mentions_transformer-workflow_lang_multilingual_slate.153m.distilled-cpu is a CPU-only model, optimized for better speed on CPU. These models are trained on 24 languages and are based on IBM Slate Multilingual pretrained resource.

All models output non-overlapping entity mention spans. That is, each character in the input text can belong to either no entity type or exactly one entity type and there are no overlapping entities.

The PII entity models

The PII models recognize personal identifiable information such as person names, SSN, bank account numbers, credit card numbers, etc.

Due to the nature of PII, it is difficult to train machine learning models for the majority of PII, especially credit card numbers, passport numbers and other identifiers. Therefore, the PII model has two parts:

• A rule-based model handles the majority of the types by identifying common formats of PII entities and performing possible checksum/validations as appropriate for each entity type. For example, credit card number candidates are validated using the Luhn algorithm.

• A model trained on labeled data for types where labeled data can be obtained, such as person and location. For this, use one of the models available for the Entity-mention V2 type system.

REST API

curl -s \
  "http://localhost:8080/v1/watson.runtime.nlp.v1/NlpService/EntityMentionsPredict" \
  -H "accept: application/json" \
  -H "content-type: application/json" \
  -H "Grpc-Metadata-mm-model-id: entity-mentions_rbr_lang_multi_pii" \
  -d '{ "raw_document": { "text": "My email is john@ibm.com." }, "language_code": "en" }'

{"mentions":[
  {"span":{
    "begin":12,
    "end":24,
    "text":"john@ibm.com"
    },
   "type":"EmailAddress",
   "producerId":{
    "name":"RBR mentions",
    "version":"0.0.1"
    },
   "confidence":0.8,
   "mentionType":"MENTT_UNSET",
   "mentionClass":"MENTC_UNSET",
   "role":""
   }
   ],
   "producerId":{
    "name":"RBR mentions",
    "version":"0.0.1"
   }
  }

Python

import grpc

from watson_nlp_runtime_client import (
    common_service_pb2,
    common_service_pb2_grpc,
    syntax_types_pb2,
)

channel = grpc.insecure_channel("localhost:8085")

stub = common_service_pb2_grpc.NlpServiceStub(channel)

request = common_service_pb2.EntityMentionsRequest(
    raw_document=syntax_types_pb2.RawDocument(text="My email is john@ibm.com"),
    language_code='en'
)

  response = stub.EntityMentionsPredict(
    request, metadata=[("mm-model-id", "entity-mentions_rbr_lang_multi_pii")]
)

print(response)

mentions {
  span {
    begin: 12
    end: 24
    text: "john@ibm.com"
  }
  type: "EmailAddress"
  producer_id {
    name: "RBR mentions"
    version: "0.0.1"
  }
  confidence: 0.8
}
producer_id {
  name: "RBR mentions"
  version: "0.0.1"
}