Elasticsearch configuration

By default, from Elastic 5.0, Elasticsearch maps string fields to the text type. Elasticsearch parses the contents of text fields into tokens for full-text search.

You might not want that default behavior. Many string fields in log data are names or identifiers. It makes more sense to search these fields as whole values, so the keyword type is a better choice. You can configure Elasticsearch by creating an index template that maps string fields to the keyword type.

Here is an example of why you might choose to map string fields to the keyword type. Suppose you forward to Elasticsearch a string field named tran with the value STC@CICS. By default, Elasticsearch analyzes (tokenizes) that string field. When you refer to the tran field, Elasticsearch returns the value of the first token, stc. To return the complete, original value, you need to refer to the name of a separate, keyword (raw) version of that field.

If you want to avoid that default behavior, then, as a starting point, consider creating an index template that maps all string fields forwarded by Transaction Analysis Workbench to the keyword type. Then consider adjusting the mapping of specific string fields that you might want analyzed, such as URLs, to the text type.

For details on how Elasticsearch analyzes string fields, the differences in behavior when searching text fields versus keyword fields, how to create index templates, and the syntax for specific versions of Elasticsearch, see the Elasticsearch documentation.

Example: From Elastic 5.0

The following index template for the index pattern fuw-* maps all string fields to the keyword type.

{
  "template": "fuw-*",  1 
  "mappings": {
    "_default_": {
      "dynamic_templates": [ {
        "string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      } ]
    }
  }
}
 1 
For this index template to apply to data that you forward from Transaction Analysis Workbench, you must forward data to an Elasticsearch index that matches the fuw-* index pattern. The output section of your Logstash configuration file must specify an index property value that begins with fuw-.

Example: Elasticsearch 2.x

For the same reasons that you might wish to map string fields to keyword in later versions of Elastic, you might wish to set string fields in Elasticsearch 2.x to not_analyzed.

The following index template for the index pattern fuw-* maps all string fields to not_analyzed. This example is for Elasticsearch 2.4.

{
  "template" : "fuw-*",
  "mappings" : {
    "_default_" : {
      "dynamic_templates" : [ {
        "string_fields" : {
          "mapping" : {
            "index" : "not_analyzed",
            "omit_norms" : true,
            "type" : "string"
          },
          "match_mapping_type" : "string",
          "match" : "*"
        }
      } ]
    }
  }
}