Filters for JSON-format outputs

You can optionally filter the data to send to the JSON-format outputs of OMEGAMON® Data Connect: TCP, HTTP, Kafka, and STDOUT.

Note: The filters described here do not apply to the Prometheus output. The Prometheus output has its own parameters with similar behavior, connect.output.prometheus.mappings.products.k<pp>.tables.

You can filter data by product, table, and field name. The product, table, and field names depend on the data source. In the context of OMEGAMON monitoring agents as a data source, the product is an agent, the table is an OMEGAMON attribute group, and the field is an OMEGAMON attribute.

For each table, you can conditionally filter records by specifying an expression. OMEGAMON Data Connect only sends records for which the expression is true.

You can specify a global-level filter that applies to all JSON-format outputs and an output-level filter for each output. Output-level filters replace any global-level filter.

To specify a global-level filter, insert a filter key as a child of the connect root key:

  • connect.filter

To specify an output-level filter, insert a filter key as a child of the key for that output:

  • connect.output.stdout.filter
  • connect.output.tcp.sinks.<sink_name>.filter
  • connect.output.http.endpoints.<endpoint_name>.filter
  • connect.output.kafka.filter

If you specify a filter, then only the fields enabled by the filter are sent.

If you do not specify a filter, then all fields from all tables from all products are sent.

You can specify filters inline in the OMEGAMON Data Connect configuration file or in separate filter include files.

Global-level and output-level filters have the same format:

filter:
  enabled: <true|false>
  include: <file_path> # If specified, the products key is ignored
  products:
    k<pp>: # Product code
      enabled: <true|false>
      tables: # Optional. Default: send all tables from this product
        <table_name>:
          enabled: <true|false>
          condition: # Optional. Default: send all records from this table
            enabled: <true|false>
            expression: <SpEL expression>
            disable-table-on-error: <true|false>
          fields: # Optional. Default: send all fields from this table
            - <field_name>
            - ... # More attribute field names
        ...: # More table names
    ...: # More product codes
enabled
An enabled key can be specified at several levels in the filter parameters:
  • At the highest level, under the filter key
  • Under a k<pp> product code key
  • Under a <table_name> key
  • Under a condition key

Allowed values: true, false. Default at all levels: true.

The enabled key has the same effect at every level: setting enabled: false is equivalent to omitting, or commenting-out, the parent key and that parent key's descendants.

Key Effect of omitting the key, or setting the child key value enabled: false
filter No filter is set.

On an output-level filter: causes the global-level filter, if it is enabled, to take effect for that output.

filter.products.k<pp> No data from this product is sent.
filter.products.k<pp>.tables.<table_name> No data from this table is sent.
filter.products.k<pp>.tables.<table_name>.condition No condition is set. All records of this table are sent.
include
Optional. Uses the filter defined in a separate filter include file.

If you specify an include key, then OMEGAMON Data Connect ignores the sibling products key.

A filter include file is a YAML document that has the same format as the filter key in an OMEGAMON Data Connect configuration file, but without the root filter key.

Example filter include file:

enabled: true
products:
  km5:
    tables:
      ascpuutil: # Send all fields
        enabled: true

The filter include <file_path> can be absolute or relative. OMEGAMON Data Connect treats a relative file path as being relative to the working directory.

You cannot nest filter includes; you cannot specify an include key in a filter include file.

Filter include files must be encoded in UTF-8.

products
Only fields from the specified products (monitoring agents) are sent.

Strictly speaking, the products key is optional. Omitting the products key specifies an empty filter with no criteria, which has the same effect as no filter.

If you specify an include key, then the products key is ignored.

k<pp>
The 3-character kpp product code of the monitoring agent that owns the table.

You must specify at least one child key under the k<pp> key.

To send all tables from the product, omit the child tables key and explicitly specify enabled: true.

tables
Optional. Only fields from the specified tables are sent.

You must specify at least one child <table_name> key under the tables key.

If all <table_name> keys under a tables key are set to enabled: false, then no data from the product is sent.

<table_name>
The name of a table owned by the product.

You must specify at least one child key under the <table_name> key.

To send all fields from the table, omit the fields key and explicitly specify enabled: true.

condition
Optional. OMEGAMON Data Connect only sends records for which the condition expression is true. If the expression is false, OMEGAMON Data Connect discards the record.

The expression child key specifies an expression in the Spring Expression Language (SpEL). The expression can test field values in the table. For example:

condition:
  expression: cpu_time > 2

To test field values, you can either use relational operators or methods:

Expression using a relational operator Equivalent expression using a method
syncpoint_elapsed_time == 0 syncpoint_elapsed_time.equals(0)
cpu_time > 2 cpu_time.compareTo(2) > 0
cpu_time < 2 cpu_time.compareTo(2) < 0
(cics_region_name == 'CICSPRD' or cics_region_name == 'TSTRGN1') and syncpoint_elapsed_time == 0 cics_region_name.equals('CICSPRD') or cics_region_name.equals('TSTRGN1') and syncpoint_elapsed_time.equals(0)
Tip: The matches method offers a shorthand for testing alternative values, but uses regular expressions, which are typically more computationally expensive:

cics_region_name.matches('CICSPRD|TSTRGN1') and syncpoint_elapsed_time.equals(0)

(transaction_id != null) and (transaction_id matches 'PFX.*') transaction_id?.matches('PFX.*')
(total_other_wait_times != null) and (total_other_wait_times != 0) total_other_wait_times?.compareTo(0) > 0

The methods that you can use with a field depend on the Java class to which OMEGAMON Data Connect maps the field: String, Double, Integer, Long, or, for timestamp fields such as write_time, OffsetDateTime. All of these classes support the equals and compareTo methods. The String class also supports the matches method, for testing a field value against a regular expression. For details on these and other methods, see the Java documentation for each class.

Attention: Expressions can cause runtime errors or undesirable behavior. Test expressions thoroughly with your data before deploying them in a production environment.

If the expression syntax is invalid, then the Spring framework reports APPLICATION FAILED TO START, followed by the error details, and OMEGAMON Data Connect does not start.

If OMEGAMON Data Connect encounters a runtime exception while evaluating the expression, then OMEGAMON Data Connect performs the following actions:

  1. Discard the record currently being processed.
  2. Report warning message KAYC0057W.
  3. Depending on the value of disable-table-on-error:
    false (default)
    Continue processing records that use the expression.
    true
    1. Stop processing records that use the expression; disable the table for outputs that use this filter.

      If the expression is in an output-level filter, then OMEGAMON Data Connect disables the table for that output only. If the expression is in a global-level filter, then OMEGAMON Data Connect disables the table for all outputs that use the global-level filter.

    2. Report information message KAYC0056I.
Tip:
  • If an expression refers to a field that might not be in every record, then, to avoid throwing a null pointer runtime exception, either explicitly test the field for a null value (<field_name> != null) or use the safe navigation operator when accessing a method or property of the field. The safe navigation operator is a question mark (?) immediately after the field name.
  • Division by integer zero causes an error. However, division by floating-point zero does not cause an error. For details, see the Java™ documentation for division by zero.

For more information about SpEL, such as comprehensive details of the operators that you can use in an expression, see the Spring documentation.

To break long expressions over multiple lines in the configuration file, use one of the YAML folding styles. For example, line folding (>-):

condition:
  expression: >-
    cics_region_name.matches('CCVQ.*') and
    (total_io_wait_times +
    total_other_wait_times == 0)
fields
Optional. A list of field names to send from this table.

OMEGAMON Data Connect always sends the common fields write_time, product_code, and table_name; do not specify these in the list of field names. However, other common fields, such as interval_seconds, are sent only if you specify them in this list.

Example: Global-level filter to send all data from one product only

The following filter sends all data from the OMEGAMON z/OS® monitoring agent.

connect:
  filter:
    products:
      km5: # z/OS
        enabled: true

Example: No filter: send all data from all products

The following filter is the same as the previous example except for enabled: false directly under the filter key, disabling the entire filter.

connect:
  filter:
    enabled: false # Disables the entire filter
    products:
      km5:
        enabled: true

Example: Global-level filter to send all data from some products only

The following filter sends all data from the OMEGAMON z/OS, CICS, and CICS TG monitoring agents, but blocks all data from other agents. For instance, if OMEGAMON Data Connect receives data from the OMEGAMON Db2 monitoring agent, then OMEGAMON Data Connect does not send the data from that agent, because the filter does not enable the corresponding kd5 product code.

connect:
  filter:
    enabled: true
    products:
      km5:
        enabled: true
      kc5: # CICS
        enabled: true
      kgw: # CICS TG
        enabled: true

The following filter is equivalent to the previous filter. The resulting behavior is identical. The only difference is that the following filter contains an entry for the Db2 monitoring agent marked enabled: false (effectively, a comment).

connect:
  filter:
    enabled: true
    products:
      km5:
        enabled: true
      kc5:
        enabled: true
      kgw:
        enabled: true
      kd5: # Db2: do not send
        enabled: false

Example: Global-level and output-level filters to send data from different products to different outputs

In the following example:

  • The global-level filter sends data from the OMEGAMON z/OS monitoring agent only.
  • The Kafka output and the logstash1 TCP output have no output-level filters, so they use the global-level filter.
  • Two of the TCP outputs have output-level filters: the logstash2 output sends data from the OMEGAMON Db2 and IMS monitoring agents only, and the splunk output sends data from the CICS and Java monitoring agents only.
  • Only required enabled keys are shown; in this example, all of the omitted enabled keys default to true.
connect:
  filter: # Global-level
    products:
      km5:
        enabled: true
  output:
    kafka: # Uses global-level filter
      enabled: true
      servers: kafka.example.com:9095
      topic: omegamon-json
    tcp:
      enabled: true
      sinks:
        logstash1: # Uses global-level filter
          hostname: elastic1.example.com
          port: 5046
        logstash2:
          hostname: elastic2.example.com
          port: 5046
          filter: # Output-level
            products:
              kd5:
                enabled: true
              ki5:
                enabled: true
        splunk:
          hostname: splunk.example.com
          port: 5047
          filter: # Output-level
            products:
              kc5:
                enabled: true
              kjj:
                enabled: true

Example: Filter to send selected fields from one product only

The following global-level filter sends data from the OMEGAMON z/OS monitoring agent: all fields from table ascpuutil, but only the specified fields from table km5wlmclpx.

connect:
  filter:
    enabled: true
    products:
      km5:
        enabled: true
        tables:
          ascpuutil: # Send all fields
            enabled: true
          km5wlmclpx:
            fields: # Send only these fields
              - managed_system
              - class_name
              - class_type
              - transaction_rate
              - transaction_completions
              - transaction_total

In the following example, there is no global-level filter. Only the Kafka output is filtered.

connect:
  output:
    kafka:
      enabled: true
      servers: kafka.example.com:9095
      topic: omegamon-json
      filter: # Output-level: applies to Kafka output only
        enabled: true
        products:
          km5:
            enabled: true
            tables:
              ascpuutil: # Send all fields
                enabled: true
              km5wlmclpx:
                fields: # Send only these fields
                  - managed_system
                  - class_name
                  - class_type
                  - transaction_rate
                  - transaction_completions
                  - transaction_total
    stdout: # Unfiltered
      enabled: true
    tcp:
      enabled: true
      sinks:
        logstash: # Unfiltered
          hostname: elastic1.example.com
          port: 5046

Example: Global-level filter with condition

The following global-level filter restricts output to records of the OMEGAMON z/OS monitoring agent table ascpuutil that are for sysplex PLEXA.

connect:
  filter:
    enabled: true
    products:
      km5:
        tables:
          ascpuutil: # Send all fields
            condition:
              expression: sysplex_name?.equals('PLEXA')

  output:
    tcp:
      enabled: true
      sinks:
        logstash:
          hostname: elastic1.example.com
          port: 5046

Example: Output-level filters with conditions

The following output-level filters send records of the OMEGAMON z/OS monitoring agent table ascpuutil to different outputs for different sysplexes.

connect:
  output:
    tcp:
      enabled: true
      sinks:
        logstash1: # Sysplex PLEXA output
          hostname: elastic1.example.com
          port: 5046
          filter:
            products:
              km5:
                tables:
                  ascpuutil:
                    condition:
                      expression: sysplex_name?.equals('PLEXA')
        logstash2: # Sysplex PLEXB output
          hostname: elastic2.example.com
          port: 5046
          filter:
            products:
              km5:
                tables:
                  ascpuutil:
                    condition:
                      expression: sysplex_name?.equals('PLEXB')

Example: Filter include file

The following TCP output uses a filter include file.

connect:
  output:
    tcp:
      enabled: true
      sinks:
        analytics:
          hostname: analytics.example.com
          port: 5046
          filter:
            include: /var/omdp/filters/analytics.yaml

Example: Empty output-level filter to send all data from all products

Suppose that you have a global-level filter that restricts output, but you want a particular output to be unfiltered. You can achieve this by specifying an output-level filter with enabled: true but no criteria; no products key or include key. The empty output-level filter replaces the global-level filter.

In the following example, STDOUT output is unfiltered:

connect:
  filter:
    enabled: true
    products:
      km5:
        ascpuutil: # Send all fields
        enabled: true
  output:
    stdout:
      enabled: true
      filter: # Enabled but empty: does not restrict output
        enabled: true

Example: Global-level filter that disables the table if a runtime exception occurs in the condition expression

connect:
  filter:
    enabled: true
    products:
      km5:
        tables:
          ascpuutil:
            condition:
              expression: sysplex_name.equals('PLEXA') # No safe navigation operator (?) after sysplex_name
              disable-table-on-error: true # If sysplex_name field is missing, stop processing records from this table