Filters for JSON-format outputs

You can filter data by product, table, and field name. The product, table, and field names depend on the data source. In the context of OMEGAMON monitoring agents as a data source, the product is an agent, the table is an OMEGAMON attribute group, and the field is an OMEGAMON attribute.

For each table, you can conditionally filter records by specifying an expression. OMEGAMON Data Connect only sends records for which the expression is true.

You can specify a global-level filter that applies to all JSON-format outputs and an output-level filter for each output. Output-level filters replace any global-level filter.

To specify a global-level filter, insert a filter key as a child of the connect root key:

• connect.filter

To specify an output-level filter, insert a filter key as a child of the key for that output:

• connect.output.stdout.filter
• connect.output.tcp.sinks.<sink_name>.filter
• connect.output.http.endpoints.<endpoint_name>.filter
• connect.output.kafka.filter

If you specify a filter, then only the fields enabled by the filter are sent.

If you do not specify a filter, then all fields from all tables from all products are sent.

You can specify filters inline in the OMEGAMON Data Connect configuration file or in separate filter include files.

Global-level and output-level filters have the same format:

filter:
  enabled: <true|false>
  include: <file_path> # If specified, the products key is ignored
  products:
    <product_code>:
      enabled: <true|false>
      tables: # Optional. Default: send all tables from this product
        <table_name>:
          enabled: <true|false>
          condition: # Optional. Default: send all records from this table
            enabled: <true|false>
            expression: <SpEL expression>
            disable-table-on-error: <true|false>
          fields: # Optional. Default: send all fields from this table
            - <field_name>
            - ... # More field names
        ...: # More table names
    ...: # More product codes

enabled

An enabled key can be specified at several levels in the filter parameters:

• At the highest level, under the filter key
• Under a <product_code> key
• Under a <table_name> key
• Under a condition key

Allowed values: true, false. Default at all levels: true.

The enabled key has the same effect at every level: setting enabled: false is equivalent to omitting, or commenting-out, the parent key and that parent key's descendants.

Key	Effect of omitting the key, or setting the child key value enabled: false
filter	No filter is set. On an output-level filter: causes the global-level filter, if it is enabled, to take effect for that output.
filter.products.<product_code>	No data from this product is sent.
filter.products.<product_code>.tables.<table_name>	No data from this table is sent.
filter.products.<product_code>.tables.<table_name>.condition	No condition is set. All records of this table are sent.

include

Optional. Uses the filter defined in a separate filter include file.

If you specify an include key, then OMEGAMON Data Connect ignores the sibling products key.

A filter include file is a YAML document that has the same format as the filter key in an OMEGAMON Data Connect configuration file, but without the root filter key.

Example filter include file:

enabled: true
products:
  km5:
    tables:
      ascpuutil: # Send all fields
        enabled: true

The filter include <file_path> can be absolute or relative. OMEGAMON Data Connect treats a relative file path as being relative to the working directory.

You cannot nest filter includes; you cannot specify an include key in a filter include file.

Filter include files must be encoded in UTF-8.

products

Only fields from the specified products, such as the specified OMEGAMON monitoring agents, are sent.

Strictly speaking, the products key is optional. Omitting the products key specifies an empty filter with no criteria, which has the same effect as no filter.

If you specify an include key, then the products key is ignored.

<product_code>

The product code of a data source that you want to filter.

For OMEGAMON monitoring agents as a data source, the product code is the 3-character kpp product code of the monitoring agent.

For the product codes of other data sources, see the separate documentation for that software.

You must specify at least one child key under the <product_code> key.

To send all tables from the product, omit the child tables key and explicitly specify enabled: true.

tables

Optional. Only fields from the specified tables are sent.

You must specify at least one child <table_name> key under the tables key.

If all <table_name> keys under a tables key are set to enabled: false, then no data from the product is sent.

<table_name>

The name of a table owned by the product.

You must specify at least one child key under the <table_name> key.

To send all fields from the table, omit the fields key and explicitly specify enabled: true.

condition

Optional. OMEGAMON Data Connect only sends records for which the condition expression is true. If the expression is false, OMEGAMON Data Connect discards the record.

The expression child key specifies an expression in the Spring Expression Language (SpEL). The expression can test field values in the table. For example:

condition:
  expression: cpu_time > 2

To test field values, you can either use relational operators or methods:

Expression using a relational operator	Equivalent expression using a method
syncpoint_elapsed_time == 0	syncpoint_elapsed_time.equals(0)
cpu_time > 2	cpu_time.compareTo(2) > 0
cpu_time < 2	cpu_time.compareTo(2) < 0
(cics_region_name == 'CICSPRD' or cics_region_name == 'TSTRGN1') and syncpoint_elapsed_time == 0	cics_region_name.equals('CICSPRD') or cics_region_name.equals('TSTRGN1') and syncpoint_elapsed_time.equals(0) Tip: The matches method offers a shorthand for testing alternative values, but uses regular expressions, which are typically more computationally expensive: cics_region_name.matches('CICSPRD|TSTRGN1') and syncpoint_elapsed_time.equals(0)
(transaction_id != null) and (transaction_id matches 'PFX.*')	transaction_id?.matches('PFX.*')
(total_other_wait_times != null) and (total_other_wait_times != 0)	total_other_wait_times?.compareTo(0) > 0

The methods that you can use with a field depend on the Java class to which OMEGAMON Data Connect maps the field: String, Double, Integer, Long, or, for timestamp fields such as write_time, OffsetDateTime. All of these classes support the equals and compareTo methods. The String class also supports the matches method, for testing a field value against a regular expression. For details on these and other methods, see the Java documentation for each class.

Attention: Expressions can cause runtime errors or undesirable behavior. Test expressions thoroughly with your data before deploying them in a production environment.

If the expression syntax is invalid, then the Spring framework reports APPLICATION FAILED TO START, followed by the error details, and OMEGAMON Data Connect does not start.

If OMEGAMON Data Connect encounters a runtime exception while evaluating the expression, then OMEGAMON Data Connect performs the following actions:

1. Discard the record currently being processed.
2. Report warning message KAYC0057W.
3. Depending on the value of disable-table-on-error:

   false (default)

   Continue processing records that use the expression.

   true

   1. Stop processing records that use the expression; disable the table for outputs that use this filter.

      If the expression is in an output-level filter, then OMEGAMON Data Connect disables the table for that output only. If the expression is in a global-level filter, then OMEGAMON Data Connect disables the table for all outputs that use the global-level filter.

   2. Report information message KAYC0056I.

Tip:

• If an expression refers to a field that might not be in every record, then, to avoid throwing a null pointer runtime exception, either explicitly test the field for a null value (<field_name> != null) or use the safe navigation operator when accessing a method or property of the field. The safe navigation operator is a question mark (?) immediately after the field name.
• Division by integer zero causes an error. However, division by floating-point zero does not cause an error. For details, see the Java™ documentation for division by zero.

For more information about SpEL, such as comprehensive details of the operators that you can use in an expression, see the Spring documentation.

To break long expressions over multiple lines in the configuration file, use one of the YAML folding styles. For example, line folding (>-):

condition:
  expression: >-
    cics_region_name.matches('CCVQ.*') and
    (total_io_wait_times +
    total_other_wait_times == 0)

fields

Optional. A list of field names to send from this table.

OMEGAMON Data Connect always sends the common fields write_time, product_code, and table_name; do not specify these in the list of field names. However, other common fields, such as interval_seconds, are sent only if you specify them in this list.