Expression language

The IBM StreamSets expression language enables you to create expressions that evaluate or modify data. The IBM StreamSets expression language is based on the JSP 2.0 expression language.

Use the expression language to configure expressions and conditions in processors, such as the Expression Evaluator or Stream Selector. Some target properties also allow the expression language, such as the directory template for the Local FS target.

You can use the expression language to define any stage or flow property that represents a numeric or string value. You can also use field path expressions to select the fields to use in some processors.

Use expression completion to determine where you can use an expression and the expression elements that you can use in that location.

You can use the following elements in an expression:

For more information about using expressions in flows, see Expression configuration.

Expression completion in properties

Expression completion provides a list of functions and other elements of the StreamSets expression language that are valid for use at that location.

When an element does not display in the list, it is not a valid element at the specified location.

Tips for expression completion

Use the following information and tips to work with expression completion:

  1. To invoke expression completion and see all available elements, place the cursor where you want to create an expression and click Ctrl + Space Bar.

    A list of valid expression elements displays. When an element does not display in the list, it is not a valid element at the specified location. Scroll to view the entire list.

    List of valid expression elements

    You can use expression completion wherever you can define an expression.

  2. You can filter the element list by typing a few letters of the element name.

    Element names that include your input appear at the top of the list. Below that, it displays elements that include the individual letters that you entered.

  3. Expression completion automatically displays when possible when you type in an expression.

    For example, after typing in re in an expression, the following list displays:

    List of valid expression elements that start with "re"

  4. Expression completion lists elements using the following icons:
    • c icon for constants

    • f icon for functions

    • Square icon for fields
  5. To view more information about an element, highlight or select the element name:

    List of valid expression elements with one element highlighted a description of that element displayed

  6. To add an element to an expression, click the element name or use the Enter/Return key on the highlighted element.
  7. When you add a function, it displays with syntax placeholders. You can view the full syntax by clicking within the parentheses.

    A function with the syntax of the function displayed

Expression examples

The following tables include some examples of how you might use expressions in the Data Collector.

Conditions
The following table includes some examples of conditions:
Sample Condition Expressions Description
${record:value('/payment_type') == 'CRD'} In a Stream Selector, this condition routes records where the payment type is credit card to the stream.
${record:value('[3]/State') == "MD"} The same as above, but for a record in list format.
${record:errorCode()=="FIELD_MERGER_02"} Used in a Stream Selector in an error flow, this condition routes records with the FIELD_MERGER_02 error to a stream that corrects the error.
${record:type('/ID')=STRING} In the Stream Selector, routes records where the ID is a String to a branch that includes a Field Type Converter to convert the field to a numeric data type.
if-then-else
The following table includes some examples of if-then-else expressions:
Sample if-then-else Expressions Description

${record:value('/gender')=='1'?'M':(record:value('/gender')=='2'?'F':'U')}

 Replaces 1 with "M", 2 with "F", and any other value with "U" for unknown.

${record:value('/phone_home') != ' ' ? record:value('/phone_home'):

(record:value('/phone_mobile') != ' ' ? record:value('/phone_mobile'):

(record:value('/phone_work') != ' ' ? record:value('/phone_work'): ' '))}
The phone number to use for a primary_phone field based on phone numbers in the home_phone, mobile_phone and work_phone fields:
  • If there's a home phone number, use the home phone.
  • If there's no home phone number, use the mobile phone number.
  • If there's no home or mobile phone number, use the work phone number.
  • If there are no home, mobile, or work numbers, use a null value.
String manipulation
The following table includes some string manipulation examples:
Sample String Manipulation Expressions Description
${record:valueOrDefault('/Payment', 'unknown')} Replaces missing or null values in the Payment field with "unknown".
${str:toUpper(record:value('/STATE')} Capitalizes all strings in the STATE field.
Numeric operation
The following table includes an example of a numeric operation:
Sample Numeric Operations  
${record:value('/total_amount') - (record:value('/tip_amount') + record:value('/tolls') + record:value('/mta_tax'))} An expression that calculates trip revenue by subtracting the tip, taxes, and tolls from the total fare amount