Working with arbitrary tokens

TokenList parameter types and Subcommand elements with IsArbitrary=True consist of arbitrary tokens, such as character strings, literals (strings delimited by single or double quotes), and operators like '>'. Typical scenarios where arbitrary tokens are required include variable lists that may contain BY or WITH specifications and model effects lists. For example, a subcommand that allows arbitrary tokens for specifying model interaction effects might be given as:

/MODEL var1*var2(var3)

When parsed, this results in the following list of six tokens passed to the Run function of the implementation code.

['var1','*','var2','(','var3',')']

Likewise, an anonymous subcommand that allows arbitrary tokens to specify variables with optional BY or WITH keywords might be specified as:

DepVar BY A B WITH X Y

When parsed, this results in the following list of seven tokens passed to the Run function of the implementation code.

['DepVar','BY','A','B','WITH','X','Y']

When working with arbitrary tokens you'll want to test the various forms of input that your implementation function will need to handle, since the tokens that result from the specified input may not always be what you expect. For example, a TokenList parameter named TOKENS might be specified as:

TOKENS = 1a 2b

When parsed, this results in the following list of four tokens passed to the Run function of the implementation code.

['1','a','2','b']

The result reflects the standard way that IBM® SPSS® Statistics tokenizes command syntax. In particular, when a set of digits precedes a set of characters, the digits are treated as a separate token. You can force a set of characters to be passed as a single token by enclosing them in quotes. For example, specifying TOKENS = '1a' '2b' results in the token list ['1a','2b']. The same applies if you need to specify multi-word phrases and have each phrase passed as a single token. For example, TOKENS = 'two words' results in the single token 'two words'.