Tokens

The basic syntactical units of the language are called tokens. A token consists of one or more characters, excluding blanks, control characters, and characters within a string constant or delimited identifier. (These terms are defined later.)

Tokens are classified as ordinary or delimiter tokens:

An ordinary token is a numeric constant, an ordinary identifier, a host identifier, or a keyword.
Examples
```
    1        .1        +2        SELECT        E        3
```
A delimiter token is a string constant, a delimited identifier, an operator symbol, or any of the special characters shown in the syntax diagrams. A question mark (?) is also a delimiter token when it serves as a parameter marker, as explained under PREPARE.
Examples
```
    ,      'Myst Island'        "fld1"        =        .
```

Spaces:

A space is a sequence of one or more blank characters.

Control Characters:

A control character is a special character that is used for string alignment. The following table contains the control characters that are handled by the database manager:

Table 1. Control Characters
Control Character	EBCDIC Hex Value	Unicode Graphic Hex Value
Tab	05	U+0009
Form Feed	0C	U+000C
Carriage Return	0D	U+000D
New Line	15	U+0085
Line Feed (New line)	25	U+000A
DBCS Space	—	U+3000

Tokens, other than string constants and certain delimited identifiers, must not include a control character or space. A control character or space can follow a token. A delimiter token, a control character, or a space must follow every ordinary token. If the syntax does not allow a delimiter token to follow an ordinary token, then a control character or a space must follow that ordinary token. The following examples illustrate the rule that is stated in this paragraph.

Here are some examples of combinations of the above ordinary tokens that, in effect, change the tokens:

   1.1        .1+2        SELECTE        .1E        E3        SELECT1

This demonstrates why ordinary tokens must be followed by a delimiter token or a space.

Here are some examples of combinations of the above ordinary tokens and the above delimiter tokens that, in effect, change the tokens:

   1.        .3

The period (.) is a delimiter token when it is used as a separator in the qualification of names. Here the dot is used in combination with an ordinary token of a numeric constant. Thus, the syntax does not allow an ordinary token to be followed by a delimiter token. Instead, the ordinary token must be followed by a space.

If the decimal point has been defined to be the comma, as described in Decimal point, the comma is interpreted as a decimal point in numeric constants. Here are some examples of these numeric constants:

   1,2        ,1        1,        1,e1

If '1,2' and '1,e1' are meant to be two items, both the ordinary token (1) and the delimiter token (,) must be followed by a space, to prevent the comma from being interpreted as a decimal point. Although the comma is usually a delimiter token, the comma is part of the number when it is interpreted as a decimal point. Therefore, the syntax does not allow an ordinary token (1) to be followed by a delimiter token (,). Instead, an ordinary token must be followed by a space.

Comments:

Dynamic SQL statements can include SQL comments. Static SQL statements can include host language comments or SQL comments. Comments may be specified wherever a space may be specified, except within a delimiter token or between the keywords EXEC and SQL. In Java™, SQL comments are not allowed within embedded Java expressions. There are two types of SQL comments:

simple comments: Simple comments are introduced by two consecutive hyphens (--). Simple comments cannot continue past the end of the line. For more information, see SQL comments.
bracketed comments: Bracketed comments are introduced by /* and end with */. A bracketed comment can continue past the end of the line. For more information, see SQL comments.

Uppercase and Lowercase:

Any token in an SQL statement may include lowercase letters, but a lowercase letter in an ordinary token is folded to uppercase, except for variables in the C and Java languages, which have case-sensitive identifiers. Delimiter tokens are never folded to uppercase. Thus, the statement:

   select * from EMP where lastname = 'Smith';

is equivalent, after folding, to:

   SELECT * FROM EMP WHERE LASTNAME = 'Smith';