uniq — Report or filter out repeated lines in a file

Format

uniq [–c|–d|–u] [–B] [–f number1] [–s number2] [-W option[,option]...] [input_file [output_file]]
uniq [–c|–d|–u] [–-number] [-+number] [–B] [-W option[,option]...] [input_file [output_file]]

Description

uniq manipulates lines that occur more than once in a file. The file must be sorted, since uniq only compares adjacent lines. When you invoke this command with no options, it writes only one copy of each line in input_file to output_file. If you do not specify input_file or you specify –, uniq reads the standard input.

If you do not specify output_file, uniq uses the standard output. The specified output_file cannot be a FIFO.

Options

–B

Disables the automatic conversion of tagged files. This option is ignored if the filecodeset or pgmcodeset options (-W option) are specified.

–c

Precedes each output line with the number of times that line occurred in the input.

–d

Displays only lines that are repeated (one copy of each line).

–f number1

Ignores the first number1 fields when comparing lines. Blanks separate fields in the input.

–s number2

Ignores the first number2 characters when comparing lines. If you specify both –s and –f, uniq ignores the first number2 characters after the first number1 fields.

–u

Displays only those lines that are not repeated.

-W option[,option]...

Specifies z/OS-specific options. The option keywords are case-sensitive. Possible options are:

filecodeset=codeset

Performs text conversion from one code set to another when reading from the file. The coded character set of the file is codeset. codeset can be a code set name known to the system or a numeric coded character set identifier (CCSID). Note that the command iconv -l lists existing CCSIDs along with their corresponding code set names. The filecodeset and pgmcodeset options can be used on files with any file tag.

If pgmcodeset is specified but filecodeset is omitted, then the default file code set is ISO8859-1 even if the file is tagged with a different code set. If neither filecodeset nor pgmcodeset is specified, text conversion will not occur unless automatic conversion is enabled or the _TEXT_CONV environment variable indicates text conversion. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

If filecodeset or pgmcodeset is specified, then automatic conversion is disabled for this command invocation and the -B option is ignored if it is also specified. See z/OS UNIX System Services Planning for more information about automatic conversion.

When specifying values for filecodeset, use the values that Unicode Service supports. For more information about supported code sets, see z/OS Unicode Services User's Guide and Reference.

pgmcodeset=codeset

Performs text conversion from one code set to another when reading from the file. The coded character set of the program (command) is codeset. codeset can be a code set name known to the system or a numeric coded character set identifier (CCSID). Note that the command iconv -l lists existing CCSIDs along with their corresponding code set names. The filecodeset and pgmcodeset options can be used on files with any file tag.

If filecodeset is specified but pgmcodeset is omitted, then the default program code set is IBM-1047. If neither filecodeset nor pgmcodeset is specified, text conversion will not occur unless automatic conversion is enabled or the _TEXT_CONV environment variable indicates text conversion. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

Restriction: The only supported values for pgmcodeset are IBM-1047 and 1047.

You can choose only one of the –c, –d, or –u options.

–number: Equivalent to –f number (obsolescent).
+number: Equivalent to –s number (obsolescent).

Examples

The command:
```
uniq
```
is a filter which prints one copy of each different line in its sorted input.
The command:
```
uniq -f 2 -s 1
```
compares lines starting with the second character of the third field.
The command:
```
uniq -d
```
prints one instance of each repeated line in the input (and omits all unique lines).
To print one copy of each different line in a text file containing UTF-8 characters, assuming that
- The text file is untagged and you do not want to tag it or enable automatic conversion, and
- You cannot alter the tag (for example, you are displaying an untagged public text file or a read-only text file)
then issue:
```
uniq -W filecodeset=UTF-8,pgmcodeset=IBM-1047 myUtf8File
```
To print only those lines that are not repeated in a text file containing EBCDIC characters, assuming that automatic conversion has been enabled but the text file is incorrectly tagged as ASCII:
```
uniq -u -B myMisTaggedFile
```

Localization

uniq uses the following localization environment variables:

LANG
LC_ALL
LC_CTYPE
LC_MESSAGES
NLSPATH

See Localization for more information.

Environment variables

uniq uses the following environment variable:

_TEXT_CONV: Contains text conversion information for the command. The text conversion information is not used when either the -B option or the filecodeset or pgmcodeset option (-W option) is specified. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

Exit values

0

Successful completion

1

Failure due to any of the following:

Incorrect command-line option
Missing number after –f
Missing or incorrect number after –s
Inability to open the input or output file
The code set is not valid
Could not turn off automatic conversion
Could not perform requested text conversion

3

Failure while reading the input file:

Error during system call
Input line too long
Incorrect character in input

Messages

Possible error messages include:

Missing character skip count: You specified –s but did not supply a number after the –s.
Missing number of fields to skip: You specified –f but did not supply a number after the –f.
Field skip not a number in string: In a –number or +number construct, number was not a valid number. This could happen because of a typographical error in entering a – option.

Portability

POSIX.2, X/Open Portability Guide, UNIX systems.

The –B and -W options are extensions of the POSIX standard. The –number and +number options are considered obsolete.

Related information

comm, sort